Image alignment method, device, and storage medium

Abstract

Embodiments of the present application provide an image alignment method and device and a storage medium. The method comprises: obtaining a plurality of original images, each original image adopting a Bayer array arrangement format, and each original image comprising a plurality of G channels; for any one of the plurality of original images, determining a corresponding gray image of the original image based on the plurality of G channels contained in the original image; determining a descriptor of a feature point in the corresponding gray image of the original image; and aligning the plurality of original images based on the descriptors of the feature points in the corresponding gray images of the plurality of original images. The method solves the problem of unstable feature extraction of Bayer original images, provides high-quality gray input for subsequent feature matching, and significantly improves the robustness and efficiency of the ORB algorithm in the original domain.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image alignment method, device and storage medium. Background Technology

[0002] In current photographic techniques, multi-frame noise reduction (MFNR) and high dynamic range (HDR) imaging have become core methods for improving image quality. MFNR technology effectively suppresses sensor noise while preserving details by fusing multiple images exposed at the same time; HDR technology expands the dynamic range by fusing images exposed at different times, while preserving details in both bright and dark areas. However, the prerequisite for multi-frame image fusion is that the displacement and distortion caused by camera shake, shooting time differences, or user hand movements in each frame must be aligned. If the alignment is inaccurate, the fused image will exhibit ghosting, blurring, or localized distortion, severely affecting image quality.

[0003] Given the limited resources (such as computing power, memory bandwidth, and power consumption) of mobile platforms, Oriented Features from Accelerated SegmentTest and Rotated Binary Robust Independent Elementary Features (ORB) algorithms have become commonly used feature matching algorithms on mobile devices due to their high computational efficiency, low memory consumption, and moderate matching accuracy. However, when traditional ORB algorithms are directly applied to Bayer raw images, problems such as discontinuous gradient calculations and unstable feature point extraction arise because adjacent pixels have different color components. Summary of the Invention

[0004] The image alignment method, device, and storage medium provided in this application embodiment solve the technical problem of unstable Bayer original image feature extraction.

[0005] In a first aspect, embodiments of this application provide an image alignment method, including:

[0006] Acquire multiple frames of raw images, each frame using a Bayer array arrangement format, and each frame includes multiple G channels;

[0007] For any one of the multiple original images, a grayscale image corresponding to the original image is determined based on the multiple G channels contained in the original image.

[0008] Determine the descriptors of feature points in the grayscale image corresponding to the original image;

[0009] The original images are aligned based on the descriptors of feature points in the grayscale images corresponding to the original images.

[0010] In one possible implementation, determining the grayscale image corresponding to the original image based on the multiple G channels contained in the original image includes:

[0011] Obtain the pixel values ​​of multiple G channels in the original image;

[0012] Based on the pixel values ​​of multiple G channels in the original image, determine the average pixel value of multiple G channels in the original image;

[0013] Based on the pixel mean of multiple G channels in the original image, the grayscale image corresponding to the original image is determined.

[0014] In one possible implementation, the original image includes multiple color channels, including the multiple G channels; before determining the descriptors of feature points in the grayscale image corresponding to the original image, the method further includes:

[0015] Determine the overexposed region in the grayscale image corresponding to the original image, wherein the grayscale value of each pixel in the overexposed region is greater than or equal to the overexposed threshold;

[0016] For any pixel in the overexposed area, determine the pixel values ​​of multiple color channels corresponding to that pixel.

[0017] Replace the grayscale value of the pixel with the average of the pixel values ​​of the multiple color channels corresponding to the pixel.

[0018] In one possible implementation, the multiple original images include a reference frame image and at least one frame to be aligned; when the original image is any one of the at least one frames to be aligned, the method further includes, before determining the descriptor of feature points in the grayscale image corresponding to the original image:

[0019] Determine the exposure ratio of the image to be aligned, wherein the exposure ratio is the ratio of the exposure time of the reference frame image to the exposure time of the image to be aligned;

[0020] Based on the exposure ratio of the image to be aligned, brightness compensation is performed on the grayscale image corresponding to the image to be aligned to obtain a brightness-compensated image.

[0021] Determine the cumulative histogram of the grayscale image corresponding to the reference frame image and the cumulative histogram of the brightness-compensated image;

[0022] The mapping function is determined based on the cumulative histogram of the grayscale image corresponding to the reference frame image and the cumulative histogram of the brightness compensation image;

[0023] The brightness of the brightness-compensated image is corrected based on the mapping function.

[0024] In one possible implementation, before determining the descriptors of feature points in the grayscale image corresponding to the original image, the method further includes:

[0025] Determine the average total brightness of the grayscale image corresponding to the original image;

[0026] The gamma is determined based on the relationship between the average total luminance and the average threshold.

[0027] Based on the gamma, adjust the brightness of the grayscale image corresponding to the original image.

[0028] In one possible implementation, before determining the descriptors of feature points in the grayscale image corresponding to the original image, the method further includes:

[0029] Obtain the ISO value of the original image;

[0030] Based on the ISO value, the noise reduction intensity is determined;

[0031] Based on the noise reduction intensity, the grayscale image corresponding to the original image is subjected to noise reduction processing.

[0032] In one possible implementation, the multi-frame original images include a reference frame image and at least one frame to be aligned; aligning the multi-frame original images based on the descriptors of feature points in the grayscale images corresponding to the multi-frame original images includes:

[0033] For any one of the at least one frames of images to be aligned, based on the descriptors of feature points in the grayscale image corresponding to the image to be aligned and the descriptors of feature points in the grayscale image corresponding to the reference frame image, feature point matching is performed on the grayscale image corresponding to the image to be aligned and the grayscale image corresponding to the reference frame image to obtain a first set of matching point pairs.

[0034] Obtain the face region and rotation angle corresponding to the reference frame image, and the face region and rotation angle corresponding to the image to be aligned;

[0035] Based on the face region and rotation angle corresponding to the reference frame image, a first portrait region is determined, which is the portrait region in the grayscale image corresponding to the reference frame image; based on the face region and rotation angle corresponding to the image to be aligned, a second portrait region is determined, which is the portrait region in the grayscale image corresponding to the image to be aligned.

[0036] Traverse the matching point pairs in the first matching point pair set. If any feature point in the matching point pair is located in the first portrait region or the second portrait region, delete the matching point pair to obtain the second matching point pair set.

[0037] Based on the second set of matching point pairs, the image to be aligned is aligned with the reference frame image.

[0038] In one possible implementation, aligning the image to be aligned with the reference frame image based on the second set of matching point pairs includes:

[0039] The grayscale image corresponding to the reference frame image is divided into N×N sub-blocks, where N is a positive integer;

[0040] Perform multiple iterations based on the aforementioned N×N sub-blocks;

[0041] Each iteration includes: randomly selecting multiple different sub-blocks from the N×N sub-blocks; randomly selecting a feature point from each of the multiple sub-blocks as a first feature point; determining multiple matching point pairs from the second matching point pair set based on the multiple first feature points; and determining the homography matrix and the number of interior points corresponding to the homography matrix based on the multiple matching point pairs.

[0042] The optimal homography matrix is ​​determined based on multiple matching point pairs corresponding to the homography matrix with the most interior points in the multiple iterations.

[0043] Based on the optimal homography matrix, the image to be aligned is aligned with the reference frame image.

[0044] Secondly, embodiments of this application provide an image alignment device, comprising:

[0045] The first acquisition module is used to acquire multiple frames of raw images. Each frame of raw image adopts the Bayer array arrangement format and includes multiple G channels.

[0046] The first determining module is used to determine the grayscale image corresponding to any one of the multiple frames of original images, based on the multiple G channels contained in the original image.

[0047] The second determining module is used to determine the descriptor of feature points in the grayscale image corresponding to the original image;

[0048] An alignment module is used to align the multiple original images based on the descriptors of feature points in the grayscale images corresponding to the multiple original images.

[0049] In one possible implementation, the first determining module is specifically used for:

[0050] Obtain the pixel values ​​of multiple G channels in the original image;

[0051] Based on the pixel values ​​of multiple G channels in the original image, determine the average pixel value of multiple G channels in the original image;

[0052] Based on the pixel mean of multiple G channels in the original image, the grayscale image corresponding to the original image is determined.

[0053] In one possible implementation, the original image includes multiple color channels, the multiple color channels including the multiple G channels; the apparatus further includes:

[0054] The third determining module is used to determine the overexposed region in the grayscale image corresponding to the original image, wherein the grayscale value of each pixel in the overexposed region is greater than or equal to the overexposed threshold.

[0055] The fourth determining module is used to determine the pixel values ​​of multiple color channels corresponding to any pixel in the overexposed area.

[0056] The replacement module is used to replace the grayscale value of the pixel with the average value of the pixel values ​​of multiple color channels corresponding to the pixel.

[0057] In one possible implementation, the multiple original images include a reference frame image and at least one frame to be aligned; if the original image is any one of the at least one frame to be aligned, the apparatus further includes:

[0058] The fifth determining module is used to determine the exposure ratio of the image to be aligned, wherein the exposure ratio is the ratio of the exposure time of the reference frame image to the exposure time of the image to be aligned;

[0059] The brightness compensation module is used to perform brightness compensation on the grayscale image corresponding to the image to be aligned based on the exposure ratio of the image to be aligned, so as to obtain a brightness compensation image.

[0060] The sixth determining module is used to determine the cumulative histogram of the grayscale image corresponding to the reference frame image and the cumulative histogram of the brightness compensation image;

[0061] The seventh determining module is used to determine the mapping function based on the cumulative histogram of the grayscale image corresponding to the reference frame image and the cumulative histogram of the brightness compensation image;

[0062] A brightness correction module is used to perform brightness correction on the brightness-compensated image based on the mapping function.

[0063] In one possible implementation, the device further includes:

[0064] The eighth determining module is used to determine the average total brightness of the grayscale image corresponding to the original image;

[0065] The ninth determining module is used to determine the gamma based on the relationship between the total average brightness and the average threshold;

[0066] A brightness adjustment module is used to adjust the brightness of the grayscale image corresponding to the original image based on the gamma.

[0067] In one possible implementation, the device further includes:

[0068] The second acquisition module is used to acquire the ISO value of the original image;

[0069] The tenth determining module is used to determine the noise reduction intensity based on the ISO value;

[0070] The noise reduction module is used to perform noise reduction processing on the grayscale image corresponding to the original image based on the noise reduction intensity.

[0071] In one possible implementation, the multi-frame original image includes a reference frame image and at least one frame to be aligned; the alignment module is specifically used for:

[0072] For any one of the at least one frames of images to be aligned, based on the descriptors of feature points in the grayscale image corresponding to the image to be aligned and the descriptors of feature points in the grayscale image corresponding to the reference frame image, feature point matching is performed on the grayscale image corresponding to the image to be aligned and the grayscale image corresponding to the reference frame image to obtain a first set of matching point pairs.

[0073] Obtain the face region and rotation angle corresponding to the reference frame image, and the face region and rotation angle corresponding to the image to be aligned;

[0074] Based on the face region and rotation angle corresponding to the reference frame image, a first portrait region is determined, which is the portrait region in the grayscale image corresponding to the reference frame image; based on the face region and rotation angle corresponding to the image to be aligned, a second portrait region is determined, which is the portrait region in the grayscale image corresponding to the image to be aligned.

[0075] Traverse the matching point pairs in the first matching point pair set. If any feature point in the matching point pair is located in the first portrait region or the second portrait region, delete the matching point pair to obtain the second matching point pair set.

[0076] Based on the second set of matching point pairs, the image to be aligned is aligned with the reference frame image.

[0077] In one possible implementation, the alignment module is specifically used for:

[0078] The grayscale image corresponding to the reference frame image is divided into N×N sub-blocks, where N is a positive integer;

[0079] Perform multiple iterations based on the aforementioned N×N sub-blocks;

[0080] Each iteration includes: randomly selecting multiple different sub-blocks from the N×N sub-blocks; randomly selecting a feature point from each of the multiple sub-blocks as a first feature point; determining multiple matching point pairs from the second matching point pair set based on the multiple first feature points; and determining the homography matrix and the number of interior points corresponding to the homography matrix based on the multiple matching point pairs.

[0081] The optimal homography matrix is ​​determined based on multiple matching point pairs corresponding to the homography matrix with the most interior points in the multiple iterations.

[0082] Based on the optimal homography matrix, the image to be aligned is aligned with the reference frame image.

[0083] Thirdly, embodiments of this application provide an electronic device, including: a memory and a processor;

[0084] The memory stores computer-executed instructions;

[0085] The processor executes computer execution instructions stored in the memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.

[0086] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect and / or various possible implementations of the first aspect.

[0087] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the first aspect and / or various possible implementations of the first aspect.

[0088] The image alignment method, device, and storage medium provided in this application embodiment determine the grayscale image by using multiple G channels of the Bayer original image. Since multiple G channels can more stably preserve image edge information in low-light scenes, they reduce feature point loss caused by noise and solve problems such as discontinuous gradient calculation and unstable feature point extraction caused by different color components of adjacent pixels in the Bayer original image. Attached Figure Description

[0089] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0090] Figure 1 Flowchart of the image alignment method provided in the embodiments of this application Figure 1 ;

[0091] Figure 2 A schematic diagram illustrating the process of overexposure area compensation provided in an embodiment of this application;

[0092] Figure 3 This is a schematic diagram of the brightness unification processing in HDR mode provided in the embodiments of this application;

[0093] Figure 4 A schematic diagram of the adaptive brightness adjustment process provided in an embodiment of this application;

[0094] Figure 5 A schematic diagram of the ISO adaptive single-frame noise reduction process provided in the embodiments of this application;

[0095] Figure 6 Flowchart of the image alignment method provided in the embodiments of this application Figure 2 ;

[0096] Figure 7 Flowchart of the image alignment method provided in the embodiments of this application Figure 3 ;

[0097] Figure 8 This is a schematic diagram of the structure of the image alignment device provided in the embodiments of this application;

[0098] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0099] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0100] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0101] In the embodiments of this application, the term "at least one" refers to one or more, "multiple" refers to two or more, and other quantifiers are similar.

[0102] The terms "first," "second," etc., used in the embodiments of this application are for illustrative purposes and to distinguish the objects being described. They do not indicate any order or a specific limitation on the number of objects in the embodiments of this application, and cannot constitute any limitation on the embodiments of this application. For example, the use of descriptions such as "first matching point pair set" and "second matching point pair set" is only to distinguish different matching point pair sets, and does not indicate any difference in the size, priority, or importance of these two matching point pair sets.

[0103] It should be further understood that the terms "comprising" or "including" indicate the presence of the aforementioned features, steps, operations, elements, components, types, and / or groups, but do not exclude the presence, occurrence, or addition of one or more other features, steps, operations, elements, components, types, and / or groups.

[0104] In this application, terms such as "exemplary," "in some embodiments," and "in other embodiments" are used to indicate that something is an example, illustration, or description. Any embodiment or design described as "exemplary" in this application should not be construed as being more preferred or advantageous than other embodiments or designs. Rather, the term "exemplary" is used to present the concept in a specific manner.

[0105] It should be understood that although the steps in the flowcharts of this application's embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the figures may include at least one sub-step or at least one stage. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least a portion of the sub-steps or stages of other steps.

[0106] In related technologies, directly performing image alignment using raw images captured by a camera faces several problems:

[0107] 1. Bayer's original images have a unique Color Filter Array (CFA), where each pixel contains only one component from the three channels: Red (R), Green (G), and Blue (B). This non-uniform sampling characteristic makes it difficult to directly apply traditional computer vision algorithms. Specifically:

[0108] (1) Traditional ORB, Scale-Invariant Feature Transform (SIFT) and other feature detection algorithms rely on complete gradient information and texture features. However, the color components of adjacent pixels in Bayer raw images are different, which leads to discontinuous gradient calculation and unstable feature point extraction.

[0109] (2) The original image contains sensor noise, and the signal-to-noise ratio is extremely low under low light conditions. The matching accuracy of traditional feature matching algorithms drops sharply in high noise environments.

[0110] (3) The number of pixels in the G channel in the Bayer array is twice that of the R and B channels, which leads to an unbalanced feature distribution in different color channels and affects the robustness of feature matching.

[0111] 2. Smartphone platforms have strict real-time and resource consumption requirements for image processing algorithms. Specifically:

[0112] (1) Currently, mobile phones usually need to complete the entire photo-taking process in a short time. However, traditional high-precision alignment algorithms (such as SIFT-based or optical flow-based methods) have high computational complexity and cannot meet the real-time requirements.

[0113] (2) Complex image processing algorithms can significantly increase the load on the central processing unit (CPU) or graphics processing unit (GPU), leading to rapid battery consumption and affecting the user experience.

[0114] 3. Current image alignment algorithms have significant adaptability issues in mobile phone photography scenarios. Specifically:

[0115] (1) Although traditional algorithms such as SIFT or Speeded-Up Robust Features (SURF) have high accuracy, they have huge computational load, high computational complexity, and require a lot of floating-point operations, making them unsuitable for mobile platforms.

[0116] (2) Although the traditional ORB algorithm is computationally efficient, it was originally designed for grayscale or RGB images. When applied to the original domain, due to the lack of adaptation to the characteristics of the original image, the feature matching accuracy drops significantly, especially in low-texture areas and high-noise conditions.

[0117] (3) The optical flow method has a large computational load and is sensitive to noise, resulting in poor stability when applied in the original domain.

[0118] 4. In real-world mobile phone photography environments, image alignment faces many extreme conditions. For example, in low-light scenes, such as at night or in low-light indoor environments, the signal-to-noise ratio of the original image is extremely low (possibly below 5dB), rendering traditional feature matching algorithms ineffective.

[0119] In summary, the core technical problem to be solved by this application is: how to design an efficient, robust, and high-precision image alignment algorithm for multi-frame original images in Bayer format on a resource-constrained mobile platform, which can meet the strict requirements of MFNR and HDR fusion for real-time performance, accuracy, and robustness while preserving the original image information, and overcome the technical challenges under extreme shooting conditions such as low light and high noise.

[0120] The image alignment method, device, and storage medium provided in this application embodiment determine the grayscale image by using multiple G channels of the Bayer original image. Since multiple G channels can more stably preserve image edge information in low-light scenes, they reduce feature point loss caused by noise and solve problems such as discontinuous gradient calculation and unstable feature point extraction caused by different color components of adjacent pixels in the Bayer original image.

[0121] It should be noted that, in the embodiments of this application, the Bayer original image may refer to the original image arranged in the Bayer array format.

[0122] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0123] Figure 1 Flowchart of the image alignment method provided in the embodiments of this application Figure 1 ,like Figure 1 As shown, the method includes the following steps:

[0124] S101. Acquire multiple frames of raw images. Each frame of raw image adopts the Bayer array arrangement format and includes multiple G channels.

[0125] The execution subject of this application embodiment can be an electronic device, a chip, or a chip module, or it can be an image alignment device disposed in an electronic device, a chip, or a chip module. The image alignment device can be implemented by software or by a combination of software and hardware.

[0126] Among them, electronic devices can be digital devices with photography capabilities, such as mobile phones, digital SLR cameras, tablet computers, etc.

[0127] Multi-frame raw images can refer to raw images captured by a camera, such as raw images directly output by a complementary metal-oxide-semiconductor (CMOS) sensor.

[0128] Each pixel in the original image contains only one of the R, G, and B components.

[0129] In a Bayer array, pixels with the G component account for half of all pixels. For example, the Bayer array can be RGGB, BGGR, GBRG, or GRBG, etc., and this application does not limit this.

[0130] Each frame of the raw image includes multiple G channels. For example, each frame of the raw image includes two G channels.

[0131] S102. For any one of the multiple original images, determine the grayscale image corresponding to the original image based on the multiple G channels contained in the original image.

[0132] A grayscale image can refer to a single-channel image generated by calculating the pixel values ​​of multiple color channels.

[0133] In one possible implementation, for any given frame of the original image, its corresponding grayscale image can be determined as follows:

[0134] Obtain the pixel values ​​of multiple G channels in the original image; determine the average pixel value of multiple G channels in the original image based on the pixel values ​​of multiple G channels in the original image; determine the grayscale image corresponding to the original image based on the average pixel value of multiple G channels in the original image.

[0135] For example, assuming the Bayer array is RGGB, the pixel values ​​of the two G channels in the original image can be obtained. Based on the average pixel value of the two G channels in the original image, the corresponding grayscale image can be determined. That is, a 2×2 RGGB block can be... The values ​​are merged into a single grayscale pixel, and the pixel value of this grayscale pixel is the average value of the pixels in the two G channels, which is (G+G) / 2.

[0136] The grayscale image converted using the above method has a resolution that is half that of the original image.

[0137] Using the pixel mean of the dual G channels as the pixel value of the grayscale image can make full use of the characteristics of the Bayer array, which has the largest number of G channel pixels and is most sensitive to human vision. Compared with the traditional RGB weighted average method, this scheme significantly reduces the computational complexity while preserving high-frequency details.

[0138] S103. Determine the descriptors of feature points in the grayscale image corresponding to the original image.

[0139] That is, to determine the descriptor of the feature points in the grayscale image corresponding to each frame of the original image.

[0140] Determining the descriptors of feature points in the grayscale image corresponding to the original image is the core process of the traditional ORB algorithm, which specifically includes the following steps:

[0141] (1) The grayscale image corresponding to the original image is downsampled multiple times to generate grayscale images at multiple scale levels, i.e., to construct an image pyramid.

[0142] Each scale level is typically generated using Gaussian blur and downsampling, with a scale factor of 1.2. Feature point extraction and descriptor calculation can be performed independently at each pyramid level.

[0143] For example, assuming the original image corresponds to a grayscale image with a width of 640 pixels and a height of 480 pixels, the image pyramid has 8 layers and a scale factor of 1.2, the image pyramid is constructed using the following steps:

[0144] ① First, set the grayscale image corresponding to the original image as the 0th layer of the image pyramid. You can then apply a slight Gaussian blur (denoise removal) to the grayscale image corresponding to the original image without changing its size.

[0145] ② Layer-by-layer downsampling: starting from layer 0, each layer is obtained by dividing the image size of the previous layer by the scale factor (1.2). This is convenient for recording and calculation, and the actual pixel size is usually rounded down.

[0146] The width of the first layer image is approximately 533 pixels (640 / 1.2), and the height is approximately 400 pixels (480 / 1.2). The construction process of the first layer image is as follows: first, Gaussian blur is applied to the image of the 0th layer (to prevent jagged edges), and then the size is reduced to 533×400.

[0147] The width of the second layer image is approximately 444 pixels (533 / 1.2), and the height is approximately 333 pixels (400 / 1.2). The construction process of the second layer image is as follows: first, the first layer image is Gaussian blurred (to prevent jagged edges), and then the size is reduced to 444×333 pixels.

[0148] The width of the third layer image is 444 / 1.2 = 370, and the height is 333 / 1.2 ≈ 277. The construction process of the third layer image is as follows: first, Gaussian blur is applied to the second layer image (to prevent jagged edges), and then the size is reduced to 370×277.

[0149] The width of the fourth layer image is approximately 308 (370 / 1.2 ≈ 308), and the height is approximately 230 (277 / 1.2 ≈ 230). The construction process of the fourth layer image is as follows: first, Gaussian blur is applied to the third layer image (to prevent jagged edges), and then the size is reduced to 308×230.

[0150] The width of the fifth layer image is approximately 256 (308 / 1.2), and the height is approximately 191 (230 / 1.2). The construction process of the fifth layer image is as follows: first, Gaussian blur is applied to the fourth layer image (to prevent jagged edges), and then the size is reduced to 256×191.

[0151] The width of the 6th layer image is approximately 256 / 1.2 ≈ 213, and the height is approximately 191 / 1.2 ≈ 159. The construction process of the 6th layer image is as follows: first, Gaussian blur is applied to the 5th layer image (to prevent jagged edges), and then the size is reduced to 213×159.

[0152] The width of the 7th layer image is approximately 177 pixels (213 / 1.2), and the height is approximately 132 pixels (159 / 1.2). The construction process of the 7th layer image is as follows: first, Gaussian blur is applied to the 6th layer image (to prevent jagged edges), and then the size is reduced to 177×132.

[0153] (2) On the grayscale image at each scale level, feature points are detected using the Features from Accelerated Segment Test (FAST) algorithm.

[0154] Specifically, for any pixel p in the grayscale image at each scale level, an annulus of 16 pixels with a radius of 3 is examined. A brightness threshold t is set, and it is determined whether all M consecutive pixels (usually M=12) on the annulus are brighter than p+t or darker than pt. Pixel p that meets the condition is identified as a candidate feature point. ② A non-maximum suppression (NMS) strategy is employed to ensure that only the candidate feature point with the largest response value is retained within each local region. For any candidate feature point, the response value can be the sum of the absolute deviations of the 16 pixels surrounding the candidate feature point from the candidate feature point itself. For example, a local region can refer to a 3×3 neighborhood centered on each candidate feature point; this application does not limit the size of the local region.

[0155] (3) For any feature point O detected in step (2), calculate the gray-level centroid C of the image based on the circular region centered on feature point O. Specifically, analyze the gray-level distribution of pixels within the circular region centered on feature point O, and determine the gray-level centroid C of the circular region (i.e., the centroid of the gray-level distribution in the circular region); construct a direction vector based on feature point O and gray-level centroid C. This direction vector is the principal direction of the feature point. It is independent of image rotation and depends only on the gray-level distribution characteristics of the local image content.

[0156] (4) After assigning an orientation to each feature point, a rotation-invariant descriptor is generated for each feature point. This descriptor can be a Binary Robust Independent Elementary Features (BRIEF) descriptor. The BRIEF descriptor is represented by a 256-bit binary string and is generated by comparing the brightness values ​​of 256 pairs of selected pixels (i.e., test points) in the neighborhood of the feature point. The brightness value of each pair of pixels is (p1, p2). If p1 > p2, the corresponding bit is 1, otherwise it is 0. To enhance rotation invariance, the test point pair is rotated according to the orientation of the feature point.

[0157] In this context, the neighborhood of a feature point can refer to the image region surrounding that feature point, such as a 31×31 pixel image region centered on the feature point. In ORB, the 256 pixel pairs are not randomly generated for each image, but rather follow a pre-trained, fixed sampling pattern. That is, for each feature point, the relative coordinates of the predefined 256 pixel pairs can be multiplied by the rotation matrix of that feature point to obtain the corresponding 256 pixel pairs.

[0158] The goal of a descriptor is to transform the image region surrounding a feature point into a compact, fast-matching binary fingerprint (256 bits). Descriptors don't care about specific grayscale values; they only care which points within the image region surrounding the feature point are brighter than others.

[0159] S104. Align the multiple original images based on the descriptors of feature points in the grayscale images corresponding to the multiple original images.

[0160] The multi-frame original image includes a reference frame image and at least one frame to be aligned. Aligning the multi-frame original image means aligning at least one frame to be aligned with the reference frame image.

[0161] The reference frame image can be a normally exposed image from multiple original images, or the image with the best image quality; this application does not impose any restrictions on this.

[0162] For any frame of image to be aligned, alignment can be performed as follows: Using Hamming distance as a similarity metric, the descriptors of the grayscale images corresponding to the image to be aligned and the grayscale images corresponding to the reference frame image are matched. The nearest neighbor / second nearest neighbor ratio method is used to filter reliable matches, resulting in multiple matching point pairs. For these multiple matching point pairs, the homography matrix H is estimated using the Random Sample Consensus (RANSAC) algorithm: ① Randomly select four matching point pairs; ② Calculate the homography matrix H; ③ Count the number of inliers (points with reprojection error < a preset error threshold); ④ Repeat the iteration, selecting the H with the largest number of inliers; ⑥ Finally, recalculate the optimal H using all inliers corresponding to the H with the largest number of inliers. Based on the optimal H, the image to be aligned is aligned with the reference frame image.

[0163] The Hamming distance refers to the number of different characters at corresponding positions between two binary strings of equal length. For example, suppose descriptor A is 10101100 and descriptor B is 10111001. By comparing each character, we can see that the 4th, 6th, and 8th characters are different. Therefore, the Hamming distance between descriptor A and descriptor B is 3.

[0164] Suppose we have a feature point f in Figure A A Find its corresponding feature point in Figure B. Using the nearest neighbor / second nearest neighbor ratio method to screen reliable matches can refer to: calculating f... A Find the Hamming distance of the point f to all feature points in Figure B; find the point f with the smallest distance. B1 Let the minimum distance be d. min Find the point f that is the second smallest distance away. B2 Let the minimum distance be d. second Calculate d min With d second The ratio of f to f is less than a preset ratio threshold (usually 0.6~0.8), indicating that f... B1 Compared to f B2 More a match for f A If so, then accept this match, and denote the matching point pair as (f A f B1 Alternatively, if the ratio is close to 1, it indicates that f in Figure B... B1 and f B2 All with f A The textures are very similar (possibly duplicates), making them difficult to distinguish. To avoid errors, this match is rejected.

[0165] An interior point can be defined as follows: using H to verify all matching point pairs, for any matching point pair (A1, B1), the coordinates of feature point A1 in Figure A are projected onto Figure B through H to obtain the predicted position; the distance between the predicted position and the actual position of feature point B1 in Figure B (i.e., reprojection error) is calculated; if the reprojection error is less than a preset error threshold (e.g., 1.5 pixels), the matching point pair is considered to be an interior point that conforms to H.

[0166] exist Figure 1 In the embodiment shown, the grayscale image is determined by multiple G channels of the Bayer original image. Since multiple G channels can more stably preserve image edge information in low-light scenes, reduce feature point loss caused by noise, and solve problems such as discontinuous gradient calculation and unstable feature point extraction caused by different color components of adjacent pixels in the Bayer original image.

[0167] exist Figure 1 Based on the illustrated embodiment, for any one of the original images in the multi-frame original images, after executing S101 to S102, at least one of the following operations can be performed: overexposure area compensation, brightness unification processing in high dynamic range (HDR) mode, adaptive brightness adjustment, and ISO adaptive single-frame noise reduction, and then S103 and S104 are executed.

[0168] If all four operations—overexposure area compensation, HDR mode brightness unification processing, adaptive brightness adjustment, and ISO adaptive single-frame noise reduction—are performed, the execution order can be as follows: overexposure area compensation, HDR mode brightness unification processing, adaptive brightness adjustment, and ISO adaptive single-frame noise reduction.

[0169] The following is combined Figure 2 , Figure 3 , Figure 4 and Figure 5 The above four operations will be explained separately.

[0170] Figure 2 This is a schematic diagram of the overexposure area compensation process provided in an embodiment of this application, as shown below. Figure 2 As shown, it includes the following steps:

[0171] S201. Determine the overexposed areas in the grayscale image corresponding to the original image, where the grayscale value of each pixel in the overexposed area is greater than or equal to the overexposed threshold.

[0172] In this embodiment, the original image includes multiple color channels, including multiple G channels. For example, the original image includes four color channels, which consist of two G channels, one R channel, and one B channel.

[0173] The overexposure threshold T can be 0.95 × I.max I max This represents the maximum pixel value of the original image.

[0174] S202. For any pixel in the overexposed area, determine the pixel values ​​of the multiple color channels corresponding to the pixel.

[0175] For example, the pixel values ​​of multiple color channels corresponding to a pixel can refer to the pixel values ​​of the color channels in a 2×2 RGGB block in the original image corresponding to the pixel in the grayscale image.

[0176] S203. Replace the grayscale value of the pixel with the average value of the pixel values ​​of the multiple color channels corresponding to the pixel.

[0177] For example, the average pixel value of multiple color channels corresponding to a pixel can refer to the average pixel value of the four color channels in the 2×2 RGGB block of the original image corresponding to the pixel in the grayscale image, that is, the average value is (R+G+G+B) / 4.

[0178] exist Figure 2 In the illustrated embodiment, replacing the overexposed grayscale value of a pixel with the average pixel value of multiple color channels corresponding to that pixel ensures that effective feature information is still preserved in highlight areas. That is, Figure 2 The method shown can effectively avoid the loss of feature information in overexposed areas and ensure the stability of feature extraction across the entire scene.

[0179] When HDR shooting mode is detected, it is necessary to unify the brightness of the grayscale images corresponding to multiple original frames. The following section combines... Figure 3 Detailed explanation of the brightness unification process in HDR mode.

[0180] Figure 3 This is a schematic diagram of the brightness unification processing in HDR mode provided in the embodiments of this application, such as... Figure 3 As shown, it includes the following steps:

[0181] S301. Determine the exposure ratio of the image to be aligned. The exposure ratio is the ratio of the exposure time of the reference frame image to the exposure time of the image to be aligned.

[0182] In this embodiment, the multiple original images include a reference frame image and at least one frame to be aligned. If the original image is any one of the at least one frame to be aligned, this embodiment can also be executed before executing S103.

[0183] In the embodiments of this application, the image to be aligned may also be referred to as an underexposed frame image, and this application does not limit it in this regard.

[0184] S302. Based on the exposure ratio of the image to be aligned, perform brightness compensation on the grayscale image corresponding to the image to be aligned to obtain a brightness-compensated image.

[0185] The grayscale image corresponding to the image to be aligned can refer to the grayscale image obtained after executing S102, or it can refer to the grayscale image obtained after executing S102 and... Figure 2 The grayscale image obtained after the embodiment shown.

[0186] The grayscale image, brightness-compensated image, and exposure ratio corresponding to the image to be aligned satisfy the following formula:

[0187] I_current' = I_current × ER

[0188] Where I_current' is the pixel value of pixel A in the brightness-compensated image, I_current is the pixel value of pixel A in the grayscale image corresponding to the image to be aligned, ER is the exposure ratio of the image to be aligned, and pixel A is any pixel in the grayscale image, whose position is the same in both the grayscale image and the brightness-compensated image.

[0189] S303. Determine the cumulative histogram of the grayscale image corresponding to the reference frame image and the cumulative histogram of the brightness compensation image.

[0190] The grayscale image corresponding to the reference frame image can refer to the grayscale image obtained after executing S102, or it can refer to the grayscale image obtained after executing S102 and... Figure 2 The grayscale image obtained after the embodiment shown.

[0191] The cumulative histogram of each image can be determined using existing methods, and this application does not impose any restrictions on this.

[0192] S304. Determine the mapping function based on the cumulative histogram of the grayscale image corresponding to the reference frame image and the cumulative histogram of the brightness-compensated image.

[0193] The mapping function can be determined using existing methods, and this application does not impose any restrictions on this.

[0194] S305. Perform brightness correction on the brightness-compensated image based on the mapping function.

[0195] The histogram distribution of the brightness-compensated image after brightness correction is consistent with the histogram distribution of the grayscale image corresponding to the reference frame image. The corrected image can also be referred to as the grayscale image corresponding to the image to be aligned, or the grayscale image corresponding to the corrected image to be aligned.

[0196] exist Figure 3In the embodiment shown, for original images with different exposure times, brightness compensation is first performed on the images to be aligned according to the exposure ratio. Then, histogram matching technology is used to further achieve consistency in brightness distribution between frames, providing a basis for feature matching across exposure frames.

[0197] Figure 4 This is a schematic diagram of the adaptive brightness adjustment process provided in the embodiments of this application, as shown below. Figure 4 As shown, it includes the following steps:

[0198] S401. Determine the average total brightness of the grayscale image corresponding to the original image.

[0199] In this embodiment, the grayscale image corresponding to the original image may refer to the grayscale image obtained after executing S102, or it may refer to the grayscale image obtained after executing S102 and... Figure 2 The grayscale image obtained after the illustrated embodiment, or the image obtained after executing S102 and... Figure 3 The grayscale image obtained after the illustrated embodiment, or the image obtained after executing S102, Figure 2 The illustrated embodiments and Figure 3 The grayscale image obtained after the embodiment shown.

[0200] The average total brightness of a grayscale image can refer to the average brightness value or pixel value of all pixels in the grayscale image.

[0201] S402. Determine the gamma based on the relationship between the mean total luminance and the mean threshold.

[0202] Two mean thresholds can be preset: a first mean threshold and a second mean threshold, where the first mean threshold is smaller than the second mean threshold. If the total brightness mean is less than the first mean threshold, the image needs strong brightening, and the gamma can be set to 0.6. If the total brightness mean is greater than or equal to the first mean threshold and less than the second mean threshold, the image needs medium brightening, and the gamma can be set to 0.8. If the total brightness mean is greater than or equal to the second mean threshold, the image needs no brightening or only weak brightening, and the gamma can be set to 1.

[0203] For example, when the total brightness mean is a floating-point number between 0 and 1 (inclusive), the first mean threshold can be 0.1 and the second mean threshold can be 0.3.

[0204] S403. Based on gamma, adjust the brightness of the grayscale image corresponding to the original image.

[0205] For any given pixel, the brightness value I of that pixel in the adjusted grayscale image out The brightness value of that pixel in the grayscale image before adjustment. Satisfies the following formula: Iout = .

[0206] pass Figure 4 The embodiment shown performs non-linear brightening processing on dark areas based on the overall brightness distribution of the image. This can enhance the feature visibility of dark areas, ensure the detectability of feature points in low-light areas, improve the matching success rate under low-light conditions, and avoid overexposure in bright areas.

[0207] Figure 5 This is a schematic diagram of the ISO adaptive single-frame noise reduction process provided in the embodiments of this application, as follows: Figure 5 As shown, it includes the following steps:

[0208] S501. Obtain the ISO value of the original image.

[0209] S502. Determine the noise reduction intensity based on the ISO value.

[0210] For example, low ISO (<800): mild noise reduction, noise reduction intensity σ1; medium ISO (800-1600): moderate noise reduction, noise reduction intensity σ2; high ISO (>1600): strong noise reduction, noise reduction intensity σ3.

[0211] S503. Based on the noise reduction intensity, perform noise reduction processing on the grayscale image corresponding to the original image.

[0212] Noise reduction can be achieved using a simple and fast bilateral filter, prioritizing the preservation of edge features. Noise reduction can be performed on the grayscale plane after adaptive brightness adjustment to avoid noise interfering with subsequent feature extraction.

[0213] exist Figure 5 In the embodiment shown, the noise reduction intensity of a single frame is dynamically adjusted according to the ISO value of the original image. When the ISO is high, a strong noise reduction strategy is used to suppress sensor noise, and when the ISO is low, a weak noise reduction strategy is used to preserve details and avoid feature point loss caused by excessive smoothing.

[0214] Based on the above, regarding S104, the following will be combined with... Figure 6 and Figure 7 Provide a detailed explanation of the image alignment process.

[0215] Figure 6 Flowchart of the image alignment method provided in the embodiments of this application Figure 2 ,like Figure 6 As shown, it includes the following steps:

[0216] S601. For any frame of at least one image to be aligned, based on the descriptors of feature points in the grayscale image corresponding to the image to be aligned and the descriptors of feature points in the grayscale image corresponding to the reference frame image, feature point matching is performed on the grayscale image corresponding to the image to be aligned and the grayscale image corresponding to the reference frame image to obtain a first set of matching point pairs.

[0217] In this embodiment, the multiple original images include a reference frame image and at least one image to be aligned. The grayscale image corresponding to any image can be the grayscale image obtained after performing S101 to S102, or the grayscale image obtained after performing at least one of the following operations after performing S101 to S102: overexposure area compensation, brightness unification processing in HDR mode, adaptive brightness adjustment, and ISO adaptive single-frame noise reduction.

[0218] For example, Hamming distance can be used as a similarity metric to match the descriptor of the grayscale image corresponding to the image to be aligned with the descriptor of the grayscale image corresponding to the reference frame image. The nearest neighbor-second nearest neighbor ratio method is used to filter reliable matches and obtain the first set of matching point pairs.

[0219] The descriptions of Hamming distance and the use of the nearest neighbor-second nearest neighbor ratio method to screen reliable matches can be found in the corresponding descriptions in S104, and will not be repeated here.

[0220] S602. Obtain the face region and rotation angle corresponding to the reference frame image, and the face region and rotation angle corresponding to the image to be aligned.

[0221] The rotation angle corresponding to the reference frame image can refer to the device rotation angle during the acquisition of the reference frame image. The rotation angles corresponding to other images are similar and will not be elaborated here.

[0222] For example, assuming the executing entity is a mobile phone, it can receive the face regions and rotation angles corresponding to each image from the mobile phone system.

[0223] S603. Based on the face region and rotation angle corresponding to the reference frame image, determine the first portrait region, which is the portrait region in the grayscale image corresponding to the reference frame image; based on the face region and rotation angle corresponding to the image to be aligned, determine the second portrait region, which is the portrait region in the grayscale image corresponding to the image to be aligned.

[0224] In one possible implementation, a third portrait region can be extended on the reference frame image based on the face region corresponding to the reference frame image; then, based on the rotation angle corresponding to the reference frame image, the third portrait region is rotated, and then the rotated third portrait region is mapped onto the grayscale image corresponding to the reference frame image to obtain the first portrait region on the grayscale image corresponding to the reference frame image.

[0225] The face region corresponding to the reference frame image can be expanded into a third portrait region based on the ratio of shoulder width to head and the ratio of head to torso; this application does not impose any restrictions on this.

[0226] In another possible implementation, the face region corresponding to the reference frame image can be mapped to the grayscale image corresponding to the reference frame image first; then, based on the face region on the grayscale image, a fourth portrait region can be extended on the grayscale image; then, based on the rotation angle corresponding to the reference frame image, the fourth portrait region can be rotated to obtain the first portrait region on the grayscale image.

[0227] The face region on a grayscale image can be expanded into a fourth portrait region based on the ratio of shoulder width to head and the ratio of head to torso; this application does not impose any restrictions on this.

[0228] The method for determining the second portrait region based on the face region and rotation angle corresponding to the image to be aligned can refer to the method for determining the first portrait region described above, and will not be repeated here.

[0229] S604. Traverse the matching point pairs in the first matching point pair set. If any feature point in the matching point pair is located in the first portrait region or the second portrait region, delete the matching point pair to obtain the second matching point pair set.

[0230] That is, the feature points in any matching point pair in the second matching point pair set are not located in the first portrait region and the second portrait region.

[0231] Specifically, in the first set of matching point pairs, one feature point in each pair is a feature point on the grayscale image corresponding to the reference frame image, and the other feature point is a feature point on the grayscale image corresponding to the image to be aligned. If, in this pair, the feature point on the grayscale image corresponding to the reference frame image is located within the first portrait region, and / or the feature point on the grayscale image corresponding to the image to be aligned is located within the second portrait region, then the matching point pair is deleted. That is, in the second set of matching point pairs, the feature point on the grayscale image corresponding to the reference frame image is located outside the first portrait region, and the feature point on the grayscale image corresponding to the image to be aligned is located outside the second portrait region.

[0232] S605. Based on the second set of matching point pairs, align the image to be aligned with the reference frame image.

[0233] For example, the RANSAC algorithm can be used to estimate the homography matrix H: ① Randomly select 4 matching point pairs from the second set of matching point pairs; ② Calculate the homography matrix H; ③ Count the number of inliers (points with reprojection error < preset error threshold); ④ Repeat the iteration, selecting H with the most inliers; ⑥ Finally, recalculate the optimal H using all inliers corresponding to H with the most inliers. Align the image to be aligned with the reference frame image based on the optimal H.

[0234] The relevant explanations of interior points can be found in the corresponding description in S104, and will not be repeated here.

[0235] exist Figure 6 In the illustrated embodiment, before the RANSAC iteration begins, all feature points within the portrait region are actively excluded to ensure that subsequent geometric transformation estimation is not disturbed by foreground motion, i.e., to ensure that slight shaking of the portrait does not interfere with the global alignment of the background region. Prioritizing the global alignment accuracy of the background region, the alignment problem of the portrait region is delegated to the motion detection and compensation mechanism of the subsequent MFNR / HDR module, achieving separation of responsibilities and collaborative optimization among algorithm modules.

[0236] Figure 7 Flowchart of the image alignment method provided in the embodiments of this application Figure 3 ,like Figure 7 As shown, it includes the following steps:

[0237] S701. Divide the grayscale image corresponding to the reference frame image into N×N sub-blocks, where N is a positive integer.

[0238] In this embodiment, the grayscale image corresponding to any image can be the grayscale image obtained after performing S101 to S102, or the grayscale image obtained after performing at least one of the following operations after performing S101 to S102: overexposure area compensation, brightness unification processing in HDR mode, adaptive brightness adjustment, and ISO adaptive single-frame noise reduction.

[0239] For example, N can be 4, that is, the size of each sub-block is (W / 4)×(H / 4), where W is the width of the grayscale image corresponding to the reference frame image, and H is the height of the grayscale image corresponding to the reference frame image.

[0240] S702. Perform multiple iterations based on N×N sub-blocks; wherein each iteration includes: randomly selecting multiple different sub-blocks from the N×N sub-blocks; randomly selecting a feature point in each of the multiple sub-blocks as the first feature point; determining multiple matching point pairs from the second matching point pair set based on the multiple first feature points; and determining the homography matrix and the number of interior points corresponding to the homography matrix based on the multiple matching point pairs.

[0241] The second set of matching point pairs in this embodiment can be Figure 6 The second set of matching point pairs in the illustrated embodiment may be... Figure 6 The first set of matching point pairs in the illustrated embodiment.

[0242] For example, assuming N is 4, each iteration includes: randomly selecting 4 different sub-blocks from 16 sub-blocks (sampling without replacement); then, within each selected sub-block, randomly selecting 1 feature point as the first feature point; and then determining the homography matrix and the number of interior points corresponding to the matching point pairs corresponding to the 4 spatially evenly distributed first feature points.

[0243] S703. Determine the optimal homography matrix based on multiple matching point pairs corresponding to the homography matrix with the most interior points in multiple iterations.

[0244] S704. Based on the optimal homography matrix, align the image to be aligned with the reference frame image.

[0245] For example, the optimal homography matrix can be used to transform the four planes (R plane, G plane, G plane and B plane) of the image to be aligned, and then the four transformed planes can be recombined according to the Bayer array of the image to be aligned to obtain the aligned complete original image.

[0246] Figure 7 The sampling strategy in the illustrated embodiment enforces the uniformity of the spatial distribution of sampling points, avoiding point clustering that may occur in traditional random sampling (such as all sampling points being concentrated in one corner of the image). The homography matrix estimated accordingly can more accurately reflect the global motion of the entire image, rather than the motion of local areas, thereby improving the overall alignment quality.

[0247] Figure 6 The illustrated embodiments and Figure 7 The illustrated embodiment significantly improves alignment robustness while maintaining computational efficiency. It also preserves the real-time performance of the algorithm, fully meeting the performance requirements of mobile phone camera systems.

[0248] Figure 8 This is a schematic diagram of the image alignment device provided in an embodiment of this application. Figure 8 As shown, the device 10 includes a first acquisition module 11, a first determination module 12, a second determination module 13, and an alignment module 14.

[0249] The first acquisition module 11 is used to acquire multiple frames of original images. Each frame of original image adopts the Bayer array arrangement format and includes multiple G channels.

[0250] The first determining module 12 is used to determine the grayscale image corresponding to any one of the multiple frames of original images based on the multiple G channels contained in the original image.

[0251] The second determining module 13 is used to determine the descriptors of feature points in the grayscale image corresponding to the original image;

[0252] Alignment module 14 is used to align multiple original images based on the descriptors of feature points in the grayscale images corresponding to the original images.

[0253] In one possible implementation, the first determining module 12 is specifically used for:

[0254] Obtain pixel values ​​from multiple G channels in the original image;

[0255] Determine the average pixel value of multiple G channels in the original image based on the pixel values ​​of multiple G channels in the original image;

[0256] The grayscale image corresponding to the original image is determined based on the pixel mean of multiple G channels in the original image.

[0257] In one possible implementation, the original image includes multiple color channels, the multiple color channels including multiple G channels; the device 10 further includes:

[0258] The third determining module is used to determine the overexposed areas in the grayscale image corresponding to the original image, where the grayscale value of each pixel in the overexposed area is greater than or equal to the overexposed threshold.

[0259] The fourth determination module is used to determine the pixel values ​​of multiple color channels corresponding to any pixel in the overexposed area.

[0260] The replacement module is used to replace the grayscale value of a pixel with the average value of the pixel values ​​of multiple color channels corresponding to the pixel.

[0261] In one possible implementation, the multiple original images include a reference frame image and at least one frame to be aligned; when the original image is any one of the at least one frames to be aligned, the apparatus 10 further includes:

[0262] The fifth determining module is used to determine the exposure ratio of the image to be aligned, which is the ratio of the exposure time of the reference frame image to the exposure time of the image to be aligned;

[0263] The brightness compensation module is used to perform brightness compensation on the grayscale image corresponding to the image to be aligned based on the exposure ratio of the image to be aligned, so as to obtain a brightness-compensated image.

[0264] The sixth determining module is used to determine the cumulative histogram of the grayscale image corresponding to the reference frame image and the cumulative histogram of the brightness compensation image;

[0265] The seventh determining module is used to determine the mapping function based on the cumulative histogram of the grayscale image corresponding to the reference frame image and the cumulative histogram of the brightness compensation image;

[0266] The brightness correction module is used to perform brightness correction on the brightness-compensated image based on the mapping function.

[0267] In one possible implementation, the device 10 further includes:

[0268] The eighth determining module is used to determine the average total brightness of the grayscale image corresponding to the original image;

[0269] The ninth determination module is used to determine the gamma based on the relationship between the total luminance mean and the mean threshold;

[0270] The brightness adjustment module is used to adjust the brightness of the grayscale image corresponding to the original image based on gamma.

[0271] In one possible implementation, the device 10 further includes:

[0272] The second acquisition module is used to acquire the ISO value of the original image;

[0273] The tenth determination module is used to determine the noise reduction intensity based on the ISO value;

[0274] The noise reduction module is used to perform noise reduction processing on the grayscale image corresponding to the original image based on the noise reduction intensity.

[0275] In one possible implementation, the multiple original images include a reference frame image and at least one image to be aligned; the alignment module 14 is specifically used for:

[0276] For any frame of at least one image to be aligned, based on the descriptors of feature points in the grayscale image corresponding to the image to be aligned and the descriptors of feature points in the grayscale image corresponding to the reference frame image, feature point matching is performed on the grayscale image corresponding to the image to be aligned and the grayscale image corresponding to the reference frame image to obtain the first set of matching point pairs.

[0277] Obtain the face region and rotation angle corresponding to the reference frame image, and the face region and rotation angle corresponding to the image to be aligned;

[0278] Based on the face region and rotation angle corresponding to the reference frame image, a first portrait region is determined, which is the portrait region in the grayscale image corresponding to the reference frame image; based on the face region and rotation angle corresponding to the image to be aligned, a second portrait region is determined, which is the portrait region in the grayscale image corresponding to the image to be aligned.

[0279] Iterate through the matching point pairs in the first matching point pair set. If any feature point in the matching point pair is located in the first portrait region or the second portrait region, delete the matching point pair to obtain the second matching point pair set.

[0280] Based on the second set of matching point pairs, the image to be aligned is aligned with the reference frame image.

[0281] In one possible implementation, the alignment module 14 is specifically used for:

[0282] The grayscale image corresponding to the reference frame image is divided into N×N sub-blocks, where N is a positive integer;

[0283] Perform multiple iterations based on N×N sub-blocks;

[0284] Each iteration includes: randomly selecting multiple different sub-blocks from N×N sub-blocks; randomly selecting a feature point in each of the multiple sub-blocks as the first feature point; determining multiple matching point pairs from the second matching point pair set based on the multiple first feature points; and determining the homography matrix and the number of interior points corresponding to the homography matrix based on the multiple matching point pairs.

[0285] The optimal homography matrix is ​​determined based on multiple matching point pairs corresponding to the homography matrix with the most interior points in multiple iterations.

[0286] Based on the optimal homography matrix, the image to be aligned is aligned with the reference frame image.

[0287] The image alignment device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0288] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 9 As shown, the electronic device 20 includes a transceiver 21, a memory 22, and a processor 23. The transceiver 21 may include a transmitter and / or a receiver. The transmitter may also be referred to as a transmitter, transmitter port, or transmitter interface, etc., and the receiver may also be referred to as a receiver, receiver port, or receiver interface, etc. Exemplarily, the transceiver 21, memory 22, and processor 23 are interconnected via a bus 24.

[0289] Memory 22 is used to store program instructions;

[0290] The processor 23 is used to execute the program instructions stored in the memory to cause the electronic device to perform any of the image alignment methods shown above.

[0291] Transceiver 21 is used to perform the sending and receiving functions of electronic devices.

[0292] In one possible implementation, the memory 22 may be the storage medium described above.

[0293] Electronic devices can include chips, modules, integrated development environments (IDEs), etc.

[0294] Figure 9 The electronic device shown in the embodiments can execute the technical solutions shown in the above method embodiments. Its implementation principle and beneficial effects are similar, and will not be repeated here.

[0295] This application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the image alignment method described above.

[0296] This application provides a computer program product, including a computer program that, when executed by a processor, can implement any of the above-mentioned image alignment methods.

[0297] This application provides a chip on which a computer program is stored. When the computer program is executed by the chip, it implements the above-described image alignment method.

[0298] In one possible implementation, the chip is a chip in a chip module.

[0299] The computer-readable storage medium and computer program product of the present application embodiments can execute the technical solutions shown in the above-described image alignment method embodiments. Their implementation principles and beneficial effects are similar and will not be repeated here.

[0300] All or part of the steps in the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a readable memory. When the program is executed, it performs the steps of the above-described method embodiments; and the aforementioned memory (storage medium) includes: read-only memory (ROM), random access memory (RAM), flash memory, hard disk, solid-state drive, magnetic tape, floppy disk, optical disc, and any combination thereof.

[0301] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processing unit of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processing unit of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0302] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0303] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0304] Obviously, those skilled in the art can make various modifications and variations to the embodiments of this application without departing from the spirit and scope of this application. Therefore, if these modifications and variations to the embodiments of this application fall within the scope of the claims of this application and their equivalents, this application also intends to include these modifications and variations.

Claims

1. An image alignment method, characterized in that, include: Acquire multiple frames of raw images, each frame using a Bayer array arrangement format, and each frame includes multiple green G channels; For any one of the multiple original images, a grayscale image corresponding to the original image is determined based on the multiple G channels contained in the original image. Determine the descriptors of feature points in the grayscale image corresponding to the original image; The original images are aligned based on the descriptors of feature points in the grayscale images corresponding to the original images.

2. The method according to claim 1, characterized in that, Based on the multiple G channels contained in the original image, determine the grayscale image corresponding to the original image, including: Obtain the pixel values ​​of multiple G channels in the original image; Based on the pixel values ​​of multiple G channels in the original image, determine the average pixel value of multiple G channels in the original image; Based on the pixel mean of multiple G channels in the original image, the grayscale image corresponding to the original image is determined.

3. The method according to claim 1 or 2, characterized in that, The original image includes multiple color channels, including the multiple G channels; before determining the descriptors of feature points in the grayscale image corresponding to the original image, the method further includes: Determine the overexposed region in the grayscale image corresponding to the original image, wherein the grayscale value of each pixel in the overexposed region is greater than or equal to the overexposed threshold; For any pixel in the overexposed area, determine the pixel values ​​of multiple color channels corresponding to that pixel. Replace the grayscale value of the pixel with the average of the pixel values ​​of the multiple color channels corresponding to the pixel.

4. The method according to any one of claims 1-3, characterized in that, The multiple original images include a reference frame image and at least one frame to be aligned; when the original image is any one of the at least one frames to be aligned, before determining the descriptor of the feature points in the grayscale image corresponding to the original image, the method further includes: Determine the exposure ratio of the image to be aligned, wherein the exposure ratio is the ratio of the exposure time of the reference frame image to the exposure time of the image to be aligned; Based on the exposure ratio of the image to be aligned, brightness compensation is performed on the grayscale image corresponding to the image to be aligned to obtain a brightness-compensated image. Determine the cumulative histogram of the grayscale image corresponding to the reference frame image and the cumulative histogram of the brightness-compensated image; The mapping function is determined based on the cumulative histogram of the grayscale image corresponding to the reference frame image and the cumulative histogram of the brightness compensation image; The brightness of the brightness-compensated image is corrected based on the mapping function.

5. The method according to any one of claims 1-4, characterized in that, Before determining the descriptors of feature points in the grayscale image corresponding to the original image, the method further includes: Determine the average total brightness of the grayscale image corresponding to the original image; The gamma is determined based on the relationship between the average total luminance and the average threshold. Based on the gamma, adjust the brightness of the grayscale image corresponding to the original image.

6. The method according to any one of claims 1-5, characterized in that, Before determining the descriptors of feature points in the grayscale image corresponding to the original image, the method further includes: Obtain the ISO value of the original image; Based on the ISO value, the noise reduction intensity is determined; Based on the noise reduction intensity, the grayscale image corresponding to the original image is subjected to noise reduction processing.

7. The method according to any one of claims 1-6, characterized in that, The multi-frame original images include a reference frame image and at least one frame to be aligned; the alignment of the multi-frame original images based on the descriptors of feature points in the grayscale images corresponding to the multi-frame original images includes: For any one of the at least one frames of images to be aligned, based on the descriptors of feature points in the grayscale image corresponding to the image to be aligned and the descriptors of feature points in the grayscale image corresponding to the reference frame image, feature point matching is performed on the grayscale image corresponding to the image to be aligned and the grayscale image corresponding to the reference frame image to obtain a first set of matching point pairs. Obtain the face region and rotation angle corresponding to the reference frame image, and the face region and rotation angle corresponding to the image to be aligned; Based on the face region and rotation angle corresponding to the reference frame image, a first portrait region is determined, which is the portrait region in the grayscale image corresponding to the reference frame image; based on the face region and rotation angle corresponding to the image to be aligned, a second portrait region is determined, which is the portrait region in the grayscale image corresponding to the image to be aligned. Traverse the matching point pairs in the first matching point pair set. If any feature point in the matching point pair is located in the first portrait region or the second portrait region, delete the matching point pair to obtain the second matching point pair set. Based on the second set of matching point pairs, the image to be aligned is aligned with the reference frame image.

8. The method according to claim 7, characterized in that, The step of aligning the image to be aligned with the reference frame image based on the second set of matching point pairs includes: The grayscale image corresponding to the reference frame image is divided into N×N sub-blocks, where N is a positive integer; Perform multiple iterations based on the aforementioned N×N sub-blocks; Each iteration includes: randomly selecting multiple different sub-blocks from the N×N sub-blocks; randomly selecting a feature point from each of the multiple sub-blocks as a first feature point; determining multiple matching point pairs from the second matching point pair set based on the multiple first feature points; and determining the homography matrix and the number of interior points corresponding to the homography matrix based on the multiple matching point pairs. The optimal homography matrix is ​​determined based on multiple matching point pairs corresponding to the homography matrix with the most interior points in the multiple iterations. Based on the optimal homography matrix, the image to be aligned is aligned with the reference frame image.

9. An electronic device, characterized in that, include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-8.

Images