Image processing method, electronic device, and computer-readable storage medium

Abstract

The present application discloses an image processing method, an electronic device and a computer-readable storage medium, which relate to the field of electronic technology. The method includes: obtaining an image to be registered and a reference image; determining a plurality of feature point pairs in the first region of each of the reference image and the image to be registered, wherein the variation pattern of the pixel points in the first region of the image to be registered is the same as the variation pattern of the pixel points in the first region of the reference image, and the plurality of feature point pairs include a plurality of first feature points located in the first region of the reference image, and each second feature point located in the first region of the image to be registered and corresponding to each first feature point; determining the transformation relationship between the image to be registered and the reference image based on the plurality of feature point pairs; and registering the image to be registered and the reference image based on the transformation relationship. Based on the above technical solution, the electronic device screens the feature points, avoids the situation where the feature point pairs are not accurately matched, can improve the calculation accuracy of the registration matrix, and thus improve the registration accuracy between images, reduce the blurring phenomenon of the local motion area in the image after registration, improve the clarity of the image texture details, and improve the image quality of the output image.

Description

Technical Field

[0001] The present application relates to the field of electronic technology, and in particular to an image processing method, an electronic device, and a computer-readable storage medium. Background Art

[0002] Image registration involves mapping two images from a dataset captured under different shooting conditions (e.g., different shooting times, different shooting angles, or using different sensors) using a spatial transformation relationship. This maps one image (hereinafter referred to as the image to be registered) to another image (hereinafter referred to as the reference image) so that pixels corresponding to the same spatial location in the two images are aligned one-to-one, thereby achieving information fusion. Currently, when registering multiple frames, a series of feature points are typically determined in the reference image. Points matching each feature point are then searched for in the image to be registered, resulting in feature point pairs. A registration matrix is ​​then calculated based on these feature point pairs to align the reference image with the image to be registered.

[0003] When an image has rich textures and local motion, there are some differences between the reference image and the image to be registered. Using the current registration scheme may result in inaccurate feature point pair matching. For example, when shooting grass and leaves in an outdoor scene, a reference image and an image to be registered are obtained during the shooting. Because the grass and leaves sway in the wind or there is jitter during the shooting, there are differences between the reference image and the image to be registered. At this time, the feature point pairs obtained may have feature points and matching points corresponding to different positions in the same space. As a result, the registration matrix calculated based on the feature point pairs has low accuracy, and there are errors in the registration between images. As a result, the fused image after registration has problems such as blurring of local motion areas and loss of texture details, resulting in poor image quality. Summary of the Invention

[0004] This application provides an image processing method, electronic device, and computer-readable storage medium that can improve the accuracy of the registration matrix, enhance the clarity of image texture details, and improve the image quality of the output image. The technical solution is as follows:

[0005] In a first aspect, an embodiment of the present application provides an image processing method, applied to an electronic device, the method comprising:

[0006] Acquire an image to be registered and a reference image, where the reference image and the image to be registered are obtained by photographing the same object;

[0007] Determining, in a first region of each of the reference image and the image to be registered, a plurality of feature point pairs, wherein a variation pattern of pixels in the first region of the image to be registered is identical to a variation pattern of pixels in the first region of the reference image, the plurality of feature point pairs comprising a plurality of first feature points located in the first region of the reference image, and second feature points located in the first region of the image to be registered and corresponding to the first feature points;

[0008] Determine the transformation relationship between the image to be registered and the reference image based on multiple feature point pairs;

[0009] According to the transformation relationship, the image to be registered and the reference image are registered.

[0010] Based on the above technical solution, after acquiring the reference image and the image to be registered, the electronic device determines the first area of ​​each of the reference image and the image to be registered so that multiple feature point pairs are located in the same area between the reference image and the image to be registered, and are not located in the area where there is a difference between the image to be registered and the reference image. That is, the feature point pairs located in the area where there is a difference between the image to be registered and the reference image are screened out, avoiding the situation of inaccurate matching of feature point pairs, and then determining the transformation relationship (that is, the registration matrix) between the image to be registered and the reference image based on multiple feature point pairs, which can improve the calculation accuracy of the registration matrix, thereby improving the registration accuracy between images, reducing the blurring phenomenon of local motion areas in the image after registration, improving the clarity of image texture details, and improving the image quality of the output image.

[0011] In conjunction with the first aspect, in certain implementations of the first aspect, the method further includes: a first region in the reference image includes a second region, the second region including at least one texture unit; and determining multiple feature point pairs in the first region of each of the reference image and the image to be registered, including: determining multiple feature point pairs in the second region of the reference image and the first region of the image to be registered, the multiple feature point pairs including multiple first feature points located in the second region of the reference image and each second feature point corresponding to each first feature point in the image to be registered. In this way, while filtering out feature point pairs in regions with differences between the images, further selecting feature point pairs located in regions with rich texture, can improve the clarity of image texture details while improving the calculation accuracy of the registration matrix.

[0012] In combination with the first aspect and the above-mentioned implementations, in certain implementations of the first aspect, determining multiple feature point pairs in the second area of ​​the reference image and the first area of ​​the image to be registered includes: determining multiple third feature points from the reference image and determining each fourth feature point corresponding to each third feature point from the image to be registered; determining the first area of ​​each of the reference image and the image to be registered, and determining the second area of ​​the reference image; and determining multiple feature point pairs based on the third feature point located in the second area of ​​the reference image and the fourth feature point located in the first area of ​​the image to be registered, wherein the multiple first feature points include the third feature point located in the second area of ​​the reference image, and each second feature point includes each fourth feature point located in the first area of ​​the image to be registered. In this way, the feature point pairs can be determined first and then the feature point pairs can be screened.

[0013] In combination with the first aspect and the above-described implementations, in certain implementations of the first aspect, determining multiple feature point pairs in a second region of the reference image and a first region of the image to be registered includes: determining the first region of each of the reference image and the image to be registered, and determining the second region of the reference image; determining multiple first feature points in the second region of the reference image, and determining second feature points corresponding to each first feature point in the first region of the image to be registered, to obtain multiple feature point pairs. This can narrow the region for searching feature points, thereby reducing the time required to search for feature points and improving registration speed.

[0014] In combination with the first aspect and the above-described implementations, in certain implementations of the first aspect, determining the first region of each of the reference image and the image to be registered includes: determining the local standard deviation of each pixel in the reference image and the local standard deviation of each pixel in the image to be registered; determining the relative difference value of each pixel in the image to be registered relative to each pixel in the reference image based on the local standard deviation of each pixel in the reference image and the local standard deviation of each pixel in the image to be registered; and determining the first region of each of the reference image and the image to be registered based on the relative difference value of each pixel in the image to be registered relative to each pixel in the reference image. The local standard deviation can reflect the change of pixels in the reference image or the image to be registered. The difference between the local standard deviations can be used to measure whether the changes in the local regions containing the same pixels in the two images are the same, thereby determining the first region.

[0015] In combination with the first aspect and the above-mentioned implementation methods, in certain implementation methods of the first aspect, determining the first regions of the reference image and the image to be registered based on the relative difference value of each pixel in the image to be registered relative to each pixel in the reference image includes: determining whether the relative difference value corresponding to each pixel is less than or equal to a first threshold; and determining the region where the pixels whose relative difference value is less than or equal to the first threshold are located as the first regions corresponding to the reference image and the image to be registered. Among them, the region where the pixels whose relative difference value is greater than the first threshold are located is the region where there is a difference between the image to be registered and the reference image. Furthermore, determining whether multiple third feature points are located in the first region includes: determining whether the relative difference value corresponding to the third feature point is less than or equal to the first threshold. If the relative difference value corresponding to the third feature point is less than or equal to the first threshold, the third feature point is located in the first region; otherwise, the third feature point is located in the region where there is a difference.

[0016] In combination with the first aspect and the above-mentioned implementations, in certain implementations of the first aspect, determining the first regions of the reference image and the image to be registered based on the relative difference value of each pixel in the image to be registered relative to each pixel in the reference image includes: setting the values ​​of pixels whose relative difference values ​​are less than or equal to a first threshold to 0, and setting the values ​​of pixels whose relative difference values ​​are greater than the first threshold to 1, to obtain a binary texture difference; performing a morphological opening and closing operation on the binary texture difference map; and determining the regions in the binary texture difference map after the morphological opening and closing operation where the pixel values ​​are 0 as the first regions corresponding to the reference image and the image to be registered. Furthermore, determining whether multiple third feature points are located in the first region includes: determining whether the corresponding values ​​of the third feature points are 0; if the corresponding values ​​of the third feature points are 0, the third feature points are located in the first region; and if the corresponding values ​​of the third feature points are 1, the third feature points are located in the region where there is a difference.

[0017] In conjunction with the first aspect and the above implementations, in certain implementations of the first aspect, the method further includes: determining coordinates of multiple third feature points in a reference image; and determining whether the multiple third feature points are located in the first region based on the coordinates of the multiple third feature points in the reference image. Here, whether the multiple pixels included in the first region are the third feature points can be determined based on the coordinates; if the third feature points are included, the third feature points are located in the first region; if the third feature points are not included, the third feature points are located in a different region.

[0018] In combination with the first aspect and the above-mentioned implementation methods, in certain implementation methods of the first aspect, determining the second area of ​​the reference image includes: determining the local standard deviation corresponding to each pixel point in the reference image; and determining the area in the first area of ​​the reference image where the pixel points whose local standard deviation is greater than or equal to the second threshold are located as the second area.

[0019] In combination with the first aspect and the above implementations, in some implementations of the first aspect, determining multiple first feature points includes: determining multiple first feature points in a reference image according to a Harris corner algorithm.

[0020] In combination with the first aspect and the above implementations, in some implementations of the first aspect, the transformation relationship is an affine transformation.

[0021] In combination with the first aspect and the above-mentioned implementation methods, in some implementation methods of the first aspect, obtaining the image to be registered and the reference image includes: starting a camera application in an electronic device; and obtaining multiple frames of images in response to a user's shooting instruction, wherein the multiple frames of images include the reference image and at least one frame of the image to be registered.

[0022] In a second aspect, an embodiment of the present application provides an electronic device comprising: one or more processors; one or more memories; the memories storing one or more programs, which, when executed by the processor, enables the electronic device to execute the method described in any one of the above-mentioned first aspect and the first aspect.

[0023] In a third aspect, an embodiment of the present application provides a device that is included in an electronic device and has the function of implementing the electronic device behavior in the above aspects and possible implementations of the above aspects. The functions can be implemented by hardware or by hardware executing corresponding software implementations. The hardware or software includes one or more modules or units corresponding to the above functions. For example, a display module or unit, a detection module or unit, a processing module or unit, etc.

[0024] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein instructions are stored in the computer-readable storage medium, which, when executed on a computer, enables the computer to execute the method described in any one of the first and second aspects above.

[0025] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it is used to execute the method described in any one of the first aspect and the first aspect.

[0026] In the sixth aspect, an embodiment of the present application provides a chip system, which includes one or more interface circuits and one or more processors; the interface circuits and the processors are interconnected through lines; and the processors are used to execute the method described in any one of the first aspect and the first aspect.

[0027] The technical effects obtained by the above-mentioned second, third, fourth, fifth and sixth aspects are similar to the technical effects obtained by the corresponding technical means in the above-mentioned first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 A schematic diagram of an application scenario provided by an embodiment of the present application is shown;

[0029] Figure 2 A schematic diagram illustrating a process of an image processing method provided in an embodiment of the present application is shown;

[0030] Figure 3 A schematic diagram of a feature point extraction principle provided by an embodiment of the present application is shown;

[0031] Figure 4 A schematic diagram of a third feature point in a reference image provided by an embodiment of the present application is shown;

[0032] Figure 5 A schematic diagram of a binarized image provided by an embodiment of the present application is shown;

[0033] Figure 6 A schematic diagram of another binary image provided by an embodiment of the present application is shown;

[0034] Figure 7 A schematic diagram of a first feature point located in a first area of ​​a reference image provided by an embodiment of the present application is shown;

[0035] Figure 8 A schematic diagram of a feature point matching result provided by an embodiment of the present application is shown;

[0036] Figure 9 A schematic diagram illustrating a process of another image processing method provided in an embodiment of the present application is shown;

[0037] Figure 10 A schematic diagram of a first feature point located in a second area of ​​a reference image provided by an embodiment of the present application is shown;

[0038] Figure 11 A schematic structural diagram of an electronic device 100 provided in an embodiment of the present application is shown;

[0039] Figure 12 A schematic diagram of the software structure of an electronic device provided in an embodiment of the present application is shown;

[0040] Figure 13 A schematic diagram of a device provided in an embodiment of the present application is shown;

[0041] Figure 14 A schematic diagram of a chip structure provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solutions and advantages of the present application clearer, the embodiments of the present application will be further described in detail below with reference to the accompanying drawings. In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of this embodiment, unless otherwise specified, "multiple" means two or more.

[0043] Image registration is the process of aligning and matching multiple frames of images of the same scene captured under different shooting conditions (such as different shooting times, different shooting angles, or using different sensors). Specifically, for two images in a dataset, a spatial transformation is found to map one image (hereinafter referred to as the image to be registered) to the other image (hereinafter referred to as the reference image, also known as the benchmark image) so that pixels corresponding to the same spatial location in the two images are aligned one-to-one, thereby achieving the goal of information fusion.

[0044] Currently, when registering multiple frames of images, a series of feature points are usually determined in the reference image. Based on the local template of each feature point in the reference image, points matching each feature point are found in the image to be registered. Through similarity comparison, the point in the image to be registered that is most similar to the feature point is found as the corresponding matching point to obtain feature point pairs. Based on each feature point pair, the registration matrix is ​​calculated and spatial transformation is performed to achieve alignment between the reference image and the image to be registered.

[0045] Current registration schemes only consider corner points, not image content. When images have rich textures and local motion, there are texture differences between the reference image and the image to be registered, leading to inaccurate feature point pair matching. For example, when shooting grass and leaves in an outdoor scene (i.e., rich textures), the reference image and the image to be registered are acquired during the shooting. The grass and leaves sway in the wind or there is jitter during the shooting, resulting in texture differences between the reference image and the image to be registered. However, the pixel values ​​of the differences are relatively close. In this case, the feature point pairs obtained may have feature points and matching points corresponding to different locations in the same space, resulting in inaccurate feature point pair matching. Consequently, the registration matrix calculated based on the feature point pairs has low accuracy, resulting in errors in the inter-image registration. Consequently, the fused image after registration suffers from blurring of local motion regions and loss of texture details, resulting in poor image quality.

[0046] In view of this, an embodiment of the present application provides an image processing method, which, after acquiring a reference image and an image to be registered, determines multiple feature point pairs in the first area of ​​each of the reference image and the image to be registered, wherein the multiple feature point pairs include multiple first feature points located in the first area of ​​the reference image, and second feature points corresponding to the first feature points located in the first area of ​​the image to be registered; based on the multiple feature point pairs, determines the transformation relationship between the image to be registered and the reference image; and based on the transformation relationship, aligns the image to be registered and the reference image. Among them, the change pattern of the pixels in the first area of ​​the image to be registered is no different from the change pattern of the pixels in the first area of ​​the reference image. That is to say, the present application determines the first area of ​​the reference image and the image to be registered so that multiple feature point pairs are located in the same area between the reference image and the image to be registered, and are not located in the area where there is a difference between the image to be registered and the reference image. The feature point pairs in the difference area between the images can be screened out, thereby avoiding inaccurate matching of feature point pairs. Then, based on multiple feature point pairs, the transformation relationship between the image to be registered and the reference image (i.e., the registration matrix) is determined, which can improve the calculation accuracy of the registration matrix, thereby improving the registration accuracy between images, reducing the blurring phenomenon of local motion areas in the image after registration, improving the clarity of image texture details, and improving the image quality of the output image.

[0047] The image processing method provided in the embodiments of the present application can be applied to various electronic devices. The electronic devices may be, but are not limited to, mobile phones, action cameras (GoPro), digital cameras, tablet computers, desktop computers, laptop computers, handheld computers, notebook computers, vehicle-mounted devices, ultra-mobile personal computers (UMPCs), netbooks, cellular phones, personal digital assistants (PDAs), augmented reality (AR) and virtual reality (VR) devices, mobile phones, etc., and the embodiments of the present application are not limited to these.

[0048] The image processing method provided in the embodiments of the present application can be applied to a variety of application scenarios. For example, when used in camera photography or video shooting, when multiple frames of images need to be fused in some shooting modes (such as night scene shooting), image registration can be performed before fusion. After accurate image registration, better fusion effect can usually be obtained. For example, in medicine, the multimodal data of patients can be fused into the same coordinate system to assist diagnosis. For example, it can be applied to 3D reconstruction, image alignment, motion tracking, etc.

[0049] For example, Figure 1 A schematic diagram of a scene in which the image processing method of the present application is applied to a camera taking a photo is shown. Figure 1 As shown, the texture of the subject A is relatively rich, and the image obtained when photographing the subject A is easily blurred; Figure 1 (a) is an image of the object A obtained by image processing using a method provided by conventional technology; Figure 1 (b) is an image of the same photographed object A obtained by the image processing method provided in an embodiment of the present application; Figure 1 The image shown in (b) is the same as Figure 1 Compared with the image shown in (a), the texture in the framed area is clearer; it can be seen that the image processing method of the embodiment of the present application can enhance the clarity of image texture details and improve image quality.

[0050] The following combination Figures 2 to 10 The image processing method provided in the embodiments of the present application is described in detail.

[0051] Figure 2 This is a flow chart of an image processing method provided in an embodiment of the present application. The method is executed by an electronic device, or by a chip configured in an electronic device. Figure 2 The image processing method shown includes the following steps S201 to S207.

[0052] S201: Acquire the image to be registered and the reference image.

[0053] The electronic device captures multiple frames of images of the same subject and selects one image from the multiple frames as the reference image. The remaining images in the multiple frames are the images to be registered. The electronic device transforms the image to be registered based on the reference image, thereby achieving multi-frame image registration.

[0054] An electronic device can capture multiple images in real time. For example, an electronic device has a camera and a camera application. In a photo-taking scenario, a user can open the camera application in the electronic device, select a shooting mode, and click the capture button to take a photo. In response to the user's click operation, the electronic device continuously captures multiple frames of images of the subject and aligns the multiple frames. The multiple frames of images can be images of the subject captured by the electronic device at different time points in the same scene. The reference image can be the first frame of the multiple frames with the earliest acquisition time point, or the reference image can be the frame with the highest image quality among the multiple frames.

[0055] The electronic device may also obtain multiple images from a storage medium, which may be a memory of the electronic device itself, a mobile hard disk, etc. The electronic device may also download multiple images from the Internet.

[0056] After determining the reference image and the image to be registered, the electronic device can extract multiple third feature points from the reference image and multiple fourth feature points from the image to be registered; and determine the difference areas and the same areas between the image to be registered and the reference image, and match the first feature point located in the same area among the multiple third feature points with the second feature point located in the same area among the multiple fourth feature points to obtain a feature point pair, and then perform subsequent operations on the matched feature point pair.

[0057] S202 : extracting a plurality of third feature points from the reference image, and extracting a plurality of fourth feature points from the image to be registered.

[0058] A feature refers to the ability to represent a scene or target in a representative form in multiple similar images containing the same scene or target. The above-mentioned feature points mainly refer to obvious points in the image, such as prominent corner points, edge endpoints, extreme points, etc. In the embodiment of the present application, corner points are selected as feature points. In real life, corner points correspond to the corners of tables, buildings, etc., crossroads of roads, T-junctions, etc. In the field of image processing, corner points have the following two definitions: (1) Corner points can be the intersection of two or more edges, such as the pixel points corresponding to the corners of an object; (2) Corner points can be the pixel points in the image where the first-order derivative is the largest and the second-order derivative is zero, indicating the place where the edge of the object changes discontinuously. While retaining the important features of the image graphics, corner points can effectively reduce the amount of information data, effectively improve the speed of calculation, and facilitate reliable image matching.

[0059] The electronic device can use an algorithm to extract feature points. Specifically, the electronic device can use different algorithms corresponding to different types of feature points to extract feature points containing corresponding characteristics. In the embodiment of the present application, corner points are selected as feature points, and a corner algorithm can be used to extract feature points, wherein the corner algorithm includes: Harris corner algorithm, accelerated segment test (Feature from Accelerated Segment Test, FAST) corner algorithm, scale-invariant feature transform (SIFT) corner algorithm, or accelerated robust features (SURF) algorithm, etc.

[0060] The following describes the process of extracting feature points in detail using the Harris corner algorithm as an example. Figure 3 A schematic diagram of a set of corner feature extraction methods provided in this application is exemplified.

[0061] Since the average brightness changes dramatically when a corner moves in any direction, the basic idea of ​​the Harris corner detection algorithm is to use a fixed-size window (i.e., a neighborhood window of a pixel) to slide in any direction on the image and compare the grayscale changes of the pixels framed by the window before and after the sliding. If the grayscale of the pixels within the window does not change when the window moves in any direction, then there is no corner in the window. If the grayscale of the pixels framed by the window changes significantly when the window slides in any direction, then it is considered that there is a corner in the window.

[0062] refer to Figure 3 , Figure 3 (a) in the figure only illustrates a small area in the reference image and a window used to extract corner points in the reference image. The window is initialized at, for example, the center of the reference image. The window is then slid in any direction across the reference image based on this initial position, and the grayscale change of the pixels framed by the window before and after the sliding is compared. This process continues until all pixels in the reference image are traversed, resulting in all corner points of the reference image.

[0063] like Figure 3 As shown in (b) in the figure, assuming that the window slides a certain number of pixels from the current position (solid line position in the figure) to any surrounding direction (dashed line position in the figure), and the grayscale of the pixels in the window does not change, it is considered that there are no corner points in all the pixels at the solid line position of the window frame.

[0064] like Figure 3 As shown in (c) in the figure, assuming that the window slides a certain number of pixels from the current position (solid line position in the figure) to any surrounding direction (dashed line position in the figure), the grayscale of the pixels in the window changes in multiple directions, then it is considered that there are corner points in all the pixels at the solid line position of the window frame.

[0065] based on Figure 3 The specific implementation process of the Harris corner algorithm introduced above can obtain all corner points contained in the reference image and the image to be registered, that is, extract multiple third feature points from the reference image and multiple fourth feature points from the image to be registered. Specifically, the electronic device can obtain the position coordinates of each third feature point in the reference image, recorded as (x, y); the electronic device can also obtain the position coordinates of each fourth feature point in the image to be registered, recorded as (x', y').

[0066] For example, Figure 4 A plurality of third feature points in the reference image are shown. Figure 4As shown, the third feature points included in the reference image include: third feature point 1, third feature point 2, third feature point 3, third feature point 4, third feature point 5, and third feature point 6. Moreover, third feature points 1 to 6 may be located in different areas of the reference image or in the same area of ​​the reference image. Third feature points 1 to 6 may be located in texture areas of the reference image or in non-texture areas of the reference image.

[0067] It is understandable that Figure 4 Only some of the third feature points included in the reference image are shown as examples. The reference image may include more third feature points. Figure 4 It should not constitute a limitation to the present application. Similarly, the fourth feature point extracted from the image to be registered includes the fourth feature point 1 to the fourth feature point 6. Figure 4 Similar, omitted here.

[0068] S203: Determine the difference area and the same area between the image to be registered and the reference image.

[0069] Due to the movement of the photographed object, shaking during shooting and other factors, there are differences between the image to be registered and the reference image, for example, the texture area in the two images changes. In an embodiment of the present application, the difference area in the two images is first detected based on the image content, and then the feature points in the difference area are eliminated, that is, the feature points are filtered. Among them, the same area means that there is no difference between the change pattern of the pixel points in the image to be registered in this area and the change pattern of the pixel points in the reference image in this area, and the change of the pixel points includes the change of the grayscale value of the pixel points. No difference means that the similarity is greater than a preset threshold, for example, the similarity is greater than 98%. The same area corresponds to the aforementioned first area. Correspondingly, the difference area means that there is a difference between the change pattern of the pixel points in the image to be registered in this area and the change pattern of the pixel points in the reference image in this area.

[0070] When detecting different areas in two images, a quantitative index is required to measure the difference between the two. In the embodiment of the present application, relative texture difference is used to represent the quantitative index. Step S203 may specifically include S2031 to S2032. S2031 to S2032 are described in detail below.

[0071] S2031, calculate texture complexity.

[0072] The electronic device first calculates the standard deviation of the grayscale values ​​of pixels in the local area img of the image, as shown in the following formula (1).

[0073]

[0074] Where σ can be either X or Y. X(i,j) represents the local standard deviation at position (i,j) in the reference image, and Y(i,j) represents the local standard deviation at position (i,j) in the image to be registered. The local window is a window centered at position (i,j). n represents the size of the local window. For example, n=11 represents an 11×11 window. The local standard deviation is calculated using the grayscale values ​​of the pixels within the window. sd(.) represents the standard deviation operator. The local standard deviation at position (i,j) in the image can be used to represent the texture complexity of that position.

[0075] The local standard deviation of each pixel in the reference image and the local standard deviation of each pixel in the image to be registered can be obtained by formula (1), that is, the texture complexity of the reference image and the texture complexity of the image to be registered can be obtained.

[0076] In other implementations, the texture complexity of each pixel in the reference image and the texture complexity of each pixel in the image to be registered can also be calculated using the global standard deviation. For the reference image, although the texture complexity of each pixel calculated using the global standard deviation and the texture complexity of each pixel calculated using the local standard deviation may differ, the relativity between the texture complexities of multiple pixels is consistent.

[0077] S2032: Calculate relative texture differences.

[0078] Then, the difference between the two local standard deviations at position (i, j) in the reference image and the image to be registered is calculated as shown in Equation (2). The difference between the standard deviations can measure the texture difference d between the image to be registered and the reference image. A larger texture difference d often means that there is a difference between the two images. Equation (2) can be used to obtain the difference between the pixels at the same position in the reference image and the image to be registered.

[0079] d(X,Y)=(XY) 2 (2)

[0080] The texture difference d represents the absolute difference between the local regions of the two images, which is strongly related to the texture complexity of the image itself. When faced with similar d, blocks with different semantics have completely different visual perceptions. To more accurately measure the difference between the image to be registered and the reference image, the electronic device further calculates the relative texture difference D, as shown in Equation (3). The relative texture difference D is a relative indicator independent of the original texture changes and can reflect the size of the signal difference between the images.

[0081]

[0082] In some implementations, the relative texture difference D may be normalized to [0, 1] as shown in Equation (4).

[0083]

[0084] Here, X is X(i, j), Y is Y(i, j), and C is a constant. Introducing the constant C can stabilize the division operation, handle the case where the denominator is relatively small, and increase the robustness of the calculation result.

[0085] The texture difference map is obtained by calculating the relative texture difference D of each pixel. The texture difference map can be used to determine the same area and the different area between the reference image and the image to be registered. Specifically, the grayscale value (D value) corresponding to each pixel in the texture difference map obtained by the relative texture difference D is between 0 and 1 (calculated by formula (4)), or the corresponding D value is between 0 and 255 (calculated by formula (3)). The area where the pixel with a D value of 0 in the texture difference map is located is the same area between the reference image and the image to be registered, and the area where the pixel with a D value not equal to 0 in the texture difference map is located is the different area between the reference image and the image to be registered. The D value corresponds to the aforementioned relative difference value.

[0086] Optionally, step S203 may further include: S2033, binarizing the texture difference map.

[0087] Different semantic regions have different tolerances for texture differences. In some scenarios, some pixels in the texture difference map with non-zero D values ​​may also belong to the same region between the reference image and the image to be registered. Electronic devices can binarize the texture difference map using threshold segmentation and use the binarized difference map to determine the areas of commonality and difference between the reference image and the image to be registered. Binarization specifically involves converting the texture difference map into a binary image with pixel values ​​of only 0 and 1, or into a binary image with pixel values ​​of only 0 and 255.

[0088] Taking a binary image of 0 and 1 as an example, the main method of binarization is to first obtain a first threshold, and then adjust the pixel grayscale values ​​greater than the first threshold to 1 in the pixel grayscale value matrix corresponding to the texture difference map, and adjust the pixel grayscale values ​​less than or equal to the first threshold to 0. The first threshold can be determined based on the value of each pixel in the texture difference map. For example, the D value of each pixel is arranged in descending order, and the D value at the 85% ratio is set as the first threshold. For another example, the D value of each pixel is arranged in descending order, and the D value at the median is set as the first threshold. For another example, the average value of the D value of each pixel is calculated, and the average value is set as the first threshold.

[0089] Figure 5The figure shows the binary difference map obtained based on the reference image and the image to be registered, where the black area is the area with a pixel value of 0, and the white area is the area with a pixel value of 1. That is, the black area is the same area between the reference image and the image to be registered, and the white area is the difference area between the reference image and the image to be registered.

[0090] Optionally, step S203 may further include: S2034, performing a morphological operation on the binarized difference map.

[0091] Due to the positional differences between the reference image and the image to be registered, there are some discrete points in the binary difference map. The electronic device can also perform opening and closing operations in morphological operations on the existing binary difference map to remove or connect discrete points, so that the binary difference map is converted into a more precise and accurate binary image with better connectivity, which is beneficial to improving the accuracy of the registration matrix. For example, see Figure 6 , Figure 6 (b) shows the binary image after morphological operation, compared with Figure 6 In the binary difference map before processing shown in (a), it can be clearly seen that some disconnected areas are connected (as shown in the boxed area 61) and some discrete points are removed (as shown in the boxed area 62).

[0092] The morphological operation processing steps described above can be as follows: first, an erosion operation is performed using a 5×5 matrix filled with all ones; then, a dilation operation is performed using a 5×5 matrix filled with all ones to connect discontinuous areas; then, a dilation operation is performed using a 3×3 matrix filled with all ones to fill holes; and finally, smaller discrete areas are filtered out. In other implementations, different morphological operation processing steps can be used, depending on the actual image content.

[0093] Through step S203, the reference image can be divided into two parts: the same area and the different area, and the image to be registered can also be divided into two parts: the same area and the different area, to further determine whether each third feature point and each fourth feature point are located in the same area.

[0094] S204 , determining whether the third feature points are located in the same area of ​​the reference image, and whether the fourth feature points are located in the same area of ​​the image to be registered.

[0095] When detecting feature points, the position coordinates of the feature points are determined. The reference image, the image to be registered, and the binary image all have dimensions of m×m. In one implementation, the position coordinates of the feature points can be mapped to a binary difference map / binary image. Based on the extent of the black area in the binary difference map / binary image, it is determined whether each third feature point is located in the same area of ​​the reference image and whether each fourth feature point is located in the same area of ​​the image to be registered. If the third feature point is located in the black area, it indicates that the third feature point is located in the same area of ​​the reference image, and the third feature point located in the same area is retained. If the third feature point is located in the white area, it indicates that the third feature point is located in a different area from the reference image, and the third feature point located in the different area is eliminated. Similarly, if the fourth feature point is located in the black area, it indicates that the fourth feature point is located in the same area of ​​the image to be registered, and the fourth feature point located in the same area is retained. If the fourth feature point is located in the white area, it indicates that the fourth feature point is located in a different area from the image to be registered, and the fourth feature point located in the different area is eliminated.

[0096] Figure 7 The first feature point obtained after screening is shown as an example. Figure 4 Among the multiple feature points shown, the third feature points 1 to 5 are located in the black area, and the third feature point 6 is located in the white area. Therefore, the third feature points 1 to 5 are retained, and the third feature point 6 is removed. Figure 7 As shown, the first feature points obtained through screening include the third feature point 1 to the third feature point 5. Similarly, the second feature points obtained through screening include the fourth feature point 1 to the fourth feature point 5.

[0097] In other implementations, whether the third feature point is located in the same region as the reference image can be determined based on its value. Specifically, the value corresponding to the third feature point in the binary difference map / binary image is determined to be 0. If it is 0, the third feature point is located in the same region as the reference image; if it is 1, the third feature point is located in a region different from the reference image. The same applies to the fourth feature point.

[0098] In other implementations, in step S203, only the relative difference value corresponding to each third feature point is calculated, and the electronic device determines whether the relative difference value corresponding to the third feature point is less than or equal to the first threshold. If it is less than or equal to the first threshold, it is determined that the third feature point is located in the same area of ​​the reference image; if it is greater than the first threshold, it is determined that the third feature point is located in a different area of ​​the reference image.

[0099] S205 , removing the third feature point located in the difference area in the reference image and the fourth feature point located in the difference area in the image to be registered.

[0100] S206 , retaining the third feature points located in the same area in the reference image and the fourth feature points located in the same area in the image to be registered, and calculating a registration matrix based on the retained third feature points and fourth feature points.

[0101] The third feature point located in the same area of ​​the reference image is the first feature point; the fourth feature point located in the same area of ​​the image to be registered is the second feature point. After obtaining the first feature point located in the same area in the reference image and the second feature point located in the same area in the image to be registered, the electronic device can use a feature matching method to match the first feature point and the second feature point to obtain a one-to-one correspondence between the first feature point and the second feature point, and the corresponding two feature points are a set of feature point pairs. Common feature matching methods include matching based on descriptor distance, matching based on similarity measurement, etc., which are not limited in this application.

[0102] like Figure 8 As shown, the first feature point 1 in the reference image matches the second feature point 1 in the image to be registered; the first feature point 2 in the reference image matches the second feature point 2 in the image to be registered; the first feature point 3 in the reference image matches the second feature point 3 in the image to be registered; the first feature point 4 in the reference image matches the second feature point 4 in the image to be registered; the first feature point 5 in the reference image matches the second feature point 6 in the image to be registered.

[0103] Based on the correspondence between the first feature point and the second feature point, the electronic device can calculate the transformation relationship between the reference image and the image to be registered, and perform image registration through the transformation relationship. Affine transformation is a transformation relationship commonly used in image registration. The meaning of affine transformation is: if two images describe the same object, due to the differences in image size, position and angle, the position information of the corresponding feature points in the two images should satisfy a certain affine transformation relationship. Therefore, when registering images, the affine transformation matrix can be estimated using the matching feature point pairs between the two images to find an affine transformation that a sufficient number of feature point pairs obey.

[0104] Specifically, the affine transformation relationship between the first feature point (x, y) and the second feature point (x', y') can be expressed as the following formula (5).

[0105]

[0106] in, is a 3×3 affine transformation matrix, through which the second feature point is transformed to the first feature point. The affine transformation matrix is ​​the registration matrix. The constraints on the transformation parameters a, b, c, d, e, and f under different basic transformations (including translation, rotation, etc.) are different. Substituting each set of feature point pairs into equation (5) can calculate the affine transformation matrix.

[0107] S207: Register the reference image and the image to be registered according to the registration matrix.

[0108] The electronic device remaps the pixels in the image to be registered based on the affine transformation matrix, mapping the points in the image to be registered to the reference image to achieve image registration. When there are multiple frames of images to be registered, each frame corresponds to an affine transformation matrix, and each image to be registered is registered with the reference image separately. In a photo shooting scenario, after the reference image and the image to be registered are registered, the image to be registered can be fused with the reference image to obtain the final captured image.

[0109] The above technical solution takes into account the texture richness of the image and the texture difference between frames, uses the image content to calculate the relative texture difference, and screens the feature points through the relative texture difference, and screens out the feature points in the difference area between images, that is, the feature points located in the black area of ​​the difference map are retained, and the feature points located in the white area of ​​the difference map are eliminated. The filtered feature points are then matched to obtain feature point pairs, thereby avoiding inaccurate matching of feature point pairs, improving the calculation accuracy of the registration matrix, improving the algorithm effect, and thus improving the registration accuracy between images, reducing the blurring phenomenon of local motion areas in the fused image after registration, improving the clarity of image texture details, and improving the image quality of the output image.

[0110] In other implementations, the electronic device can extract multiple third feature points from the reference image through the above-mentioned Harris corner algorithm in step 202, and then perform a rapid search and match at the same position in the image to be registered based on the position coordinates of each third feature point in the reference image, and the rapid search converges to the best matching position, and finds the fourth feature point corresponding to each third feature point in the reference image in the image to be registered, and obtains a matched feature point pair. After determining the difference map in step 203, it is determined whether the feature point pair is located in the black area, and the feature point pair located in the black area is retained; the feature point pair located in the white area is eliminated. Specifically, the fourth feature point located in the white area of ​​the image to be registered is determined, and the feature point pair where the fourth feature is located in the white area of ​​the image to be registered is eliminated. Alternatively, the third feature point located in the white area of ​​the reference image is determined, and the feature point pair where the third feature is located in the white area of ​​the reference image is eliminated, and then the registration matrix is ​​calculated based on the filtered feature point pairs.

[0111] In other implementations, the electronic device may extract multiple third feature points from the reference image using the Harris corner algorithm in step 202. After determining the difference map in step 203, the electronic device determines whether the multiple third feature point pairs are located in a black area, retains the third feature points located in the black area, and removes the third feature points located in the white area, thereby obtaining the first feature points. Then, for the filtered first feature points, the electronic device finds the corresponding second feature points in the image to be registered, obtains matching feature point pairs, and calculates the registration matrix based on the feature point pairs.

[0112] In other implementations, after acquiring the reference image and the image to be registered, the electronic device first performs S203 to determine identical regions and different regions. The specific determination process is described above. Then, multiple first feature points are extracted from the identical regions in the reference image, and multiple second feature points are extracted from the identical regions in the image to be registered. The multiple first feature points are matched with the multiple second feature points to obtain feature point pairs, and a registration matrix is ​​calculated based on the feature point pairs.

[0113] Figure 9 FIG2 shows a flow chart of another image processing method provided by an embodiment of the present application. The method is executed by an electronic device, or by a chip configured in the electronic device. Figure 9 The image processing method shown includes the following S901 to S911.

[0114] S901: Acquire the image to be registered and the reference image.

[0115] S902 : extracting a plurality of third feature points from the reference image, and determining a plurality of fourth feature points in the image to be registered, to obtain matching feature point pairs.

[0116] Electronic devices can be Figure 3 The Harris corner algorithm shown extracts multiple third feature points in the reference image. Then, based on the position coordinates of each third feature point in the reference image, a fast search and match is performed at the same position in the image to be registered. The fast search converges to the best matching position, and the fourth feature point corresponding to each third feature point in the reference image in the image to be registered is found to obtain a matched feature point pair.

[0117] In other implementations, the electronic device may first extract multiple third feature points from the reference image and multiple fourth feature points from the image to be registered, and then perform secondary screening on the third feature points and the fourth feature points. After the screening is completed, matching is performed to obtain feature point pairs.

[0118] S903: Determine the difference area and the same area between the image to be registered and the reference image.

[0119] For the specific content, please refer to the content of S203 above and will not be repeated here.

[0120] S904: Determine whether each third feature point is located in the same area of ​​the reference image.

[0121] For the specific content, please refer to the content of S204 above and will not be repeated here.

[0122] S905: Eliminate the feature point pair corresponding to the third feature point located in the difference area.

[0123] That is, when the third feature point is located in the same area of ​​the reference image (black area), the third feature point is retained and the corresponding feature point pair is retained; when the third feature point is located in the different area of ​​the reference image (white area), the third feature point is eliminated and the corresponding feature point pair is eliminated.

[0124] S906: retain the feature point pair corresponding to the third feature point located in the same area.

[0125] S907 , determining a texture area and a non-texture area in the reference image.

[0126] The texture region includes a plurality of regular or irregular texture units. The electronic device can calculate the local standard deviation of each pixel of the reference image according to the above formula (1) to obtain the texture complexity of each pixel of the reference image. The region where the pixels whose texture complexity is greater than or equal to the second threshold are located is determined as the texture region; the region where the pixels whose texture complexity is less than the second threshold are located is determined as the non-texture region, that is, the image content of the texture region in the reference image is more complex than that of the non-texture region.

[0127] The second threshold value can be determined based on the texture complexity of each point. For example, the texture complexity of each pixel point is sorted in descending order, and the texture complexity at the 60% ratio is set as the first threshold value. For another example, the texture complexity of each pixel point is sorted in descending order, and the texture complexity at the median is set as the first threshold value. For another example, the average value of the texture complexity of each pixel point is calculated and the average value is set as the first threshold value.

[0128] S908: Determine whether the third feature point located in the same area is located in the texture area.

[0129] The third feature point located in the same area of ​​the reference image is the first feature point. The electronic device performs a secondary screening of the first feature points located in the same area based on the texture area and non-texture area in the reference image. Specifically, the electronic device may determine whether the local standard deviation corresponding to the first feature point is greater than or equal to a second threshold. If the local standard deviation is greater than or equal to the second threshold, the first feature point is determined to be located in the texture area; if the local standard deviation is less than the second threshold, the first feature point is determined to be located in the non-texture area.

[0130] When the first feature point is located in the texture area of ​​the reference image, the first feature point is retained and the corresponding feature point pair is retained; when the first feature point is located in the non-texture area of ​​the reference image, the first feature point is removed and the corresponding feature point pair is removed. Figure 10 The corner points extracted from the reference image using the Harris corner algorithm include third feature points 1 to 6. Third feature point 6 is located in the difference region, third feature point 1 and third feature point 2 are located in the same region and in the non-texture region, and third feature points 3, 4, and 5 are located in the same region and in the texture region. After secondary screening, the first feature points retained in the reference image are third feature points 3, 4, and 5.

[0131] S909: Eliminate the feature point pair corresponding to the third feature point located in the non-texture area.

[0132] S910: retain the feature point pairs corresponding to the third feature points located in the texture area, and calculate the registration matrix based on the retained feature point pairs.

[0133] S911 , registering the reference image and the image to be registered according to the registration matrix.

[0134] Figure 9 For details not mentioned in the Figure 2 The content will not be repeated here.

[0135] It should be noted that in other implementations, the electronic device may first execute steps S907-S910, and then execute steps S903-S906, that is, first filter out the feature points located in the texture area of ​​the reference image, and then eliminate the feature points located in the texture area and in the difference area, and filter out the first feature point located in the same area among the feature points located in the texture area, thereby obtaining the filtered feature point pairs.

[0136] In other implementations, the electronic device may first execute step S903 and step S907, and then determine whether the third feature point is located in the texture area and in the same area; if the third feature point is located in the texture area and in the same area, retain the third feature point; if the third feature point is located in the texture area and in the difference area, or if the third feature point is located in the non-texture area, eliminate the third feature point; thereby obtaining the filtered feature point pair.

[0137] In other implementations, the electronic device can perform screening based on the fourth feature point, that is, S904-S910 can be replaced by: determining whether each fourth feature point is located in the same area of ​​the image to be registered; eliminating the feature point pairs corresponding to the fourth feature point located in the difference area; retaining the feature point pairs corresponding to the fourth feature point located in the same area. Determine the texture area and non-texture area in the image to be registered. Determine whether the second feature point located in the same area is located in the texture area; eliminate the feature point pairs corresponding to the second feature point located in the non-texture area; retain the feature point pairs corresponding to the second feature point located in the texture area, and calculate the registration matrix based on the retained feature point pairs. The intersection of the same area and the texture area corresponds to the aforementioned second area.

[0138] The above technical solution, on the basis of improving the accuracy of the registration matrix, further takes into account the information of the texture area. By retaining the feature points at the intersection of the same area and the texture area and eliminating the feature points in the non-texture area, it optimizes the distribution of feature points in the reference image, increases the ratio of feature point pairs in the texture area, and can further improve the clarity of the texture area.

[0139] Combination of the above Figures 1 to 10 The method provided in the embodiment of the present application is described in detail. Figures 11 to 14 The electronic devices and apparatuses provided in the embodiments of the present application are described in detail. Figure 11 A schematic diagram of the hardware structure of the electronic device 100 is shown.

[0140] The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0141] It should be understood that the structure illustrated in the embodiments of the present invention does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0142] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors.

[0143] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0144] Processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 110. If processor 110 needs to use the same instruction or data again, it can directly access the memory. This avoids duplicate accesses, reduces processor 110 latency, and thus improves system efficiency.

[0145] In some embodiments, the processor 110 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface.

[0146] It is understood that the interface connection relationship between the modules illustrated in the embodiment of the present invention is merely an illustrative illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods from the above embodiments, or a combination of multiple interface connection methods.

[0147] The charging management module 140 is configured to receive charging input from a charger. The charger can be either a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 can receive charging input from the wired charger via the USB interface 130. In some wireless charging embodiments, the charging management module 140 can receive wireless charging input via the wireless charging coil of the electronic device 100. While charging the battery 142, the charging management module 140 can also provide power to the electronic device via the power management module 141.

[0148] The power management module 141 is used to connect the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and provides power to the processor 110, the internal memory 121, the display 194, the camera 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage, impedance). In some other embodiments, the power management module 141 can also be set in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.

[0149] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0150] Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In other embodiments, the antennas can be used in conjunction with a tuning switch.

[0151] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G applied to the electronic device 100. The modulation and demodulation processor may include a modulator and a demodulator. The wireless communication module 160 can provide solutions for wireless communications including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared technology (IR), etc. applied to the electronic device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, frequency modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module 160 can also receive signals to be transmitted from the processor 110, frequency modulate them, amplify them, and convert them into electromagnetic waves for radiation via the antenna 2.

[0152] Electronic device 100 implements display functionality through a GPU, display screen 194, and an application processor. A GPU is a microprocessor for image processing that connects display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or modify display information.

[0153] Display screen 194 is used to display images, videos, and the like. Display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-oLed, or a quantum dot light-emitting diode (QLED). In some embodiments, electronic device 100 may include one or N display screens 194, where N is a positive integer greater than one.

[0154] The electronic device 100 can implement a shooting function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, and an application processor.

[0155] The ISP processes data fed back by camera 193. For example, when taking a photo, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, which is then passed to the ISP for processing and converted into a visible image. The ISP can also perform algorithmic optimization on image noise, brightness, and skin tone. It can also optimize parameters such as exposure and color temperature of the captured scene. In some embodiments, the ISP can be located within camera 193.

[0156] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then passes the electrical signal to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0157] The digital signal processor is used to process digital signals. In addition to processing digital image signals, it can also process other digital signals. For example, when the electronic device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0158] Video codecs are used to compress or decompress digital video. Electronic device 100 may support one or more video codecs. This allows electronic device 100 to play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, and MPEG4.

[0159] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 via the external memory interface 120 to implement data storage functions. For example, files such as music and videos can be stored on the external memory card.

[0160] The internal memory 121 can be used to store computer executable program codes, which include instructions. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the electronic device 100 by running instructions stored in the internal memory 121 and / or instructions stored in a memory provided in the processor.

[0161] The electronic device 100 can implement audio functions such as music playback and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0162] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be provided in the processor 110, or some functional modules of the audio module 170 can be provided in the processor 110.

[0163] The speaker 170A, also called a "speaker", is used to convert audio electrical signals into sound signals. The electronic device 100 can listen to music or listen to hands-free calls through the speaker 170A.

[0164] The receiver 170B, also called a "handset", is used to convert audio electrical signals into sound signals. When the electronic device 100 receives a call or a voice message, the user can place the receiver 170B close to the ear to hear the voice.

[0165] Microphone 170C, also known as "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to the microphone 170C to input the sound signal into the microphone 170C. The electronic device 100 can be provided with at least one microphone 170C. In other embodiments, the electronic device 100 can be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the electronic device 100 can also be provided with three, four or more microphones 170C to collect sound signals, reduce noise, identify the source of sound, realize directional recording function, etc.

[0166] The headphone jack 170D is used to connect a wired headphone and can be the USB interface 130 or a 3.5mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0167] The pressure sensor 180A is used to sense pressure signals and convert them into electrical signals. In some embodiments, the pressure sensor 180A can be disposed on the display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc.

[0168] The gyroscope sensor 180B can be used to determine the motion posture of the electronic device 100. In some embodiments, the angular velocity of the electronic device 100 around three axes (i.e., x, y, and z axes) can be determined by the gyroscope sensor 180B. The gyroscope sensor 180B can be used for anti-shake shooting. For example, when the shutter is pressed, the gyroscope sensor 180B detects the angle of the electronic device 100 shaking, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to offset the shaking of the electronic device 100 through reverse movement to achieve anti-shake. The gyroscope sensor 180B can also be used for navigation and somatosensory game scenes.

[0169] The air pressure sensor 180C is used to measure air pressure. In some embodiments, the electronic device 100 calculates the altitude using the air pressure value measured by the air pressure sensor 180C to assist in positioning and navigation.

[0170] The magnetic sensor 180D includes a Hall sensor. The electronic device 100 can use the magnetic sensor 180D to detect the opening and closing of the flip case. In some embodiments, when the electronic device 100 is a flip phone, the electronic device 100 can detect the opening and closing of the flip cover based on the magnetic sensor 180D. Based on the detected opening and closing status of the case or flip cover, features such as automatic unlocking of the flip cover can be configured.

[0171] Accelerometer 180E can detect the magnitude of acceleration of electronic device 100 in all directions (generally three axes). It can also detect the magnitude and direction of gravity when electronic device 100 is stationary. It can also be used to identify the electronic device's posture, enabling applications such as switching between landscape and portrait modes and pedometers.

[0172] The distance sensor 180F is used to measure distance. The electronic device 100 can measure distance using infrared or laser. In some embodiments, when shooting a scene, the electronic device 100 can use the distance sensor 180F to measure distance to achieve fast focusing.

[0173] The proximity light sensor 180G may include, for example, a light emitting diode (LED) and a light detector, such as a photodiode. The light emitting diode may be an infrared light emitting diode. The electronic device 100 emits infrared light outward through the light emitting diode. The electronic device 100 uses a photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the electronic device 100. When insufficient reflected light is detected, the electronic device 100 can determine that there is no object near the electronic device 100. The electronic device 100 can use the proximity light sensor 180G to detect that the user is holding the electronic device 100 close to the ear to talk, so as to automatically turn off the screen to save power. The proximity light sensor 180G can also be used in leather case mode and pocket mode to automatically unlock and lock the screen.

[0174] Ambient light sensor 180L is used to sense ambient light brightness. Electronic device 100 can adaptively adjust the brightness of display screen 194 based on the perceived ambient light. Ambient light sensor 180L can also be used to automatically adjust white balance when taking photos. Ambient light sensor 180L can also work with proximity light sensor 180G to detect whether electronic device 100 is in a pocket to prevent accidental touches.

[0175] The fingerprint sensor 180H is used to collect fingerprints. The electronic device 100 can use the collected fingerprint characteristics to implement fingerprint unlocking, access application locks, fingerprint photography, fingerprint call answering, etc.

[0176] The temperature sensor 180J is used to detect temperature. In some embodiments, the electronic device 100 uses the temperature detected by the temperature sensor 180J to execute a temperature processing strategy. For example, when the temperature reported by the temperature sensor 180J exceeds a threshold, the electronic device 100 reduces the performance of the processor located near the temperature sensor 180J to reduce power consumption and implement thermal protection. In other embodiments, when the temperature is lower than another threshold, the electronic device 100 heats the battery 142 to prevent the electronic device 100 from shutting down abnormally due to low temperature. In other embodiments, when the temperature is lower than another threshold, the electronic device 100 boosts the output voltage of the battery 142 to prevent abnormal shutdown due to low temperature.

[0177] The touch sensor 180K is also called a "touch-sensitive device." The touch sensor 180K can be disposed on the display screen 194. The touch sensor 180K and the display screen 194 form a touch screen, also called a "touch screen." The touch sensor 180K is used to detect touch operations applied thereto or in the vicinity thereof. The touch sensor can transmit the detected touch operations to the application processor to determine the type of touch event. Visual output related to the touch operations can be provided via the display screen 194. In other embodiments, the touch sensor 180K can also be disposed on the surface of the electronic device 100, at a location different from that of the display screen 194.

[0178] Bone conduction sensor 180M can acquire vibration signals. In some embodiments, bone conduction sensor 180M can acquire vibration signals from vibrating bones in the human body. Bone conduction sensor 180M can also contact the human pulse to receive blood pressure signals. In some embodiments, bone conduction sensor 180M can also be incorporated into headphones to form bone conduction earphones.

[0179] The buttons 190 include a power button, a volume button, and the like. The buttons 190 may be mechanical buttons or touch buttons. The electronic device 100 may receive key inputs and generate key signal inputs related to user settings and function control of the electronic device 100.

[0180] Motor 191 can generate vibration prompts. Motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, audio playback, etc.) can correspond to different vibration feedback effects. For touch operations acting on different areas of the display screen 194, motor 191 can also correspond to different vibration feedback effects. Different application scenarios (for example: time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

[0181] The indicator 192 may be an indicator light, which may be used to indicate the charging status, power level changes, messages, missed calls, notifications, etc.

[0182] SIM card interface 195 is used to connect a SIM card. A SIM card can be inserted into or removed from SIM card interface 195 to connect to and disconnect from electronic device 100. Electronic device 100 may support one or N SIM card interfaces, where N is a positive integer greater than 1. Electronic device 100 interacts with the network through the SIM card to implement functions such as calls and data communications. In some embodiments, electronic device 100 uses an eSIM, or embedded SIM card. The eSIM card can be embedded in electronic device 100 and cannot be separated from it.

[0183] It should be understood that the illustrated structure of the embodiment of the present invention does not constitute a specific limitation on the mobile phone 100. In other embodiments of the present application, the mobile phone 100 may include more or fewer components than shown, or some components may be combined or separated, or arranged differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0184] The software system of the electronic device 100 can adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. In the embodiment of the present invention, the Android system with a layered architecture is used as an example to illustrate the software structure of the electronic device 100. It should be noted that in the embodiment of the present application, the operator system of the electronic device may include but is not limited to (Symbian), (Android), (iOS), (Blackberry), Hongmeng (HarmonyOS) and other operating systems, this application does not make any limitation.

[0185] Figure 12 1 is a software structure block diagram of the electronic device 100 according to an embodiment of the present invention.

[0186] A layered architecture divides software into several layers, each with distinct roles and responsibilities. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into five layers: from top to bottom: the application layer, the application framework layer, the Android runtime and system libraries, the hardware abstraction layer, and the kernel layer.

[0187] The application layer can include a series of application packages.

[0188] like Figure 12As shown, the application package may include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message, etc.

[0189] The application framework layer provides an application programming interface (API) and programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0190] like Figure 12 As shown, the application framework layer may include a camera service, a window manager, a content provider, a view system, a telephony manager, a resource manager, a notification manager, and the like.

[0191] The camera service provides interfaces for managing and accessing cameras.

[0192] The window manager is used to manage window programs. The window manager can obtain the display size, determine whether there is a status bar, lock the screen, take screenshots, etc.

[0193] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0194] The view system includes visual controls, such as those for displaying text and images. The view system is used to build applications. A display interface can consist of one or more views. For example, a display interface containing a text notification icon might include a view for displaying text and a view for displaying images.

[0195] The phone manager is used to provide communication functions of the electronic device 100, such as management of call status (including answering, hanging up, etc.).

[0196] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0197] The Notification Manager allows applications to display notifications in the status bar. These messages can be displayed briefly and then disappear automatically without user interaction. For example, the Notification Manager is used to notify users of completed downloads and message reminders. The Notification Manager can also display notifications in the top status bar of the system as icons or scrolling text, such as notifications from background applications, or as dialog windows on the screen. Examples include text messages in the status bar, beeps, vibrations on electronic devices, and flashing indicator lights.

[0198] Android Runtime includes core libraries and a virtual machine. Android runtime is responsible for scheduling and management of the Android system.

[0199] The core library consists of two parts: one is the function that needs to be called by the Java language, and the other is the Android core library.

[0200] The application layer and application framework layer run in a virtual machine. The virtual machine executes Java files in the application layer and application framework layer as binary files. The virtual machine manages object lifecycles, stack management, thread management, security and exception management, and garbage collection.

[0201] The system library can include multiple functional modules, such as surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0202] The surface manager is used to manage the display subsystem and provide fusion of 2D and 3D layers for multiple applications.

[0203] The media library supports playback and recording of a variety of common audio and video formats, as well as static image files. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0204] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0205] A 2D graphics engine is a drawing engine for 2D drawings.

[0206] The Hardware Abstraction Layer (HAL) abstracts the hardware. By calling the HAL interfaces within the HAL, the application layer and framework layer above the HAL are connected to the driver layer and hardware layer below it, enabling data transmission and function control.

[0207] The hardware abstraction layer includes a registration module and an image processing module. The registration module is used to filter feature points using a difference map. The image processing module is used to perform image processing, such as image fusion and image enhancement.

[0208] The kernel layer is the layer between hardware and software. The kernel layer includes at least display driver, camera driver, audio driver, sensor driver, etc.

[0209] The following describes the workflow of the software of the electronic device 100 in conjunction with a photo-taking scenario.

[0210] When the user turns on the screen of the electronic device and controls the electronic device to be in an unlocked state, the electronic device can display a desktop. Icons of multiple installed applications are displayed on the desktop of the electronic device, including a camera application icon. The user can touch and click the camera application icon 10. The camera application calls the interface of the application framework layer to start the camera application, and then starts the camera driver by calling the kernel layer to capture static images or videos through the camera 193.

[0211] Taking high dynamic range shooting mode as an example, camera 193 captures multiple frames of images of the same scene and then sends them to the registration module. The registration module uses one frame of the multiple images as the reference image and the remaining frames as the images to be registered. The registration module then extracts multiple first feature points from the reference image and identifies multiple second feature points in the image to be registered, obtaining matching feature point pairs. The module then determines the difference and common areas between the image to be registered and the reference image. It then determines whether each first feature point lies within the same area of ​​the reference image. Feature point pairs corresponding to first feature points in the difference areas are eliminated, while retaining feature point pairs corresponding to first feature points in the common area. The registration module then determines texture and non-texture areas in the reference image. It then determines whether first feature points in the same area lie within a texture area. Feature point pairs corresponding to first feature points in the non-texture area are eliminated, while retaining feature point pairs corresponding to first feature points in the texture area. The module then calculates a registration matrix based on the retained feature point pairs. After obtaining the registration matrix, the registration module aligns the reference image and the image to be registered according to the registration matrix. The registration module sends the registration results to the image processing module, and the image processing module performs subsequent image processing procedures.

[0212] Figure 13 This is a schematic diagram of the structure of an image processing device provided in an embodiment of the present application. The image processing device may be an electronic device in an embodiment of the present application, or a chip or chip system within an electronic device.

[0213] like Figure 13 As shown, the image processing apparatus 1300 may include: a processing unit 1301. The processing unit 1301 is configured to support the image processing apparatus 1300 in executing the processing steps in the above method.

[0214] Specifically, the processing unit 1301 is used to obtain the image to be registered and the reference image, where the reference image and the image to be registered are obtained after shooting the same object; in the first area of ​​each of the reference image and the image to be registered, a plurality of feature point pairs are determined, and the change pattern of the pixel points in the first area of ​​the image to be registered is no different from the change pattern of the pixel points in the first area of ​​the reference image, and the plurality of feature point pairs include a plurality of first feature points located in the first area of ​​the reference image, and each second feature point located in the first area of ​​the image to be registered and corresponding to each first feature point; based on the plurality of feature point pairs, a transformation relationship between the image to be registered and the reference image is determined; based on the transformation relationship, the image to be registered and the reference image are registered.

[0215] In some implementations, apparatus 1000 further includes a storage unit 1302. Storage unit 1302 is connected to processing unit 1301 via a circuit. Storage unit 1302 may include one or more memories, which may be devices in one or more devices or circuits used to store programs or data. Storage unit 1302 may exist independently and be connected to processing unit 1301 via a communication bus. Storage unit 1302 may also be integrated with processing unit 1301.

[0216] The storage unit 1302 can store computer-executable instructions for the method in the electronic device, so that the processing unit 1301 can execute the method in the above embodiment. The storage unit 1302 can be a register, a cache, or a random access memory (RAM), etc., or it can be a read-only memory (ROM) or other type of static storage device that can store static information and instructions.

[0217] Figure 14 This is a schematic diagram of the structure of a chip provided in an embodiment of the present application. Figure 14 As shown, the chip 1400 includes one or more (including two) processors 1401 , a communication line 1402 and a communication interface 1403 . Optionally, the chip 1400 also includes a memory 1404 .

[0218] In some embodiments, the memory 1404 stores the following elements: executable modules or data structures, or a subset thereof, or an extended set thereof.

[0219] The method described in the above embodiment of the present application can be applied to the processor 1401, or implemented by the processor 1401. The processor 1401 may be an integrated circuit chip with signal processing capabilities. During the implementation process, the steps of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 1401. The above-mentioned processor 1401 can be a general-purpose processor (for example, a microprocessor or a conventional processor), a digital signal processor, an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates, transistor logic devices or discrete hardware components. The processor 1401 can implement or execute the disclosed methods, steps and logic block diagrams in the embodiments of the present application.

[0220] The steps of the method disclosed in the embodiments of the present application can be directly embodied as being executed by a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. Among them, the software module can be located in a mature storage medium in the field such as random access memory, read-only memory, programmable read-only memory, or electrically erasable programmable read only memory (EEPROM). The storage medium is located in memory 1404, and processor 1401 reads the information in memory 1404 and completes the steps of the above method in combination with its hardware.

[0221] The processor 1401 , the memory 1404 , and the communication interface 1403 can communicate with each other via the communication line 1402 .

[0222] In the above embodiment, the instructions stored in the memory for execution by the processor may be implemented in the form of a computer program product, wherein the computer program product may be pre-written in the memory or downloaded and installed in the memory in the form of software.

[0223] The present application embodiment also provides a computer program product, which includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a server or data center that includes one or more available media integrated. For example, available media can include magnetic media (e.g., floppy disk, hard disk or tape), optical media (e.g., digital versatile disc (DVD)), or semiconductor media (e.g., solid state drive (SSD)).

[0224] An embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory is used to store a computer program, and the processor is used to execute the computer program to perform the above method.

[0225] The present embodiment provides a chip. The chip includes a processor configured to invoke a computer program stored in a memory to execute the technical solution of the above embodiment. The implementation principles and technical effects are similar to those of the above-mentioned related embodiments and will not be further described here.

[0226] In the embodiments provided in the present application, it should be understood that the disclosed devices / electronic devices and methods can be implemented in other ways. For example, the device / electronic device embodiments described above are merely schematic. For example, the division of the modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms. In addition, the functional units in the various embodiments of the present application can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

[0227] The embodiments of the present application also provide a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, it implements the steps in the above-mentioned various method embodiments. The computer-readable storage medium stores a computer program or instructions. When the computer program or instructions are executed by the processor, the above-mentioned method is implemented. The methods described in the above embodiments can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. If implemented in software, the functions can be stored as one or more instructions or codes on a computer-readable medium or transmitted on a computer-readable medium. Computer-readable media can include computer storage media and communication media, and can also include any medium that can transfer a computer program from one place to another. The storage medium can be any target medium that can be accessed by a computer.

[0228] As a possible design, computer-readable media may include compact disc read-only memory (CD-ROM), RAM, ROM, EEPROM or other optical disc storage; computer-readable media may include magnetic disk storage or other magnetic disk storage devices. Moreover, any connecting line may also be appropriately referred to as a computer-readable medium. For example, if the software is transmitted from a website, server or other remote source using a coaxial cable, fiber optic cable, twisted pair, DSL or wireless technologies such as infrared, radio and microwave, the coaxial cable, fiber optic cable, twisted pair, DSL or wireless technologies such as infrared, radio and microwave are included in the definition of medium. Disks and optical discs as used herein include compact discs (CDs), laser discs, optical discs, DVDs, floppy disks and Blu-ray discs, where disks generally reproduce data magnetically, while optical discs reproduce data optically using lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0229] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processing unit of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0230] In the above description, specific details such as specific system structures and technologies are provided for illustration rather than limitation to facilitate a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obscuring the description of the present application with unnecessary details.

[0231] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or collections thereof.

[0232] It should also be understood that the term "plurality" in this specification and the appended claims refers to two or more. In the description of this application, unless otherwise specified, " / " represents or. For example, A / B can represent A or B. "and / or" herein is merely a description of an association relationship between associated objects, and refers to any and all possible combinations of one or more of the associated listed items, including these combinations. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone.

[0233] As used in this specification and the appended claims, the term "if" can be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0234] In addition, to facilitate the clear description of the technical solutions of this application, the words "first" and "second" are used to distinguish between identical or similar items with substantially the same functions and effects. Those skilled in the art will understand that the words "first" and "second" do not limit the quantity or order of execution, and should not be understood as indicating or implying relative importance. Furthermore, the words "first" and "second" do not necessarily define differences.

[0235] References to "one embodiment" or "some embodiments" in this specification mean that a particular feature, structure, or characteristic described in conjunction with that embodiment is included in one or more embodiments of the present application. Thus, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," and "in other embodiments" appearing in various places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more but not all embodiments," unless otherwise specifically emphasized. The terms "including," "comprising," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.

[0236] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. An image processing method, characterized in that: Applied to electronic equipment, the method includes: Acquire an image to be registered and a reference image, wherein the reference image and the image to be registered are acquired after photographing the same subject; Determining the relative texture difference of each pixel in the reference image and the image to be registered; determining, based on the relative texture difference of each pixel, a first region of each of the reference image and the image to be registered, wherein a change pattern of the pixels of the image to be registered in the first region is the same as a change pattern of the pixels of the reference image; Determining, in a first region of each of the reference image and the image to be registered, a plurality of feature point pairs, the plurality of feature point pairs comprising a plurality of first feature points located in the first region of the reference image, and second feature points located in the first region of the image to be registered and corresponding to the first feature points; determining a transformation relationship between the image to be registered and the reference image according to the plurality of feature point pairs; The image to be registered and the reference image are registered according to the transformation relationship.

2. The method according to claim 1, characterized in that The first area in the reference image includes a second area, and the second area includes at least one texture unit; determining a plurality of feature point pairs in the first area corresponding to the reference image and the image to be registered, comprising: In the second area in the reference image and the first area of ​​the image to be registered, multiple feature point pairs are determined, and the multiple feature point pairs include multiple first feature points in the reference image located in the second area, and each second feature point in the image to be registered corresponding to each first feature point.

3. The method according to claim 2, characterized in that Determining a plurality of feature point pairs between the second area in the reference image and the first area of ​​the image to be registered includes: Determine a plurality of third feature points from the reference image, and determine fourth feature points corresponding to the third feature points from the image to be registered; Determining a first region of each of the reference image and the image to be registered, and determining a second region of the reference image; Based on the third feature point located in the second area of ​​the reference image and the fourth feature point located in the first area of ​​the image to be registered, a plurality of feature point pairs are determined, wherein the plurality of first feature points include the third feature point located in the second area of ​​the reference image, and the second feature points include the fourth feature points located in the first area of ​​the image to be registered.

4. The method according to claim 2, characterized in that Determining a plurality of feature point pairs between the second area in the reference image and the first area of ​​the image to be registered includes: Determining a first region of each of the reference image and the image to be registered, and determining a second region of the reference image; A plurality of first feature points are determined in the second region of the reference image, and second feature points corresponding to the first feature points are determined in the first region of the image to be registered, to obtain a plurality of feature point pairs.

5. The method according to claim 3 or 4, characterized in that Determining a first region of each of the reference image and the image to be registered includes: Determining a local standard deviation of each pixel in the reference image and a local standard deviation of each pixel in the image to be registered; Determining a relative difference value of each pixel in the image to be registered relative to each pixel in the reference image based on a local standard deviation of each pixel in the reference image and a local standard deviation of each pixel in the image to be registered; The first regions of the reference image and the image to be registered are determined according to a relative difference value of each pixel in the image to be registered relative to each pixel in the reference image.

6. The method according to claim 5, characterized in that The determining, based on the relative difference value of each pixel point in the image to be registered relative to each pixel point in the reference image, the first region of each reference image and the image to be registered, comprises: Determine whether the relative difference value corresponding to each pixel point is less than or equal to a first threshold; The region where the pixel points having relative difference values ​​less than or equal to the first threshold are located is determined as the first region corresponding to each of the reference image and the image to be registered.

7. The method according to claim 5, characterized in that The determining, based on the relative difference value of each pixel point in the image to be registered relative to each pixel point in the reference image, the first region of each reference image and the image to be registered, comprises: The values ​​of pixels whose relative difference values ​​are less than or equal to a first threshold are set to 0, and the values ​​of pixels whose relative difference values ​​are greater than the first threshold are set to 1, to obtain a binary texture difference map; performing a morphological opening and closing operation on the binarized texture difference map; The regions where the pixel values ​​are 0 in the binarized texture difference map after the morphological opening and closing operation are determined as the first regions corresponding to the reference image and the image to be registered.

8. The method according to claim 5, characterized in that The method further comprises: determining coordinates of the plurality of third feature points in the reference image; According to the coordinates of the plurality of third feature points in the reference image, it is determined whether the plurality of third feature points are located in the second area of ​​the reference image.

9. The method according to claim 5, characterized in that The determining the second area of ​​the reference image includes: Determining a local standard deviation corresponding to each pixel in the reference image; The area where the pixels whose local standard deviation in the first area of ​​the reference image is greater than or equal to the second threshold are located is determined as the second area.

10. The method according to claim 4, characterized in that The determining of the plurality of first feature points includes: A plurality of first feature points are determined according to the Harris corner algorithm.

11. The method according to claim 1, wherein The transformation relationship is an affine transformation.

12. The method according to claim 1, characterized in that The step of obtaining the image to be registered and the reference image comprises: Starting a camera application in the electronic device; In response to a user's shooting instruction, multiple frames of images are acquired, where the multiple frames of images include the reference image and at least one frame of image to be registered.

13. An electronic device, characterized in that: include: one or more processors; one or more memories; The memory stores one or more programs, and when the one or more programs are executed by the processor, the electronic device executes the method according to any one of claims 1 to 12.

14. A computer-readable storage medium, characterized in that The computer-readable storage medium stores instructions, which, when executed on a computer, enable the computer to perform the method according to any one of claims 1 to 12.

Images