Image processing method, electronic equipment and computer readable storage medium

Abstract

The invention discloses an image processing method, electronic equipment and a computer readable storage medium, and relates to the technical field of electronics, and the method comprises the steps: obtaining a to-be-registered image and a reference image; a plurality of feature point pairs are determined in respective first areas of the reference image and the to-be-registered image, the change rule of pixel points in the first area of the to-be-registered image and the change rule of pixel points in the first area of the reference image have no difference, and the plurality of feature point pairs comprise a plurality of first feature points located in the first area of the reference image; the second feature points are located in a first area of the to-be-registered image and correspond to the first feature points; determining a transformation relation between the to-be-registered image and the reference image according to the plurality of feature point pairs; and according to the transformation relation, registering the to-be-registered image and the reference image. Based on the technical scheme, the electronic equipment screens the feature points, so that the condition of inaccurate matching of the feature point pairs is avoided, the calculation accuracy of the registration matrix can be improved, the registration precision among the images is further improved, the phenomenon that a local motion region in the registered image is blurred is reduced, the definition of image texture details is improved, and the user experience is improved. And the image quality of the output image is improved.

Description

Technical Field

[0001] The present application relates to the field of electronic technology, and in particular to an image processing method, an electronic device, and a computer-readable storage medium. Background Art

[0002] Image registration is to map one image (hereinafter referred to as the image to be registered) to another image (hereinafter referred to as the reference image) by finding a spatial transformation relationship between two images in a set of image data obtained under different shooting conditions (such as different shooting times, different shooting angles, or using different sensors), so that the pixels corresponding to the same position in the two images correspond one to one, thereby achieving the purpose of information fusion. At present, when multiple frames of images are registered, a series of feature points are usually determined in the reference image, and points matching each feature point are found in the image to be registered to obtain feature point pairs. The registration matrix is ​​calculated based on each feature point pair, and then the reference image and the image to be registered are aligned.

[0003] When the image has rich texture and local motion, there are some differences between the reference image and the image to be registered. The current registration scheme may result in inaccurate feature point pair matching. For example, when shooting grass and leaves in an outdoor scene, the reference image and the image to be registered are obtained during the shooting. Since the grass and leaves sway left and right with the wind or there is jitter during the shooting, there are differences between the reference image and the image to be registered. At this time, the feature point pairs obtained may have feature points and matching points corresponding to different positions in the same space. As a result, the registration matrix calculated based on the feature point pairs has low accuracy, and there are errors in the registration between images. As a result, the fused image after registration has problems such as blurring of local motion areas and loss of texture details, resulting in poor image quality. Summary of the invention

[0004] The present application provides an image processing method, an electronic device and a computer-readable storage medium, which can improve the accuracy of the registration matrix, enhance the clarity of image texture details, and improve the image quality of the output image. The technical solution is as follows:

[0005] In a first aspect, an embodiment of the present application provides an image processing method, which is applied to an electronic device, and the method includes:

[0006] Acquire an image to be registered and a reference image, where the reference image and the image to be registered are acquired after photographing the same photographed object;

[0007] In the first regions of the reference image and the image to be registered, a plurality of feature point pairs are determined, wherein a change pattern of the pixels in the first region of the image to be registered is the same as a change pattern of the pixels in the first region of the reference image, and the plurality of feature point pairs include a plurality of first feature points located in the first region of the reference image, and second feature points located in the first region of the image to be registered and corresponding to the first feature points;

[0008] Determine the transformation relationship between the image to be registered and the reference image according to multiple feature point pairs;

[0009] According to the transformation relationship, the image to be registered and the reference image are registered.

[0010] Based on the above technical solution, after acquiring the reference image and the image to be registered, the electronic device determines the first area of ​​each of the reference image and the image to be registered so that multiple feature point pairs are located in the same area between the reference image and the image to be registered, and are not located in the area where there are differences between the image to be registered and the reference image, that is, the feature point pairs located in the area where there are differences between the image to be registered and the reference image are screened out, thereby avoiding the situation where the feature point pairs are inaccurately matched, and then determining the transformation relationship (that is, the registration matrix) between the image to be registered and the reference image based on the multiple feature point pairs, which can improve the calculation accuracy of the registration matrix, thereby improving the registration accuracy between images, reducing the blurring phenomenon of local motion areas in the image after registration, improving the clarity of image texture details, and improving the image quality of the output image.

[0011] In combination with the first aspect, in certain implementations of the first aspect, the method further includes: the first area in the reference image includes the second area, and the second area includes at least one texture unit; in the first area of ​​each of the reference image and the image to be registered, multiple feature point pairs are determined, including: in the second area of ​​the reference image and the first area of ​​the image to be registered, multiple feature point pairs are determined, and the multiple feature point pairs include multiple first feature points located in the second area of ​​the reference image, and each second feature point corresponding to each first feature point in the image to be registered. In this way, on the basis of filtering out the feature point pairs in the difference area between the images, further selecting the feature point pairs located in the texture-rich area can further improve the clarity of the image texture details on the basis of improving the calculation accuracy of the registration matrix.

[0012] In combination with the first aspect and the above implementation, in some implementations of the first aspect, multiple feature point pairs are determined in the second area of ​​the reference image and the first area of ​​the image to be registered, including: determining multiple third feature points from the reference image, and determining each fourth feature point corresponding to each third feature point from the image to be registered; determining the first area of ​​the reference image and the image to be registered, and determining the second area of ​​the reference image; determining multiple feature point pairs based on the third feature point located in the second area of ​​the reference image and the fourth feature point located in the first area of ​​the image to be registered, wherein the multiple first feature points include the third feature point located in the second area of ​​the reference image, and each second feature point includes each fourth feature point located in the first area of ​​the image to be registered. In this way, the feature point pairs can be determined first, and then the feature point pairs can be screened.

[0013] In combination with the first aspect and the above implementation, in some implementations of the first aspect, multiple feature point pairs are determined in the second area of ​​the reference image and the first area of ​​the image to be registered, including: determining the first area of ​​the reference image and the image to be registered, and determining the second area of ​​the reference image; determining multiple first feature points in the second area of ​​the reference image, and determining each second feature point corresponding to each first feature point in the first area of ​​the image to be registered, to obtain multiple feature point pairs. In this way, the area for searching feature points can be narrowed, thereby reducing the time for searching feature points and improving the registration speed.

[0014] In combination with the first aspect and the above implementations, in certain implementations of the first aspect, determining the first regions of the reference image and the image to be registered includes: determining the local standard deviation of each pixel in the reference image and the local standard deviation of each pixel in the image to be registered; determining the relative difference value of each pixel in the image to be registered relative to each pixel in the reference image based on the local standard deviation of each pixel in the reference image and the local standard deviation of each pixel in the image to be registered; determining the first regions of the reference image and the image to be registered based on the relative difference value of each pixel in the image to be registered relative to each pixel in the reference image. The local standard deviation can reflect the change of the pixel in the reference image or the image to be registered, and the difference between the local standard deviations can be used to measure whether the changes in the local regions where the same pixel in the two images are located are the same, thereby determining the first region.

[0015] In combination with the first aspect and the above-mentioned implementation methods, in some implementation methods of the first aspect, determining the first regions of the reference image and the image to be registered respectively according to the relative difference value of each pixel in the image to be registered relative to each pixel in the reference image includes: determining whether the relative difference value corresponding to each pixel is less than or equal to the first threshold; determining the region where the pixels whose relative difference value is less than or equal to the first threshold are located as the first regions corresponding to the reference image and the image to be registered respectively. Among them, the region where the pixels whose relative difference value is greater than the first threshold are located is the region where there are differences between the image to be registered and the reference image. Furthermore, determining whether multiple third feature points are located in the first region includes: determining whether the relative difference value corresponding to the third feature point is less than or equal to the first threshold. If the relative difference value corresponding to the third feature point is less than or equal to the first threshold, the third feature point is located in the first region; otherwise, the third feature point is located in the region where there are differences.

[0016] In combination with the first aspect and the above implementation, in some implementations of the first aspect, the first regions of the reference image and the image to be registered are determined according to the relative difference value of each pixel in the image to be registered relative to each pixel in the reference image, including: setting the value of the pixel whose relative difference value is less than or equal to the first threshold to 0, and setting the value of the pixel whose relative difference value is greater than the first threshold to 1, to obtain a binary texture difference; performing a morphological opening and closing operation on the binary texture difference map; determining the region where the pixel value is 0 in the binary texture difference map after the morphological opening and closing operation as the first region corresponding to the reference image and the image to be registered. Further, determining whether multiple third feature points are located in the first region includes: determining whether the corresponding value of the third feature point is 0, if the corresponding value of the third feature point is 0, the third feature point is located in the first region; if the corresponding value of the third feature point is 1, the third feature point is located in the region where there is a difference.

[0017] In combination with the first aspect and the above implementation, in some implementations of the first aspect, the above method further includes: determining the coordinates of the plurality of third feature points in the reference image; and determining whether the plurality of third feature points are located in the first region according to the coordinates of the plurality of third feature points in the reference image. Here, it can be determined whether the third feature point is included in the plurality of pixel points included in the first region according to the coordinates, and if included, the third feature point is located in the first region; if not included, the third feature point is located in the region where there is a difference.

[0018] In combination with the first aspect and the above-mentioned implementation methods, in some implementation methods of the first aspect, determining the second area of ​​the reference image includes: determining the local standard deviation corresponding to each pixel in the reference image; and determining the area where the pixels in the first area of ​​the reference image whose local standard deviation is greater than or equal to the second threshold are located as the second area.

[0019] In combination with the first aspect and the above implementations, in some implementations of the first aspect, determining a plurality of first feature points includes: determining a plurality of first feature points in a reference image according to a Harris corner algorithm.

[0020] In combination with the first aspect and the above implementations, in some implementations of the first aspect, the transformation relationship is an affine transformation.

[0021] In combination with the first aspect and the above-mentioned implementation methods, in some implementation methods of the first aspect, obtaining the image to be registered and the reference image includes: starting a camera application in an electronic device; in response to a user's shooting instruction, obtaining multiple frames of images, the multiple frames of images including the reference image and at least one frame of the image to be registered.

[0022] In a second aspect, an embodiment of the present application provides an electronic device, comprising: one or more processors; one or more memories; the memory stores one or more programs, and when the one or more programs are executed by the processor, the electronic device executes the method described in any one of the above-mentioned first aspect and the first aspect.

[0023] In a third aspect, an embodiment of the present application provides a device, which is included in an electronic device, and the device has the function of realizing the above-mentioned aspects and the behavior of the electronic device in the possible implementation methods of the above-mentioned aspects. The function can be implemented by hardware, or the corresponding software can be implemented by hardware. The hardware or software includes one or more modules or units corresponding to the above-mentioned functions. For example, a display module or unit, a detection module or unit, a processing module or unit, etc.

[0024] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein instructions are stored in the computer-readable storage medium, which, when executed on a computer, enables the computer to execute the method described in any one of the first aspect and the first aspect above.

[0025] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it is used to execute the method described in any one of the first aspect and the first aspect.

[0026] In the sixth aspect, an embodiment of the present application provides a chip system, which includes one or more interface circuits and one or more processors; the interface circuits and the processors are interconnected through lines; the processor is used to execute the method described in any one of the first aspect and the first aspect.

[0027] The technical effects obtained by the above-mentioned second, third, fourth, fifth and sixth aspects are similar to the technical effects obtained by the corresponding technical means in the above-mentioned first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 A schematic diagram of an application scenario provided by an embodiment of the present application is shown;

[0029] Figure 2 A schematic diagram showing a flow chart of an image processing method provided in an embodiment of the present application is shown;

[0030] Figure 3 A schematic diagram of a feature point extraction principle provided by an embodiment of the present application is shown;

[0031] Figure 4 A schematic diagram of a third feature point in a reference image provided by an embodiment of the present application is shown;

[0032] Figure 5 A schematic diagram of a binary image provided by an embodiment of the present application is shown;

[0033] Figure 6 Another schematic diagram of a binary image provided by an embodiment of the present application is shown;

[0034] Figure 7 A schematic diagram of a first feature point located in a first area in a reference image provided by an embodiment of the present application is shown;

[0035] Figure 8 A schematic diagram of a feature point matching result provided in an embodiment of the present application is shown;

[0036] Fig. 9 A schematic diagram showing a process of another image processing method provided in an embodiment of the present application is shown;

[0037] Fig.10 A schematic diagram of a first feature point located in a second area in a reference image provided by an embodiment of the present application is shown;

[0038] Fig.11 A schematic diagram of the structure of an electronic device 100 provided in an embodiment of the present application is shown;

[0039] Fig.12 A schematic diagram of the software structure of an electronic device provided in an embodiment of the present application is shown;

[0040] Fig.13 A schematic diagram of a device provided in an embodiment of the present application is shown;

[0041] Fig.14 A schematic diagram of a chip structure provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical scheme and advantages of the present application clearer, the implementation mode of the present application will be further described in detail below with reference to the accompanying drawings. Hereinafter, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present embodiment, unless otherwise specified, the meaning of "multiple" is two or more.

[0043] Image registration is the process of aligning and matching multiple frames of different images acquired for the same scene under different shooting conditions (such as different shooting times, different shooting angles, or using different sensors). Specifically, for two images in a set of image data sets, one image (hereinafter referred to as the image to be registered) is mapped to another image (hereinafter referred to as the reference image, also referred to as the benchmark image) by finding a spatial transformation relationship, so that the pixels corresponding to the same position in the two images correspond one to one, thereby achieving the purpose of information fusion.

[0044] At present, when multiple frames of images are registered, a series of feature points are usually determined in the reference image. According to the local template of each feature point in the reference image, points matching each feature point are found in the image to be registered. Through similarity comparison, a point in the image to be registered that is most similar to the feature point is found as the corresponding matching point to obtain a pair of feature points. According to each pair of feature points, the registration matrix is ​​calculated for spatial transformation to achieve alignment between the reference image and the image to be registered.

[0045] At present, the registration scheme only considers corner points, but not the content information of the image. When the image has rich texture and local motion, there is a texture difference between the reference image and the image to be registered, and the feature point pair may not match accurately. For example, when shooting grass and leaves in an outdoor scene (i.e., rich texture), the reference image and the image to be registered are obtained during shooting. The grass and leaves sway left and right with the wind or there is jitter during shooting, which makes the reference image and the image to be registered have texture differences, but the pixel values ​​of the difference parts are relatively close. At this time, the feature point pairs obtained may have feature points and matching points corresponding to different positions in the same space, that is, the feature point pairs are not accurately matched. Then, the registration matrix calculated based on the feature point pairs has low accuracy, and there are errors in the registration between images. Then, the fused image after registration has problems such as blurring of local motion areas and loss of texture details, and the image quality is poor.

[0046] In view of this, an embodiment of the present application provides an image processing method, which, after acquiring a reference image and an image to be registered, determines a plurality of feature point pairs in the first area of ​​each of the reference image and the image to be registered, the plurality of feature point pairs including a plurality of first feature points located in the first area of ​​the reference image, and second feature points located in the first area of ​​the image to be registered and corresponding to the first feature points; determines a transformation relationship between the image to be registered and the reference image based on the plurality of feature point pairs; and registers the image to be registered and the reference image based on the transformation relationship. Among them, there is no difference between the change pattern of the pixels in the first area of ​​the image to be registered and the change pattern of the pixels in the first area of ​​the reference image. That is to say, the present application determines the first area of ​​the reference image and the image to be registered respectively, so that multiple feature point pairs are located in the same area between the reference image and the image to be registered, and are not located in the area where there is a difference between the image to be registered and the reference image. The feature point pairs in the difference area between the images can be screened out, thereby avoiding inaccurate matching of feature point pairs, and then determining the transformation relationship between the image to be registered and the reference image (i.e., the registration matrix) based on multiple feature point pairs, which can improve the calculation accuracy of the registration matrix, thereby improving the registration accuracy between images, reducing the blurring phenomenon of local motion areas in the image after registration, improving the clarity of image texture details, and improving the image quality of the output image.

[0047] The image processing method provided in the embodiment of the present application can be applied to various electronic devices. The electronic device can be, but is not limited to, a mobile phone, a sports camera (GoPro), a digital camera, a tablet computer, a desktop, a laptop, a handheld computer, a notebook computer, a vehicle-mounted device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook, a cellular phone, a personal digital assistant (personal digital assistant, PDA), an augmented reality (augmentedreality, AR)\virtual reality (virtual reality, VR) device, a mobile phone, etc., and the embodiment of the present application does not limit this.

[0048] The image processing method provided in the embodiment of the present application can be applied to a variety of application scenarios, for example, when applied to camera photography or video shooting, when multiple frames of images need to be fused in some shooting modes (such as night scene shooting), image registration can be performed before fusion, and better fusion effect can usually be obtained after accurate image registration. For another example, in medicine, the multimodal data of the patient can be fused into the same coordinate system to assist diagnosis. For another example, it can be applied to 3D reconstruction, image alignment, motion tracking, etc.

[0049] For example, Figure 1 A schematic diagram of a scene in which the image processing method of the present application is applied to camera photography is shown. Figure 1 As shown, the texture of the photographed object A is relatively rich, and the image obtained when photographing the photographed object A is easily blurred; Figure 1 (a) is an image of the photographed object A obtained by image processing using a method provided by conventional technology; Figure 1 (b) is an image of the same photographed object A obtained by the image processing method provided by the embodiment of the present application; Figure 1 The image shown in (b) is Figure 1 Compared with the image shown in (a), the texture in the framed area is clearer; it can be seen that the image processing method of the embodiment of the present application can enhance the clarity of image texture details and improve image quality.

[0050] Combine the following Figures 2 to 10 The image processing method provided in the embodiment of the present application is described in detail.

[0051] Figure 2 The present invention provides a flowchart of an image processing method provided in an embodiment of the present invention. The method is executed by an electronic device, or by a chip configured in the electronic device. Figure 2 The image processing method shown includes the following S201 to S207.

[0052] S201, obtaining an image to be registered and a reference image.

[0053] The electronic device acquires multiple frames of images for the same photographed object, selects one image from the multiple frames as a reference image, and the remaining images in the multiple frames are images to be registered. The electronic device transforms the image to be registered according to the reference image, thereby realizing multi-frame image registration.

[0054] An electronic device can acquire multiple images in real time. For example, an electronic device has a camera and a camera application. In a photo-taking scenario, a user can open the camera application in the electronic device, select a shooting mode, and click the shooting button to take a photo. The electronic device responds to the user's click operation, continuously captures multiple frames of images for the photographed object, and aligns the multiple frames of images. Among them, the multiple frames of images can be images acquired by the electronic device at different time points for the photographed object in the same scene, the reference image can be the first frame of the multiple frames of images with the earliest acquisition time point, and the reference image can also be the frame of the multiple frames of images with the highest image quality.

[0055] The electronic device may also obtain multiple images from a storage medium, and the storage medium may be a memory of the electronic device itself, a mobile hard disk, etc. The electronic device may also download multiple images from the Internet.

[0056] After determining the reference image and the image to be registered, the electronic device can extract multiple third feature points from the reference image and multiple fourth feature points from the image to be registered; and determine the difference area and the same area between the image to be registered and the reference image, match the first feature point located in the same area among the multiple third feature points with the second feature point located in the same area among the multiple fourth feature points to obtain a feature point pair, and then perform subsequent operations on the matched feature point pair.

[0057] S202, extracting a plurality of third feature points from the reference image, and extracting a plurality of fourth feature points from the image to be registered.

[0058] A feature refers to the ability to represent a scene or target in a representative form in multiple similar images containing the same scene or target. The above-mentioned feature points mainly refer to obvious points in the image, such as prominent corner points, edge endpoints, extreme points, etc. In the embodiment of the present application, corner points are selected as feature points. In real life, corner points correspond to the corners of tables, buildings, etc., crossroads of roads, T-junctions, etc. In the field of image processing, corner points have the following two definitions: (1) Corner points can be the intersection of two or more edges, such as the pixel points corresponding to the corners of an object; (2) Corner points can be the pixel points in the image where the first-order derivative is the largest and the second-order derivative is zero, indicating the place where the edge of the object changes discontinuously. While retaining the important features of the image graphics, corner points can effectively reduce the amount of information data, effectively improve the speed of calculation, and facilitate reliable matching of images.

[0059] The electronic device may use an algorithm to extract feature points. Specifically, the electronic device may use different corresponding algorithms to extract feature points containing corresponding characteristics according to different types of feature points. In the embodiment of the present application, corner points are selected as feature points, and a corner point algorithm may be used to extract feature points, wherein the corner point algorithm includes: Harris corner point algorithm, Feature from Accelerated Segment Test (FAST) corner point algorithm, Scale-invariant feature transform (SIFT) corner point algorithm, or Speeded-Up Robust Features (SURF) algorithm, etc.

[0060] The following takes the Harris corner algorithm as an example to introduce the process of extracting feature points in detail. Figure 3 A schematic diagram of a set of corner point feature extraction methods provided in the present application is exemplified.

[0061] Since the average brightness changes dramatically when the corner moves in any direction, the basic idea of ​​the Harris corner detection algorithm is to use a fixed-size window (i.e., a neighborhood window of a certain pixel) to slide in any direction on the image, and compare the grayscale changes of the pixels framed by the window before and after the sliding. If the grayscale of the pixels in the window does not change when the window moves in all directions, then there is no corner in the window. If the window slides in any direction and the pixels framed by the window have a large grayscale change, then it is considered that there is a corner in the window.

[0062] refer to Figure 3 , Figure 3 (a) in the figure only shows a small area in the reference image and a window for extracting corner points in the reference image. The initialization position of the window is, for example, the center of the reference image. After that, the window slides in any direction on the reference image based on the initialization position, and the grayscale change degree of the pixel framed by the window before and after the sliding is compared. This process continues until all pixels in the reference image are traversed to obtain all corner points of the reference image.

[0063] like Figure 3 As shown in (b) in the figure, assuming that the window slides a certain number of pixels from the current position (solid line position in the figure) to any surrounding direction (dashed line position in the figure), and the grayscale of the pixels in the window does not change, it is considered that there are no corner points in all the pixels at the solid line position of the window frame.

[0064] like Figure 3 As shown in (c) in the figure, assuming that the window slides a certain number of pixels from the current position (solid line position in the figure) to any surrounding direction (dashed line position in the figure), the grayscale of the pixels in the window changes in multiple directions, then it is considered that there are corner points in all the pixels at the solid line position of the window frame.

[0065] based on Figure 3 The specific implementation process of the Harris corner algorithm introduced above can obtain all corner points contained in the reference image and the image to be registered, that is, extract multiple third feature points from the reference image and extract multiple fourth feature points from the image to be registered. Specifically, the electronic device can obtain the position coordinates of each third feature point in the reference image, recorded as (x, y); the electronic device can also obtain the position coordinates of each fourth feature point in the image to be registered, recorded as (x', y').

[0066] For example, Figure 4 FIG. 4 shows a plurality of third feature points in the reference image. Figure 4As shown, the third feature points included in the reference image include: third feature point 1, third feature point 2, third feature point 3, third feature point 4, third feature point 5 and third feature point 6. Moreover, the third feature points 1 to 6 may be located in the difference area of ​​the reference image, or in the same area of ​​the reference image. The third feature points 1 to 6 may be located in the texture area of ​​the reference image, or in the non-texture area of ​​the reference image.

[0067] Understandably, Figure 4 Only some of the third feature points included in the reference image are shown as examples, and the reference image may include more third feature points. Figure 4 It should not constitute a limitation to the present application. Similarly, the schematic diagram of the fourth feature point extracted from the image to be registered includes the fourth feature point 1 to the fourth feature point 6 and Figure 4 Similar, omitted here.

[0068] S203, determining the difference area and the same area between the image to be registered and the reference image.

[0069] Due to the movement of the photographed object, shaking during shooting, and other factors, there are differences between the image to be registered and the reference image, for example, the texture area in the two images changes. In an embodiment of the present application, the difference area in the two images is first detected based on the image content, and then the feature points in the difference area are eliminated, that is, the feature points are screened. Among them, the same area means that there is no difference between the change pattern of the pixels of the image to be registered in this area and the change pattern of the pixels of the reference image in this area, and the change of the pixels includes the change of the grayscale value of the pixels. No difference means that the similarity is greater than a preset threshold, for example, the similarity is greater than 98%. The same area corresponds to the aforementioned first area. Correspondingly, the difference area means that there is a difference between the change pattern of the pixels of the image to be registered in this area and the change pattern of the pixels of the reference image in this area.

[0070] When detecting different areas in two images, a quantitative index is needed to measure the difference between the two images. In the embodiment of the present application, relative texture difference is used to represent the quantitative index. Step S203 may specifically include S2031 to S2032. S2031 to S2032 are described in detail below.

[0071] S2031, calculate texture complexity.

[0072] The electronic device first calculates the standard deviation of the grayscale values ​​of pixels in the local area img of the image, as shown in the following formula (1).

[0073]

[0074] Where σ can be X or Y, X(i,j) represents the local standard deviation of the position (i,j) in the reference image, and Y(i,j) represents the local standard deviation of the position (i,j) in the image to be registered. The local window is a window centered at (i,j), and n represents the size of the local window. For example, n=11 represents a window of size 11×11. The local standard deviation can be calculated by the grayscale values ​​of the pixels in the window. sd(.) represents the standard deviation operator. The local standard deviation of the position (i,j) in the image can represent the texture complexity of the position (i,j) in the image.

[0075] The local standard deviation of each pixel in the reference image and the local standard deviation of each pixel in the image to be registered can be obtained by formula (1), that is, the texture complexity of the reference image and the texture complexity of the image to be registered can be obtained.

[0076] In other implementations, the texture complexity of each pixel in the reference image and the texture complexity of each pixel in the image to be registered may also be calculated by the global standard deviation. For the reference image, although the texture complexity of each pixel calculated by the global standard deviation and the texture complexity of each pixel calculated by the local standard deviation may be different, the relativity between the texture complexities of multiple pixels is consistent.

[0077] S2032, calculate relative texture difference.

[0078] Then, the difference between the two local standard deviations at the position (i, j) in the reference image and the image to be registered is calculated as shown in the following formula (2). The difference between the standard deviations can measure the texture difference d between the image to be registered and the reference image. The larger the texture difference d, the more likely it is that there is a difference between the two images. The difference value between the pixels at the same position in the reference image and the image to be registered can be obtained through formula (2).

[0079] d(X,Y)=(XY) 2 (2)

[0080] The above texture difference d represents the absolute difference between the local areas of the two images, which is strongly related to the texture complexity of the image itself. When facing similar d, blocks with different semantics have completely different visual perceptions. In order to more accurately measure the difference between the image to be registered and the reference image, the electronic device further calculates the relative texture difference D, as shown in the following formula (3). The relative texture difference D is a relative indicator independent of the original texture change and can reflect the signal difference between images.

[0081]

[0082] In some implementations, the relative texture difference D may be normalized to [0, 1], as shown in equation (4).

[0083]

[0084] Where X is X(i, j), Y is Y(i, j), and C is a constant. Introducing the constant C can stabilize the division operation and handle the case where the denominator is relatively small, thereby increasing the robustness of the calculation result.

[0085] The texture difference map is obtained by calculating the relative texture difference D of each pixel. The texture difference map can be used to determine the same area and the different area between the reference image and the image to be registered. Specifically, the grayscale value (D value) corresponding to each pixel in the texture difference map obtained by the relative texture difference D is between 0-1 (calculated by formula (4)), or the corresponding D value is between 0-255 (calculated by formula (3)). The area where the pixel points with a D value of 0 in the texture difference map are located is the same area between the reference image and the image to be registered, and the area where the pixel points with a D value not equal to 0 in the texture difference map are located is the different area between the reference image and the image to be registered. The D value corresponds to the aforementioned relative difference value.

[0086] Optionally, step S203 may further include: S2033, binarizing the texture difference map.

[0087] Different semantic regions have different tolerances for texture differences. In some scenarios, some pixels in the texture difference map whose D values ​​are not 0 also belong to the same area between the reference image and the image to be registered. The electronic device can binarize the texture difference map by threshold segmentation, and use the binarized difference map to determine the same area and the different area between the reference image and the image to be registered. Binarization specifically refers to converting the texture difference map into a binary image with only pixel values ​​of 0 and 1, or into a binary image with only 0 and 255.

[0088] Taking a binary image of 0 and 1 as an example, the main method of binarization is to first obtain a first threshold, and then adjust the pixel grayscale values ​​greater than the first threshold to 1 in the pixel grayscale value matrix corresponding to the texture difference map, and adjust the pixel grayscale values ​​less than or equal to the first threshold to 0. The first threshold can be determined according to the value of each pixel in the texture difference map. For example, the D value of each pixel is arranged in descending order, and the D value at the 85% ratio is set as the first threshold. For another example, the D value of each pixel is arranged in descending order, and the D value at the median is set as the first threshold. For another example, the average value of the D value of each pixel is calculated, and the average value is set as the first threshold.

[0089] Figure 5The figure shows the binary difference map obtained based on the reference image and the image to be registered, where the black area is the area with a pixel value of 0, and the white area is the area with a pixel value of 1. That is to say, the black area is the same area between the reference image and the image to be registered, and the white area is the difference area between the reference image and the image to be registered.

[0090] Optionally, step S203 may further include: S2034, performing a morphological operation on the binary difference map.

[0091] Because of the positional differences between the reference image and the image to be registered, there are some discrete points in the binary difference map. The electronic device can also perform opening and closing operations in morphological operations on the existing binary difference map to remove or connect discrete points, so that the binary difference map is converted into a more precise and accurate binary image with better connectivity, which is conducive to improving the accuracy of the registration matrix. For example, see Figure 6 , Figure 6 (b) shows the binary image after morphological operation. Figure 6 From the binary difference map before processing shown in (a), it can be clearly seen that some disconnected places are connected (as shown in the boxed area 61) and some discrete points are removed (as shown in the boxed area 62).

[0092] The above morphological operation processing steps can be as follows: first, use a 5×5 matrix of all 1s to perform an erosion operation; then, use a 5×5 matrix of all 1s to perform an expansion operation to connect discontinuous areas; second, use a 3×3 matrix of all 1s to perform expansion to fill holes; finally, filter out smaller discrete areas. In other implementations, different morphological operation processing steps can be used, which can be determined according to the actual image content.

[0093] Through step S203, the reference image can be divided into two parts: the same area and the different area, and the image to be registered can also be divided into two parts: the same area and the different area, to further determine whether each third feature point and each fourth feature point are located in the same area.

[0094] S204, determining whether each third feature point is located in the same area of ​​the reference image, and whether each fourth feature point is located in the same area of ​​the image to be registered.

[0095] When detecting feature points, the position coordinates of the feature points are determined. The sizes of the reference image, the image to be registered, and the binary image are all m×m. In one implementation, the position coordinates of the feature points can be mapped to the binary difference map / binary image. According to the range of the black area in the binary difference map / binary image, it is determined whether each third feature point is located in the same area of ​​the reference image and whether each fourth feature point is located in the same area of ​​the image to be registered. If the third feature point is located in the black area, it means that the third feature point is located in the same area of ​​the reference image, and the third feature point located in the same area is retained; if the third feature point is located in the white area, it means that the third feature point is located in the difference area of ​​the reference image, and the third feature point located in the difference area is removed. Similarly, if the fourth feature point is located in the black area, it means that the fourth feature point is located in the same area of ​​the image to be registered, and the fourth feature point located in the same area is retained; if the fourth feature point is located in the white area, it means that the fourth feature point is located in the difference area of ​​the image to be registered, and the fourth feature point located in the difference area is removed.

[0096] Figure 7 The first feature point obtained after screening is shown as an example. Figure 4 Among the multiple feature points shown, the third feature points 1 to 5 are located in the black area, and the third feature point 6 is located in the white area. Therefore, the third feature points 1 to 5 are retained, and the third feature point 6 is removed. Figure 7 As shown, the first feature points obtained through screening include the third feature point 1 to the third feature point 5. Similarly, the second feature points obtained through screening include the fourth feature point 1 to the fourth feature point 5.

[0097] In other implementations, it is possible to determine whether the third feature point is located in the same area of ​​the reference image based on the value of the third feature point. Specifically, it is determined whether the value corresponding to the third feature point in the binary difference map / binary image is 0. If it is 0, the third feature point is located in the same area of ​​the reference image; if it is 1, the third feature point is located in the difference area of ​​the reference image. The same is true for the fourth feature point.

[0098] In other implementations, in step S203, only the relative difference value corresponding to each third feature point is calculated, and the electronic device determines whether the relative difference value corresponding to the third feature point is less than or equal to a first threshold. If it is less than or equal to the first threshold, it is determined that the third feature point is located in the same area of ​​the reference image; if it is greater than the first threshold, it is determined that the third feature point is located in a different area of ​​the reference image.

[0099] S205: remove the third feature point located in the difference area in the reference image and the fourth feature point located in the difference area in the image to be registered.

[0100] S206, retaining the third feature points located in the same area in the reference image and the fourth feature points located in the same area in the image to be registered, and calculating a registration matrix based on the retained third feature points and fourth feature points.

[0101] The third feature point located in the same area of ​​the reference image is the first feature point; the fourth feature point located in the same area of ​​the image to be registered is the second feature point. After obtaining the first feature point located in the same area in the reference image and the second feature point located in the same area in the image to be registered, the electronic device can use a feature matching method to match the first feature point and the second feature point to obtain a one-to-one correspondence between the first feature point and the second feature point, and the corresponding two feature points are a set of feature point pairs. Common feature matching methods include matching based on descriptor distance, matching based on similarity measurement, etc., which are not limited in this application.

[0102] like Figure 8 As shown, the first feature point 1 in the reference image matches the second feature point 1 in the image to be registered; the first feature point 2 in the reference image matches the second feature point 2 in the image to be registered; the first feature point 3 in the reference image matches the second feature point 3 in the image to be registered; the first feature point 4 in the reference image matches the second feature point 4 in the image to be registered; the first feature point 5 in the reference image matches the second feature point 6 in the image to be registered.

[0103] Based on the correspondence between the first feature point and the second feature point, the electronic device can calculate the transformation relationship between the reference image and the image to be registered, and perform image registration through the transformation relationship. Affine transformation is a transformation relationship commonly used in image registration. The meaning of affine transformation is: if two images describe the same object, due to the differences in image size, position and angle, the position information of the corresponding feature points in the two images should satisfy a certain affine transformation relationship. Therefore, when registering images, the affine transformation matrix can be estimated by the matching feature point pairs between the two images, and an affine transformation that is obeyed by enough feature point pairs can be found.

[0104] Specifically, the affine transformation relationship between the first feature point (x, y) and the second feature point (x', y') can be expressed as the following formula (5).

[0105]

[0106] in, is a 3×3 affine transformation matrix, through which the second feature point is transformed to the first feature point, where the affine transformation matrix is ​​the registration matrix. The constraints of transformation parameters a, b, c, d, e, and f under different basic transformations (including translation, rotation, etc.) are different. Substituting each set of feature point pairs into formula (5), the affine transformation matrix can be calculated.

[0107] S207: Register the reference image and the image to be registered according to the registration matrix.

[0108] The electronic device remaps the pixel points in the image to be registered according to the affine transformation matrix, maps the points in the image to be registered to the reference image, and realizes image registration. When there are multiple frames of images to be registered, each frame of the image to be registered corresponds to an affine transformation matrix, and each image to be registered is registered with the reference image. In the photo shooting scene, after the reference image and the image to be registered are registered, the image to be registered can be fused with the reference image to obtain the final captured image.

[0109] The above technical solution takes into account the texture richness of the image and the texture difference between frames, calculates the relative texture difference using the image content, screens the feature points through the relative texture difference, and screens out the feature points in the difference area between images, that is, retains the feature points located in the black area of ​​the difference map, and removes the feature points located in the white area of ​​the difference map. Then, the screened feature points are matched to obtain feature point pairs, thereby avoiding inaccurate matching of feature point pairs, improving the calculation accuracy of the registration matrix, improving the algorithm effect, and thus improving the registration accuracy between images, reducing the blurring phenomenon of local motion areas in the fused image after registration, improving the clarity of image texture details, and improving the image quality of the output image.

[0110] In other implementations, the electronic device can extract multiple third feature points in the reference image through the above-mentioned Harris corner algorithm in step 202, and then perform a quick search and match at the same position in the image to be registered based on the position coordinates of each third feature point in the reference image, and quickly search and converge to the best matching position to find the fourth feature point corresponding to each third feature point in the reference image in the image to be registered, and obtain a matched feature point pair. After determining the difference map in step 203, determine whether the feature point pair is located in the black area, retain the feature point pair located in the black area; remove the feature point pair located in the white area. Specifically, determine the fourth feature point located in the white area of ​​the image to be registered, and remove the feature point pair where the fourth feature located in the white area of ​​the image to be registered is located. Alternatively, determine the third feature point located in the white area of ​​the reference image, remove the feature point pair where the third feature located in the white area of ​​the reference image is located, and then calculate the registration matrix based on the screened feature point pairs.

[0111] In other implementations, the electronic device can extract multiple third feature points from the reference image by the Harris corner algorithm in step 202, and after determining the difference map in step 203, determine whether the multiple third feature point pairs are located in the black area, retain the third feature points located in the black area, and remove the third feature points located in the white area to obtain the first feature points. Then, for the first feature points after screening, find the corresponding second feature points in the image to be registered, obtain matching feature point pairs, and calculate the registration matrix based on the feature point pairs.

[0112] In other implementations, after acquiring the reference image and the image to be registered, the electronic device first performs S203 to determine the same area and the different area, and the specific determination process refers to the above content. Then, a plurality of first feature points are extracted from the same area in the reference image, and a plurality of second feature points are extracted from the same area in the image to be registered, and the plurality of first feature points and the plurality of second feature points are matched to obtain feature point pairs, and a registration matrix is ​​calculated based on the feature point pairs.

[0113] Fig. 9 A schematic diagram of a flow chart of another image processing method provided in an embodiment of the present application is shown. The method is executed by an electronic device, or by a chip configured in the electronic device. Fig. 9 The image processing method shown includes the following S901 to S911.

[0114] S901, obtaining an image to be registered and a reference image.

[0115] S902, extracting a plurality of third feature points from the reference image, and determining a plurality of fourth feature points in the image to be registered, to obtain matching feature point pairs.

[0116] Electronic devices can be Figure 3 The Harris corner algorithm shown extracts multiple third feature points in the reference image, and then performs a fast search and match at the same position in the image to be registered based on the position coordinates of each third feature point in the reference image. The fast search converges to the best matching position, and finds the fourth feature point corresponding to each third feature point in the reference image in the image to be registered, thereby obtaining a matched feature point pair.

[0117] In other implementations, the electronic device may first extract multiple third feature points from the reference image, and extract multiple fourth feature points from the image to be registered, and then perform secondary screening on the third feature points and the fourth feature points, and then perform matching to obtain feature point pairs after the screening is completed.

[0118] S903: Determine the difference area and the same area between the image to be registered and the reference image.

[0119] For the specific content, please refer to the content of S203 above, which will not be repeated here.

[0120] S904, determining whether each third feature point is located in the same area of ​​the reference image.

[0121] For the specific content, please refer to the content of S204 above, which will not be repeated here.

[0122] S905: Eliminate the feature point pair corresponding to the third feature point located in the difference area.

[0123] That is, when the third feature point is located in the same area of ​​the reference image (black area), the third feature point is retained and the corresponding feature point pair is retained; when the third feature point is located in the different area of ​​the reference image (white area), the third feature point is eliminated and the corresponding feature point pair is eliminated.

[0124] S906: retain the feature point pair corresponding to the third feature point located in the same area.

[0125] S907, determining a texture area and a non-texture area in the reference image.

[0126] The texture area includes a plurality of regular or irregular texture units. The electronic device can calculate the local standard deviation of each pixel of the reference image according to the above formula (1) to obtain the texture complexity of each pixel of the reference image. The area where the pixels whose texture complexity is greater than or equal to the second threshold are located is determined as the texture area; the area where the pixels whose texture complexity is less than the second threshold are located is determined as the non-texture area, that is, the image content of the texture area in the reference image is more complex than that of the non-texture area.

[0127] The second threshold value can be determined according to the texture complexity of each point, for example, the texture complexity of each pixel point is arranged in descending order, and the texture complexity at the 60% ratio is set as the first threshold value. For another example, the texture complexity of each pixel point is arranged in descending order, and the texture complexity at the median is set as the first threshold value. For another example, the average value of the texture complexity of each pixel point is calculated, and the average value is set as the first threshold value.

[0128] S908: Determine whether the third feature point in the same area is in the texture area.

[0129] The third feature point located in the same area of ​​the reference image is the first feature point, and the electronic device performs secondary screening on the first feature points located in the same area according to the texture area and the non-texture area in the reference image. Specifically, the electronic device can determine whether the local standard deviation corresponding to the first feature point is greater than or equal to the second threshold value, and if it is greater than or equal to the second threshold value, it is determined that the first feature point is located in the texture area; if it is less than the second threshold value, it is determined that the first feature point is located in the non-texture area.

[0130] When the first feature point is located in the texture area of ​​the reference image, the first feature point is retained and the corresponding feature point pair is retained; when the first feature point is located in the non-texture area of ​​the reference image, the first feature point is removed and the corresponding feature point pair is removed. Fig.10 , the corner points extracted from the reference image according to the Harris corner algorithm include the third feature point 1 to the third feature point 6, wherein the third feature point 6 is located in the difference area, the third feature point 1 and the third feature point 2 are located in the same area and in the non-texture area, and the third feature point 3, the third feature point 4 and the third feature point 5 are located in the same area and in the texture area. Then, after the second screening, the first feature points retained in the reference image are the third feature point 3, the third feature point 4 and the third feature point 5.

[0131] S909: Eliminate the feature point pair corresponding to the third feature point located in the non-texture area.

[0132] S910: retain the feature point pairs corresponding to the third feature points located in the texture area, and calculate the registration matrix according to the retained feature point pairs.

[0133] S911, registering the reference image and the image to be registered according to the registration matrix.

[0134] Fig. 9 For details not covered in Figure 2 The content will not be repeated here.

[0135] It should be noted that in other implementations, the electronic device may first execute steps S907-S910, and then execute steps S903-S906, that is, first filter out the feature points located in the texture area of ​​the reference image, then eliminate the feature points located in the texture area and in the difference area, and filter out the first feature point located in the same area among the feature points located in the texture area, thereby obtaining the filtered feature point pairs.

[0136] In other implementations, the electronic device may first execute step S903 and step S907, and then determine whether the third feature point is located in the texture area and in the same area; if the third feature point is located in the texture area and in the same area, retain the third feature point; if the third feature point is located in the texture area and in the difference area, or if the third feature point is located in the non-texture area, eliminate the third feature point; thereby obtaining the filtered feature point pairs.

[0137] In other implementations, the electronic device can perform screening based on the fourth feature point, that is, S904-S910 can be replaced by: determining whether each fourth feature point is located in the same area of ​​the image to be registered; eliminating the feature point pairs corresponding to the fourth feature points located in the difference area; retaining the feature point pairs corresponding to the fourth feature points located in the same area. Determine the texture area and the non-texture area in the image to be registered. Determine whether the second feature point located in the same area is located in the texture area; eliminate the feature point pairs corresponding to the second feature point located in the non-texture area; retain the feature point pairs corresponding to the second feature point located in the texture area, and calculate the registration matrix based on the retained feature point pairs. The intersection of the same area and the texture area corresponds to the aforementioned second area.

[0138] The above technical solution, on the basis of improving the accuracy of the registration matrix, further takes into account the information of the texture area. By retaining the feature points at the intersection of the same area and the texture area and eliminating the feature points in the non-texture area, the distribution of feature points in the reference image is optimized, the proportion of feature point pairs in the texture area is increased, and the clarity of the texture area can be further improved.

[0139] Combination of the above Figures 1 to 10 The method provided in the embodiment of the present application is described in detail. Figures 11 to 14 The electronic equipment and devices provided in the embodiments of the present application are described in detail. Fig.11 A schematic diagram of the hardware structure of the electronic device 100 is shown.

[0140] The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0141] It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or separate some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0142] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0143] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0144] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0145] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0146] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present invention is only a schematic illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0147] The charging management module 140 is used to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 may receive charging input from a wired charger through the USB interface 130. In some wireless charging embodiments, the charging management module 140 may receive wireless charging input through a wireless charging coil of the electronic device 100. While the charging management module 140 is charging the battery 142, it may also power the electronic device through the power management module 141.

[0148] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and supplies power to the processor 110, the internal memory 121, the display screen 194, the camera 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle number, battery health status (leakage, impedance), etc. In some other embodiments, the power management module 141 can also be set in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.

[0149] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0150] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0151] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G applied to the electronic device 100. The modulation and demodulation processor may include a modulator and a demodulator. The wireless communication module 160 can provide solutions for wireless communications including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication technology (NFC), infrared technology (IR), etc. applied to the electronic device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0152] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0153] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0154] The electronic device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0155] ISP is used to process the data fed back by camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, ISP can be set in camera 193.

[0156] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0157] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the electronic device 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0158] Video codecs are used to compress or decompress digital videos. The electronic device 100 may support one or more video codecs. Thus, the electronic device 100 may play or record videos in a variety of coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0159] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0160] The internal memory 121 can be used to store computer executable program codes, which include instructions. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the electronic device 100 by running instructions stored in the internal memory 121, and / or instructions stored in a memory provided in the processor.

[0161] The electronic device 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0162] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be arranged in the processor 110, or some functional modules of the audio module 170 can be arranged in the processor 110.

[0163] The speaker 170A, also called a "speaker", is used to convert an audio electrical signal into a sound signal. The electronic device 100 can listen to music or listen to a hands-free call through the speaker 170A.

[0164] The receiver 170B, also called a "earpiece", is used to convert audio electrical signals into sound signals. When the electronic device 100 receives a call or voice message, the voice can be received by placing the receiver 170B close to the human ear.

[0165] Microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to microphone 170C to input the sound signal into microphone 170C. The electronic device 100 can be provided with at least one microphone 170C. In other embodiments, the electronic device 100 can be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the electronic device 100 can also be provided with three, four or more microphones 170C to collect sound signals, reduce noise, identify the sound source, realize directional recording function, etc.

[0166] The earphone interface 170D is used to connect a wired earphone and can be a USB interface 130 or a 3.5 mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0167] The pressure sensor 180A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be disposed on the display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc.

[0168] The gyro sensor 180B can be used to determine the motion posture of the electronic device 100. In some embodiments, the angular velocity of the electronic device 100 around three axes (i.e., x, y, and z axes) can be determined by the gyro sensor 180B. The gyro sensor 180B can be used for anti-shake shooting. For example, when the shutter is pressed, the gyro sensor 180B detects the angle of the electronic device 100 shaking, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to offset the shaking of the electronic device 100 through reverse movement to achieve anti-shake. The gyro sensor 180B can also be used for navigation and somatosensory game scenes.

[0169] The air pressure sensor 180C is used to measure air pressure. In some embodiments, the electronic device 100 calculates the altitude through the air pressure value measured by the air pressure sensor 180C to assist positioning and navigation.

[0170] The magnetic sensor 180D includes a Hall sensor. The electronic device 100 can use the magnetic sensor 180D to detect the opening and closing of the flip leather case. In some embodiments, when the electronic device 100 is a flip phone, the electronic device 100 can detect the opening and closing of the flip cover according to the magnetic sensor 180D. Then, according to the detected opening and closing state of the leather case or the opening and closing state of the flip cover, the flip cover can be automatically unlocked.

[0171] The acceleration sensor 180E can detect the magnitude of the acceleration of the electronic device 100 in all directions (generally three axes). When the electronic device 100 is stationary, the magnitude and direction of gravity can be detected. It can also be used to identify the posture of the electronic device and is applied to applications such as horizontal and vertical screen switching and pedometers.

[0172] The distance sensor 180F is used to measure the distance. The electronic device 100 can measure the distance by infrared or laser. In some embodiments, when shooting a scene, the electronic device 100 can use the distance sensor 180F to measure the distance to achieve fast focusing.

[0173] The proximity light sensor 180G may include, for example, a light emitting diode (LED) and a light detector, such as a photodiode. The light emitting diode may be an infrared light emitting diode. The electronic device 100 emits infrared light outward through the light emitting diode. The electronic device 100 uses a photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the electronic device 100. When insufficient reflected light is detected, the electronic device 100 can determine that there is no object near the electronic device 100. The electronic device 100 can use the proximity light sensor 180G to detect that the user holds the electronic device 100 close to the ear to talk, so as to automatically turn off the screen to save power. The proximity light sensor 180G can also be used in leather case mode and pocket mode to automatically unlock and lock the screen.

[0174] The ambient light sensor 180L is used to sense the ambient light brightness. The electronic device 100 can adaptively adjust the brightness of the display screen 194 according to the perceived ambient light brightness. The ambient light sensor 180L can also be used to automatically adjust the white balance when taking pictures. The ambient light sensor 180L can also cooperate with the proximity light sensor 180G to detect whether the electronic device 100 is in a pocket to prevent accidental touches.

[0175] The fingerprint sensor 180H is used to collect fingerprints. The electronic device 100 can use the collected fingerprint characteristics to implement fingerprint unlocking, access application locks, fingerprint photography, fingerprint call answering, etc.

[0176] The temperature sensor 180J is used to detect temperature. In some embodiments, the electronic device 100 uses the temperature detected by the temperature sensor 180J to execute a temperature processing strategy. For example, when the temperature reported by the temperature sensor 180J exceeds a threshold, the electronic device 100 reduces the performance of a processor located near the temperature sensor 180J to reduce power consumption and implement thermal protection. In other embodiments, when the temperature is lower than another threshold, the electronic device 100 heats the battery 142 to avoid abnormal shutdown of the electronic device 100 due to low temperature. In other embodiments, when the temperature is lower than another threshold, the electronic device 100 performs a boost on the output voltage of the battery 142 to avoid abnormal shutdown caused by low temperature.

[0177] The touch sensor 180K is also called a "touch control device". The touch sensor 180K can be set on the display screen 194. The touch sensor 180K and the display screen 194 form a touch screen, also called a "touch control screen". The touch sensor 180K is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K can also be set on the surface of the electronic device 100, which is different from the position of the display screen 194.

[0178] The bone conduction sensor 180M can obtain vibration signals. In some embodiments, the bone conduction sensor 180M can obtain vibration signals of the vibrating bones of the human body. The bone conduction sensor 180M can also contact the human pulse to receive blood pressure beating signals. In some embodiments, the bone conduction sensor 180M can also be set in the earphones to form bone conduction earphones.

[0179] The key 190 includes a power key, a volume key, etc. The key 190 may be a mechanical key or a touch key. The electronic device 100 may receive key input and generate key signal input related to user settings and function control of the electronic device 100.

[0180] Motor 191 can generate vibration prompts. Motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, audio playback, etc.) can correspond to different vibration feedback effects. For touch operations acting on different areas of the display screen 194, motor 191 can also correspond to different vibration feedback effects. Different application scenarios (for example: time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

[0181] Indicator 192 may be an indicator light, which may be used to indicate charging status, power changes, messages, missed calls, notifications, etc.

[0182] The SIM card interface 195 is used to connect a SIM card. The SIM card can be connected to or separated from the electronic device 100 by inserting it into or removing it from the SIM card interface 195. The electronic device 100 can support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The electronic device 100 interacts with the network through the SIM card to implement functions such as calls and data communications. In some embodiments, the electronic device 100 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the electronic device 100 and cannot be separated from the electronic device 100.

[0183] It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the mobile phone 100. In other embodiments of the present application, the mobile phone 100 may include more or fewer components than shown in the figure, or combine some components, or separate some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0184] The software system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present invention takes the Android system of the layered architecture as an example to illustrate the software structure of the electronic device 100. It should be noted that in the embodiment of the present application, the operator system of the electronic device may include but is not limited to (Symbian), (Android), (iOS), (Blackberry), Hongmeng (HarmonyOS) and other operating systems, this application does not make any limitation.

[0185] Fig.12 1 is a software structure block diagram of the electronic device 100 according to an embodiment of the present invention.

[0186] The layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into five layers, from top to bottom, namely, the application layer, the application framework layer, the Android runtime and system library, the hardware abstraction layer, and the kernel layer.

[0187] The application layer can include a series of application packages.

[0188] like Fig.12As shown, the application package may include camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message and other applications.

[0189] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0190] like Fig.12 As shown, the application framework layer may include a camera service, a window manager, a content provider, a view system, a telephony manager, a resource manager, a notification manager, and the like.

[0191] The camera service provides interfaces for managing and accessing cameras.

[0192] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0193] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0194] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.

[0195] The phone manager is used to provide communication functions of the electronic device 100, such as management of call status (including connecting, hanging up, etc.).

[0196] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0197] The notification manager enables applications to display notification information in the status bar. It can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as notifications of applications running in the background, or a notification that appears on the screen in the form of a dialog window. For example, a text message is displayed in the status bar, a prompt sound is emitted, an electronic device vibrates, an indicator light flashes, etc.

[0198] Android Runtime includes core libraries and virtual machines. Android runtime is responsible for scheduling and management of the Android system.

[0199] The core library consists of two parts: one part is the function that needs to be called by the Java language, and the other part is the Android core library.

[0200] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0201] The system library may include multiple functional modules, such as surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0202] The surface manager is used to manage the display subsystem and provide the fusion of 2D and 3D layers for multiple applications.

[0203] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0204] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0205] A 2D graphics engine is a drawing engine for 2D drawings.

[0206] The hardware abstraction layer is used to abstract the hardware. By calling the hardware abstraction layer interface in the hardware abstraction layer, the application layer and application framework layer above the hardware abstraction layer can be connected with the driver layer and hardware layer below, realizing the transmission of shooting data and function control.

[0207] The hardware abstraction layer includes a registration module and an image processing module. The registration module is used to filter feature points using a difference map. The image processing module is used to perform image processing, such as image fusion and image enhancement.

[0208] The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver, camera driver, audio driver, sensor driver, etc.

[0209] The following is an example of the workflow of the software of the electronic device 100 in conjunction with a photo-taking scenario.

[0210] When the user turns on the screen of the electronic device and controls the electronic device to be in an unlocked state, the electronic device can display a desktop. Icons of multiple installed applications are displayed on the desktop of the electronic device, including a camera application icon. The user can touch and click the camera application icon 10. The camera application calls the interface of the application framework layer to start the camera application, and then starts the camera driver by calling the kernel layer to capture static images or videos through the camera 193.

[0211] Taking the high dynamic range shooting mode as an example, the camera 193 acquires multiple frames of images for the same scene, and then sends the multiple frames of images to the registration module. The registration module uses one frame of the multiple frames as the reference image and the remaining frames as the images to be registered. Then the registration module extracts multiple first feature points in the reference image, and determines multiple second feature points in the image to be registered to obtain matching feature point pairs; determines the difference area and the same area between the image to be registered and the reference image; determines whether each first feature point is located in the same area of ​​the reference image; removes the feature point pairs corresponding to the first feature points located in the difference area; and retains the feature point pairs corresponding to the first feature points located in the same area. Then the registration module determines the texture area and the non-texture area in the reference image; determines whether the first feature point located in the same area is located in the texture area; removes the feature point pairs corresponding to the first feature points located in the non-texture area; retains the feature point pairs corresponding to the first feature points located in the texture area, and calculates the registration matrix based on the retained feature point pairs; after obtaining the registration matrix, the registration module registers the reference image and the image to be registered according to the registration matrix. The registration module sends the registration result to the image processing module, and the image processing module performs subsequent image processing procedures.

[0212] Fig.13 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application. The image processing device may be an electronic device in an embodiment of the present application, or a chip or chip system in an electronic device.

[0213] like Fig.13 As shown, the image processing apparatus 1300 may include: a processing unit 1301. The processing unit 1301 is used to support the image processing apparatus 1300 to execute the processing steps in the above method.

[0214] Specifically, the processing unit 1301 is used to obtain the image to be registered and the reference image, where the reference image and the image to be registered are obtained after photographing the same object; in the first area of ​​each of the reference image and the image to be registered, a plurality of feature point pairs are determined, and there is no difference between the change pattern of the pixels in the first area of ​​the image to be registered and the change pattern of the pixels in the first area of ​​the reference image, and the plurality of feature point pairs include a plurality of first feature points located in the first area of ​​the reference image, and each second feature point located in the first area of ​​the image to be registered and corresponding to each first feature point; based on the plurality of feature point pairs, a transformation relationship between the image to be registered and the reference image is determined; based on the transformation relationship, the image to be registered and the reference image are registered.

[0215] In some implementations, the apparatus 1000 further includes a storage unit 1302. The storage unit 1302 is connected to the processing unit 1301 via a line. The storage unit 1302 may include one or more memories, and the memory may be a device used to store programs or data in one or more devices or circuits. The storage unit 1302 may exist independently and be connected to the processing unit 1301 via a communication bus. The storage unit 1302 may also be integrated with the processing unit 1301.

[0216] The storage unit 1302 can store computer-executable instructions of the method in the electronic device so that the processing unit 1301 executes the method in the above embodiment. The storage unit 1302 can be a register, a cache, or a random access memory (RAM), etc., or a read-only memory (ROM) or other types of static storage devices that can store static information and instructions.

[0217] Fig.14 This is a schematic diagram of the structure of a chip provided in an embodiment of the present application. Fig.14 As shown, the chip 1400 includes one or more (including two) processors 1401 , a communication line 1402 and a communication interface 1403 . Optionally, the chip 1400 also includes a memory 1404 .

[0218] In some implementations, the memory 1404 stores the following elements: executable modules or data structures, or a subset thereof, or an extended set thereof.

[0219] The method described in the above embodiment of the present application can be applied to the processor 1401, or implemented by the processor 1401. The processor 1401 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor 1401 or an instruction in the form of software. The above processor 1401 can be a general-purpose processor (for example, a microprocessor or a conventional processor), a digital signal processor, an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates, transistor logic devices or discrete hardware components. The processor 1401 can implement or execute the disclosed methods, steps and logic block diagrams in the embodiments of the present application.

[0220] The steps of the method disclosed in the embodiment of the present application can be directly embodied as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. Among them, the software module can be located in a mature storage medium in the field such as a random access memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable read only memory (EEPROM). The storage medium is located in the memory 1404, and the processor 1401 reads the information in the memory 1404 and completes the steps of the above method in combination with its hardware.

[0221] The processor 1401 , the memory 1404 , and the communication interface 1403 may communicate with each other via the communication line 1402 .

[0222] In the above embodiments, the instructions stored in the memory for execution by the processor may be implemented in the form of a computer program product, wherein the computer program product may be pre-written in the memory, or may be downloaded and installed in the memory in the form of software.

[0223] The present application embodiment also provides a computer program product, which includes one or more computer instructions. When the computer program instructions are loaded and executed on the computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a server, a data center, etc. that includes one or more available media integrated. For example, the available medium can include a magnetic medium (e.g., a floppy disk, a hard disk or a tape), an optical medium (e.g., a digital versatile disc (digital versatile disc, DVD)), or a semiconductor medium (e.g., a solid state drive (solid state disk, SSD)), etc.

[0224] An embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory is used to store a computer program, and the processor is used to execute the computer program to perform the above method.

[0225] The embodiment of the present application provides a chip. The chip includes a processor, and the processor is used to call a computer program in a memory to execute the technical solution in the above embodiment. Its implementation principle and technical effect are similar to those of the above related embodiments, and will not be repeated here.

[0226] In the embodiments provided in the present application, it should be understood that the disclosed device / electronic device and method can be implemented in other ways. For example, the device / electronic device embodiments described above are only schematic, for example, the division of the modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms. In addition, each functional unit in each embodiment of the present application can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units.

[0227] The embodiment of the present application also provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned various method embodiments can be implemented. The computer-readable storage medium stores a computer program or instruction. The above-mentioned method is implemented when the computer program or instruction is executed by the processor. The method described in the above embodiment can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. If implemented in software, the function can be stored as one or more instructions or codes on a computer-readable medium or transmitted on a computer-readable medium. Computer-readable media may include computer storage media and communication media, and may also include any medium that can transfer a computer program from one place to another. The storage medium may be any target medium accessible by a computer.

[0228] As a possible design, the computer-readable medium may include a compact disc read-only memory (CD-ROM), RAM, ROM, EEPROM or other optical disc storage; the computer-readable medium may include a magnetic disk storage or other magnetic disk storage device. Moreover, any connecting line may also be appropriately referred to as a computer-readable medium. For example, if the software is transmitted from a website, server or other remote source using a coaxial cable, fiber optic cable, twisted pair, DSL or wireless technology (such as infrared, radio and microwave), the coaxial cable, fiber optic cable, twisted pair, DSL or wireless technology such as infrared, radio and microwave are included in the definition of the medium. Disks and optical discs as used herein include compact discs (CDs), laser discs, optical discs, DVDs, floppy disks and Blu-ray discs, where disks generally reproduce data magnetically, while optical discs reproduce data optically using lasers. Combinations of the above should also be included in the scope of computer-readable media.

[0229] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processing unit of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0230] In the above description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0231] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0232] It should also be understood that the "multiple" mentioned in the specification and the appended claims of this application refers to two or more. In the description of this application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is only a description of the association relationship of associated objects, which means any combination of one or more of the items listed in the association and all possible combinations, and includes these combinations, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0233] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0234] In addition, in order to clearly describe the technical solution of the present application, words such as "first" and "second" are used to distinguish between identical or similar items with substantially the same functions and effects. Those skilled in the art can understand that words such as "first" and "second" do not limit the quantity and execution order, and cannot be understood as indicating or implying relative importance, and words such as "first" and "second" do not necessarily limit differences.

[0235] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0236] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. An image processing method, characterized in that: Applied to electronic equipment, the method comprises: Acquire an image to be registered and a reference image, wherein the reference image and the image to be registered are acquired after photographing the same photographed object; In the first regions of the reference image and the image to be registered, a plurality of feature point pairs are determined, wherein a change rule of the pixels in the first region of the image to be registered is the same as a change rule of the pixels in the first region of the reference image, and the plurality of feature point pairs include a plurality of first feature points located in the first region of the reference image, and second feature points located in the first region of the image to be registered and corresponding to the first feature points; Determining a transformation relationship between the image to be registered and the reference image according to the plurality of feature point pairs; The image to be registered and the reference image are registered according to the transformation relationship.

2. The method according to claim 1, characterized in that The first area in the reference image includes a second area, and the second area includes at least one texture unit; the determining of a plurality of feature point pairs in the first area corresponding to the reference image and the image to be registered includes: In the second area in the reference image and the first area of ​​the image to be registered, a plurality of feature point pairs are determined, the plurality of feature point pairs comprising a plurality of first feature points in the reference image located in the second area, and second feature points in the image to be registered corresponding to the first feature points.

3. The method according to claim 2, characterized in that Determining a plurality of feature point pairs between the second area in the reference image and the first area of ​​the image to be registered comprises: Determine a plurality of third feature points from the reference image, and determine fourth feature points corresponding to the third feature points from the image to be registered; Determine a first region of each of the reference image and the image to be registered, and determine a second region of the reference image; Based on the third feature point located in the second area of ​​the reference image and the fourth feature point located in the first area of ​​the image to be registered, a plurality of feature point pairs are determined, wherein the plurality of first feature points include the third feature point located in the second area of ​​the reference image, and the second feature points include the fourth feature points located in the first area of ​​the image to be registered.

4. The method according to claim 2, characterized in that: Determining a plurality of feature point pairs between the second area in the reference image and the first area of ​​the image to be registered comprises: Determine a first region of each of the reference image and the image to be registered, and determine a second region of the reference image; A plurality of first feature points are determined in the second region of the reference image, and second feature points corresponding to the first feature points are determined from the first region of the image to be registered, to obtain a plurality of feature point pairs.

5. The method according to claim 3 or 4, characterized in that: Determining the first region of each of the reference image and the image to be registered includes: Determining a local standard deviation of each pixel in the reference image and a local standard deviation of each pixel in the image to be registered; Determine a relative difference value of each pixel in the image to be registered relative to each pixel in the reference image according to the local standard deviation of each pixel in the reference image and the local standard deviation of each pixel in the image to be registered; According to the relative difference value of each pixel in the image to be registered with respect to each pixel in the reference image, the first regions of the reference image and the image to be registered are determined.

6. The method according to claim 5, characterized in that The determining, according to the relative difference value of each pixel in the image to be registered with respect to each pixel in the reference image, the first region of each reference image and the image to be registered, comprises: Determine whether the relative difference value corresponding to each pixel point is less than or equal to a first threshold; The region where the pixel points whose relative difference value is less than or equal to the first threshold are located is determined as the first region corresponding to the reference image and the image to be registered respectively.

7. The method according to claim 5, characterized in that The determining, according to the relative difference value of each pixel in the image to be registered with respect to each pixel in the reference image, the first region of each reference image and the image to be registered, comprises: The values ​​of pixels whose relative difference values ​​are less than or equal to a first threshold are set to 0, and the values ​​of pixels whose relative difference values ​​are greater than the first threshold are set to 1, so as to obtain a binary texture difference map; Performing a morphological opening and closing operation on the binary texture difference map; The region where the pixel value is 0 in the binarized texture difference map after the morphological opening and closing operation is determined as the first region corresponding to the reference image and the image to be registered.

8. The method according to claim 5, characterized in that The method further comprises: Determining coordinates of the plurality of third feature points in the reference image; According to the coordinates of the plurality of third feature points in the reference image, it is determined whether the plurality of third feature points are located in the second area of ​​the reference image.

9. The method according to claim 5, characterized in that The determining of the second area of ​​the reference image comprises: Determine the local standard deviation corresponding to each pixel in the reference image; The region where the pixels whose local standard deviation in the first region of the reference image is greater than or equal to the second threshold are located is determined as the second region.

10. The method according to claim 4, characterized in that The determining of a plurality of first feature points comprises: A plurality of first feature points are determined according to the Harris corner algorithm.

11. The method according to claim 1, characterized in that The transformation relationship is an affine transformation.

12. The method according to claim 1, characterized in that The step of acquiring the image to be registered and the reference image comprises: Starting a camera application in the electronic device; In response to a user's shooting instruction, a plurality of frames of images are acquired, wherein the plurality of frames of images include the reference image and at least one frame of image to be registered.

13. An electronic device, characterized in that: include: one or more processors; one or more memories; The memory stores one or more programs, and when the one or more programs are executed by the processor, the electronic device executes the method according to any one of claims 1 to 12.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, which, when executed on a computer, enable the computer to execute the method according to any one of claims 1 to 12.

Images