Image processing method and device, electronic equipment and storage medium

CN119032375APending Publication Date: 2024-11-26BEIJING XIAOMI MOBILE SOFTWARE CO LTD
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202280004185.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2022-04-29
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

During the image alignment process, the existing technology is difficult to effectively handle the image differences under different image acquisition modules or parameters, especially in high dynamic range imaging and multi-frame image super-resolution, resulting in poor alignment results, especially when feature points are relatively small. In images with few or weak textures.

Method used

By acquiring two images containing the same target object, using the position correspondence between the reference grid and the target grid, the pixel position is adjusted in one direction, reducing the amount of position correspondence determination in the other direction, and improving the accuracy of pixel position alignment. degree and efficiency.

Benefits of technology

It achieves the alignment of target object pixel positions under different image acquisition conditions, improves the accuracy and efficiency of image alignment, reduces dependence on feature points, and is suitable for the fusion and processing of multiple images.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119032375A_ABST
    Figure CN119032375A_ABST
Patent Text Reader

Abstract

The embodiment of the invention provides an image processing method and device, electronic equipment and a storage medium. The method comprises the steps that a first image and a second image comprising the same target object are acquired; adjusting the position of a second pixel in the target object in the second image to be the same as the position of a first pixel in the target object in the first image in the first direction to obtain a third image; the first pixel and the second pixel are pixels at the same position in the target object; determining a target grid in the third image according to the reference grid in the first image; the target grid comprises a target position; the target pixel value of the pixel corresponding to the target position and the reference pixel value of the pixel corresponding to the reference position in the reference grid meet a preset condition; and adjusting the position of the second pixel in the third image to be the same as the position of the first pixel in the first image in the second direction according to the position corresponding relationship between the reference grid and the target grid. Through the method, the alignment effect of the target object in the first image and the second image is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Image processing method, device, electronic device and storage medium Technical Field

[0001] The present disclosure relates to the field of image processing but is not limited to the field of image processing, and in particular to an image processing method, device, electronic device, and storage medium. Background Art

[0002] With the increasing application of image processing technology, many devices now include image acquisition components. After capturing an image, these components must process the image to produce a satisfactory image. Different image acquisition components may produce different images. For example, when capturing the same target object in the same shooting scene, different image acquisition modules, or even the same image acquisition module with different configuration parameters, may produce different images of the target object.

[0003] Summary of the Invention

[0004] The present disclosure provides an image processing method, apparatus, electronic device, and storage medium.

[0005] According to a first aspect of an embodiment of the present disclosure, there is provided an image processing method, comprising: acquiring a first image and a second image; wherein the first image and the second image both include the same target object; in a first direction, adjusting the position of a second pixel in the target object in the second image to be the same as the position of a first pixel in the target object in the first image, to obtain a third image; wherein the first pixel and the second pixel are pixels at the same position in the target object; determining a target grid in the third image based on a reference grid in the first image; wherein the target grid comprises: a target position; a target pixel value of a pixel corresponding to the target position, which satisfies a preset condition with a reference pixel value of a pixel corresponding to a reference position in the reference grid; and according to the positional correspondence between the reference grid and the target grid, adjusting the position of the second pixel in the third image to be the same as the position of the first pixel in the first image in a second direction; wherein the second direction is perpendicular to the first direction.

[0006] In some embodiments, determining the target grid in the third image based on the reference grid in the first image includes: creating M*N reference grids of preset size in the first image; determining K target pixel values ​​in the third image that meet the preset conditions with the K reference pixel values ​​based on the reference pixel values ​​corresponding to the K reference positions in the reference grid; determining K target pixels corresponding to the target pixel values ​​in the third image based on the K target pixel values; and determining the target grid based on the K target pixels; M, N and K are all positive integers.

[0007] In some embodiments, the method further includes: determining a first average pixel value of each pixel in a preset area; wherein the preset area includes: an area centered on the pixel corresponding to the reference position and with a preset number of pixels as a radius; and determining the first average pixel value as the reference pixel value.

[0008] In some embodiments, the method of determining K target pixel values ​​in the third image that satisfy a preset condition with the K reference pixel values ​​based on the reference pixel values ​​of the pixels corresponding to the K reference positions in the reference grid includes: determining K alternative pixel values ​​in the third image; determining a second average pixel value of each pixel in the alternative area; wherein the alternative area includes: an area with the pixel corresponding to the alternative pixel value as the center and the preset number of pixels as the radius; and determining the alternative pixel values ​​corresponding to the K second average pixel values ​​that satisfy the preset condition with the K reference pixel values ​​as the target pixel values.

[0009] In some embodiments, the reference grid includes at least a square grid or a triangular grid; and the reference position includes at least a corner point of the grid or a midpoint of a sideline of the grid.

[0010] In some embodiments, the preset condition includes at least: the sum of the absolute values ​​of the differences between the reference pixel value and the target pixel value is less than a target threshold.

[0011] In some embodiments, based on the position correspondence between the reference grid and the target grid, the position of the second pixel in the third image is adjusted in the second direction to be the same as the position of the first pixel in the first image, including: determining the position correspondence based on the reference position and the target position; the position correspondence includes: a mapping transformation relationship in the second direction; based on the mapping transformation relationship in the second direction, adjusting the position of the second pixel located in the target grid in the third image to be the same as the position of the first pixel located in the reference grid in the first image.

[0012] In some embodiments, the mapping transformation relationship in the second direction includes: a homography transformation matrix in the second direction.

[0013] In some embodiments, in the first direction, the position of the second pixel in the target object in the second image is adjusted to be the same as the position of the first pixel in the target object in the first image, to obtain a third image, including: extracting a first feature point of the target object in the first image; extracting a second feature point of the target object in the second image; wherein the first feature point and the second feature point are a matching feature point pair at the same position in the target object; determining a mapping transformation relationship in the first direction based on the first feature point and the second feature point; based on the mapping transformation relationship in the first direction, in the first direction, adjusting the position of the second pixel in the target object in the second image to be the same as the position of the first pixel in the target object in the first image, to obtain a third image.

[0014] In some embodiments, the mapping transformation relationship in the first direction includes: a homography transformation matrix in the first direction.

[0015] A second aspect of an embodiment of the present disclosure provides an image processing device, comprising: an image acquisition module, configured to acquire a first image and a second image; wherein both the first image and the second image include the same target object; a first processing module, configured to adjust, in a first direction, the position of a second pixel in the target object in the second image to be the same as the position of a first pixel in the target object in the first image, to obtain a third image; wherein the first pixel and the second pixel are pixels at the same position in the target object; a target grid determination module, configured to determine, based on a reference grid in the first image, a target grid in the third image; wherein the target grid includes: a target position; a target pixel value of a pixel corresponding to the target position, which satisfies a preset condition with a reference pixel value of a pixel corresponding to a reference position in the reference grid; a second processing module, configured to adjust, in a second direction, the position of the second pixel in the third image to be the same as the position of the first pixel in the first image based on the positional correspondence between the reference grid and the target grid; wherein the second direction is perpendicular to the first direction.

[0016] A third aspect of the present disclosure provides an electronic device, including:

[0017] A processor and a memory for storing executable instructions that can be run on the processor, wherein: when the processor is used to run the executable instructions, the executable instructions execute the method described in any one of the above embodiments.

[0018] According to a fourth aspect of the embodiments of the present disclosure, a non-temporary computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, implement the method described in any of the above embodiments.

[0019] The technical solutions provided by the embodiments of the present disclosure may have the following beneficial effects:

[0020] The technical solution of the disclosed embodiment adjusts the position of the second pixel of the target object in the second image to be the same as the position of the first pixel in the first image in one direction. Since the first and second images are two-dimensional images, the pixel positions can be determined in both directions. The adjusted second image is then further adjusted in the other direction based on the positional relationship between a preset grid and the target grid. Since the grid includes multiple pixels, this reduces the need to determine the positional correspondence of individual pixels in the other direction, thereby reducing the computational effort required to adjust the pixel position of the target object in the other direction.

[0021] It should be understood that the foregoing general description and the following detailed description are merely exemplary and explanatory and are not restrictive of the embodiments of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present invention and, together with the description, serve to explain the principles of the embodiments of the present invention.

[0023] FIG1 is a schematic flow chart showing an image processing method according to an exemplary embodiment;

[0024] FIG2 is a schematic diagram showing a first image and a second image according to an exemplary embodiment;

[0025] FIG3 is a schematic diagram showing a method of adjusting a second pixel position in a first direction according to an exemplary embodiment;

[0026] Part (a) of FIG4 is a schematic diagram showing a first image according to an exemplary embodiment;

[0027] Part (b) of FIG4 is a schematic diagram showing a second image according to an exemplary embodiment;

[0028] FIG5 is a schematic diagram showing an adjustment in a first direction according to an exemplary embodiment;

[0029] FIG6 is a schematic diagram showing a method of determining a target grid according to an exemplary embodiment;

[0030] Part (a) of FIG7 is a schematic diagram showing a first image including a reference grid according to an exemplary embodiment;

[0031] Part (b) of FIG7 is a schematic diagram showing a second image including a target grid according to an exemplary embodiment;

[0032] FIG8 is a schematic diagram showing another image processing method according to an exemplary embodiment;

[0033] FIG9 is a schematic diagram showing a method of determining a target pixel value according to an exemplary embodiment;

[0034] FIG10 is a schematic diagram showing an adjustment in a second direction according to an exemplary embodiment;

[0035] FIG11 is a schematic diagram of an image processing apparatus according to an exemplary embodiment;

[0036] FIG12 is a schematic diagram showing another image processing method according to an exemplary embodiment;

[0037] Fig. 13 is a block diagram showing a terminal device according to an exemplary embodiment. DETAILED DESCRIPTION

[0038] Exemplary embodiments are described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all possible implementations consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with certain aspects of the present invention.

[0039] The terms used in the embodiments of the present disclosure are for the purpose of describing specific embodiments only and are not intended to limit the embodiments of the present disclosure. The singular forms "a," "the," and "the" used in the present disclosure are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0040] It should be understood that although the terms first, second, third, etc. may be used to describe various information in the embodiments of the present disclosure, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of the embodiments of the present disclosure, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0041] In image processing, such as high dynamic range (HDR) imaging, multi-frame super-resolution, and image fusion, it is necessary to fuse the image information of multiple images to obtain an image with improved image quality. In this process, multiple images need to be aligned first to achieve alignment between image pixels before subsequent processing can be performed.

[0042] Typically, image pixel alignment can be achieved through dense alignment methods. These methods are often based on dense optical flow, which requires determining pixel correspondences in two dimensions. Due to performance limitations, most methods can only handle images with relatively low resolutions. Furthermore, these methods require that the distance between initially misaligned pixels be within a certain range. Therefore, alignment is less effective when the pixels in multiple images differ significantly, such as when there is a significant field of view change.

[0043] Another approach is grid alignment, which is mostly used in image stitching. This method uses the correspondence between feature points in the image as a constraint to form a grid. The grid is determined based on the characteristic feature points, and the movement of the grid points is constrained to achieve image alignment. This method requires feature points to be distributed across the entire image, so the alignment effect is poor when aligning images with fewer feature points. For example, it is difficult to achieve good image alignment results in images with weak textures, where it is difficult to extract image feature points.

[0044] Referring to FIG1 , there is shown a flowchart of an image processing method provided by an example of the present disclosure. The method includes:

[0045] Step S100: Acquire a first image and a second image; wherein both the first image and the second image include the same target object.

[0046] In step S200, the position of the second pixel in the target object in the second image is adjusted in the first direction to be the same as the position of the first pixel in the target object in the first image, thereby obtaining a third image; wherein the first pixel and the second pixel are pixels at the same position in the target object.

[0047] Step S300, determining a target grid in the third image based on the reference grid in the first image; wherein the target grid includes: a target position; a target pixel value of a pixel corresponding to the target position, and a reference pixel value of a pixel corresponding to a reference position in the reference grid that meets a preset condition.

[0048] Step S400: Based on the positional relationship between the reference grid and the target grid, in a second direction, the position of a second pixel located in the target grid in the third image is adjusted to be the same as the position of a first pixel located in the reference grid in the first image; wherein the second direction is perpendicular to the first direction.

[0049] The method of the disclosed example can be applied at least in a user terminal, that is, the execution subject of the method can at least include a user terminal, which can include a mobile terminal and a fixed terminal. Mobile terminals can include mobile phones, tablet computers, in-vehicle central control devices, wearable devices, smart devices, etc. Smart devices can include smart office equipment and smart home devices, etc.

[0050] The first and second images acquired in step S100 may be two-dimensional images, each containing at least one identical target object. In other words, the first and second images contain shared image information and are not identical. The target object can be determined based on the actual image capture scenario; for example, it can be an object in any scene, such as a person, an animal, a plant, a mountain, or a river.

[0051] The first image and the second image may be images obtained with different image acquisition parameters, or may be images acquired by the same image acquisition module under different image acquisition parameters, including images of the same target object. The image acquisition parameters may include field of view angle and focal length, etc.

[0052] The first image and the second image may also be images of the same target object captured by different image acquisition modules. For example, the first image acquisition module and the second image acquisition module may respectively capture images of the same scene under different image acquisition parameters, and the scene includes the target object.

[0053] The image acquisition parameters can be determined based on the configuration parameters of the image acquisition module and the image acquisition angle, etc. The first image and the second image only need to include a portion of the same image information, that is, the first image and the second image have common image information, and this common image information is the target object.

[0054] For example, if the first image includes person A and tree B, and the second image includes person A and person C, then the target object is person A. The presentation effects of person A in the first image and person A in the second image may be different. For example, the magnification of person A in the first image is greater than the magnification of person A in the second image, and the position of person A in the first image is different from the position of person A in the second image.

[0055] Refer to Figure 2, which is a schematic diagram of a first image and a second image. Part (a) of Figure 2 is a schematic diagram of a target object, part (b) of Figure 2 is a schematic diagram of the first image, and part (c) of Figure 2 is a schematic diagram of the second image.

[0056] The target objects are trees and people, the first image and the second image are images of the target object at different magnifications, and the first image and the second image both include the same target object. For example, the first image may be an image of the target object acquired by an image acquisition module with a magnification of 1x, and the second image may be an image of the target object acquired by an image acquisition module with a magnification of 3x. The first image is an image of the target object at a magnification of 1x (1X), and the second image is an image of the target object at a magnification of 3x (3X). The first image includes more image information than the second image, that is, in addition to the image information included in the second image, the first image also includes image information that is not included in the second image.

[0057] In this embodiment, the image information included in the second image is the image information common to the first image and the second image. The content displayed in the second image is the target object, which includes trees and people, and also includes background images other than trees and people.

[0058] In step S200, after obtaining the first image and the second image, since any pixel in the image has a corresponding position, the position of the pixel in the image can be determined by the position in any two directions. For example, the position of any pixel in the first image in the first image can be determined by a coordinate system, and the position of any pixel in the second image in the second image can also be determined. Two directions can determine a coordinate system.

[0059] The first direction can be any direction, and the inclination angle between the first and second directions is not limited, as long as the first direction is perpendicular to the second direction. For example, the first direction is vertical and the second direction is horizontal, or the second direction is vertical and the first direction is horizontal, etc. When the first direction is perpendicular to the second direction, after adjusting the position of the second pixel in the first direction, adjusting the position of the second pixel in the second direction again will not change the position of the second pixel in the first direction, thereby facilitating the adjustment of the position of the second pixel in the second direction alone, and improving the accuracy of the adjustment of the position of the second pixel.

[0060] The pixel of the target object in the first image is the first pixel, and the pixel of the target object in the second image is the second pixel. For a certain position in the target object, the pixel corresponding to that position in the first image is the first pixel, and the pixel corresponding to that position in the second image is the second pixel. In other words, the first pixel and the second pixel are pixels at the same position in the target object. For a certain target position in the target object, the first pixel and the second pixel match, and both the first pixel and the second pixel are pixels at that target position.

[0061] For example, in the target object shown in FIG2 , the pixel of the tree tip in the first image is the first pixel, and the pixel in the second image is the second pixel.

[0062] The position of a second pixel of the target object in the second image is adjusted in the first direction to be the same as the position of a first pixel of the target object in the first image, and the adjusted second image is used as the third image. In the adjusted second image, each pixel of the target object is aligned in the first direction with the corresponding pixel of the target object in the first image, and the aligned first pixel and second pixel are positioned in the same position in the first direction.

[0063] For example, when the target object is the tree shown in Figure 2, the first direction is the vertical direction, and the position in the vertical direction is represented by y. The position of the pixel corresponding to the tree's tip in the first image in the first direction is y1, and the position of the pixel corresponding to the tree's tip in the second image in the first direction is y2. In the first direction, y2 is adjusted to be the same as y1. Similarly, the position of each pixel corresponding to the target object in the second image in the first direction is adjusted to be the same as the position in the first direction in the first image. In this way, the position of the corresponding pixel of the target object in the second image is aligned with the position of the corresponding pixel in the first image in the first direction, thereby aligning the position of the entire target object in the second image with the position in the first image.

[0064] In step S300, after the position of the second pixel in the first direction is aligned with the position of the first pixel, a target grid in the third image is determined based on the reference grid in the first image. The first image has a preset reference grid, which can also be called a reference window. The reference grid can be a virtual grid used to divide the first image into regions. The first image can have multiple reference grids, and the size and number of the reference grids can be determined based on actual usage requirements.

[0065] The reference grid may be in the shape of a square grid or a triangular grid, etc., and may be determined according to actual needs. For example, the first image may have N*N evenly distributed square grids, which may be positive direction grids. The reference grid divides the first image into N*N grid areas.

[0066] The reference grid includes reference positions, which can be corner points of the grid or midpoints of grid edges, and can be adjusted based on actual needs. The reference positions correspond to pixels in the first image, and the pixel values ​​of the pixels corresponding to the reference positions in the first image can be determined.

[0067] The target grid includes a target position, a target pixel value for a pixel corresponding to the target position, and a reference pixel value for a pixel corresponding to a reference position in the reference grid that satisfies a preset condition. The target grid can be determined based on the target position. The preset condition can be determined based on actual needs, for example, including at least: the sum of the absolute values ​​of the differences between the reference pixel values ​​and the target pixel values ​​being less than a target threshold.

[0068] Each reference grid corresponds to a target grid. This embodiment is described by taking one of the grids as an example. The corresponding target grid can be determined based on each reference grid.

[0069] Step S400: After determining the target grid in the third image, the position of the second pixel in the third image is adjusted in the second direction to be the same as the position of the first pixel in the first image according to the position correspondence between the reference grid and the target grid.

[0070] The position correspondence between the reference grid and the target grid can be determined based on the reference position and the target position. The detailed determination process is not limited. Please refer to the subsequent embodiments. It is sufficient to determine the position correspondence.

[0071] For example, when the target object is the tree shown in Figure 2, the second direction is the horizontal direction, and the horizontal position is represented by x. The position of the pixel corresponding to the tree's tip in the first image in the first direction is y1, and the position of the pixel corresponding to the tree's tip in the second image in the first direction is y2. In the first direction, y2 is adjusted to be the same as y1. Similarly, the position of each pixel corresponding to the target object in the second image in the first direction is adjusted to be the same as the position in the first direction in the first image. In this way, the position of the corresponding pixel of the target object in the second image is aligned with the position of the corresponding pixel in the first image in the first direction, thereby aligning the position of the entire target object in the second image with the position in the first image.

[0072] In this way, the position of the second pixel is adjusted in the first direction and the second direction to be the same as the position of the first pixel in the first image, so that the position of each pixel in the target object in the second image is the same as that in the first image. When the positions are the same, the target object in the second image is aligned with the target object in the first image.

[0073] On the one hand, the second pixel of the target object in the second image is aligned with the first pixel in the first image from the first direction. In this way, the first pixel and the second pixel are not aligned in the second direction. In this way, when determining the target grid through step S300, the pixel values ​​of the pixels corresponding to the target position in the second direction and the pixel values ​​of the pixels corresponding to the reference position can be considered separately, thereby improving the amount of calculation, efficiency and accuracy of determining the target grid.

[0074] On the other hand, the target grid is determined based on the pixel values ​​of the pixels corresponding to the reference positions in the reference grid, and then the pixel positions are adjusted in the second direction based on the positional correspondence between the target grid and the reference grid, thereby reducing the dependence on the feature points of the target object in the third image. There is no need to determine the target grid based on the positions of the feature points in the target object, nor is there any need to align in the second direction based on the positions of the feature points of the target object in the third image.

[0075] This reduces the chances of poor pixel alignment accuracy in the second direction when the target object's feature points cannot be extracted from the third image or when the target object's feature points are relatively few. It also improves the accuracy of pixel alignment in the second direction when the target object's feature points cannot be extracted from the third image or when the target object's feature points are relatively few, thereby improving the alignment of the target object.

[0076] In another embodiment, referring to FIG3 , which is a schematic diagram of adjusting the position of a second pixel in a first direction, step S200 adjusts the position of a second pixel in a target object in a second image in the first direction to be the same as the position of a first pixel in the target object in the first image, thereby obtaining a third image, including:

[0077] Step S201: extracting first feature points of a target object in a first image.

[0078] Step S202: Extract a second feature point of the target object in the second image. The first feature point and the second feature point are a pair of matching feature points at the same location in the target object. Steps S201 and S202 are not necessarily sequential and can be performed simultaneously or one of them can be performed first.

[0079] Step S203: determining a mapping transformation relationship in a first direction according to the first feature point and the second feature point.

[0080] Step S204 , according to the mapping transformation relationship in the first direction, adjust the position of the second pixel in the target object in the second image in the first direction to be the same as the position of the first pixel in the target object in the first image, to obtain a third image.

[0081] After obtaining the first image and the second image, feature points of the target object in the first image and the target object in the second image are extracted. The extracted feature points of the target object in the first image are recorded as first feature points, and the extracted feature points of the target object in the second image are recorded as second feature points. The first feature point and the second feature point are a pair of matching feature points corresponding to the same feature at the same location in the target object. For the same feature in the target object, if it is the first feature point in the first image and the second feature point in the second image, the first feature point and the second feature point form a matching pair of feature points.

[0082] Referring to Figure 4 , part (a) of Figure 4 is a schematic diagram of a first image, and part (b) of Figure 4 is a schematic diagram of a second image. The house and the person are the target objects. The dots on the house and the person shown in the first image are the first features, and the dots on the house and the person shown in the second image are the second features. The first feature and the second feature, which is a feature point pair matching the first feature, are connected by a dotted line. For a first feature point of the target object in the first image, there is a second feature point in the second image that matches the first feature point.

[0083] For example, when a person's foot is used as a feature point, the feature point of the foot in the first image is the first feature point, and the feature point of the foot in the second image is the second feature point. The first feature point and the second feature point, both of which are feature points of the foot, are connected by a dotted line. The first feature point and the second feature point, both of which are feature points of the foot, are considered a matching feature point pair. The same applies to the other feature points in Figure 4.

[0084] The feature point of the foot in the first image and the feature point of the person's hand in the second image are not a matching feature point pair. When the feature point of the foot in the first image is the first feature point, the feature point of the person's hand in the second image is not the second feature point that matches the first feature point.

[0085] There may be multiple first feature points and multiple second feature points, and the number of first feature points and second feature points may be the same. The method for extracting the first feature points and the second feature points can be determined based on business needs, and it is sufficient to extract the first feature points and the second feature points. For example, feature point extraction algorithms include the scale-invariant feature transform (SIFT) algorithm and the ORB (ORiented Brief) feature extraction algorithm.

[0086] In step S203 , a mapping transformation relationship in a first direction is determined based on the matched first feature point and second feature point.

[0087] After determining the first feature point and the second feature point, the position of the first feature point in the first image and the position of the second feature point in the second image can be obtained. According to the positions of multiple pairs of matching first feature points and second feature points, the mapping transformation relationship in the first direction can be determined.

[0088] For example, the mapping transformation relationship in the first direction can be determined based on the positions of the four pairs of matching first feature points and second feature points. The mapping transformation relationship can be a homography transformation matrix. The homography transformation matrix can be a 3*3 matrix and can be expressed as follows:

[0089]

[0090] H1 represents the homography transformation matrix in the first direction; h0 to h8 represent elements in H1.

[0091] The mapping transformation relationship may also be other forms of transformation matrices, which can adjust the position of the second pixel in the second image in the first direction according to the positions of the matched first feature points and the second feature points.

[0092] Step S204, after obtaining the mapping transformation relationship in the first direction, according to the mapping transformation relationship in the first direction, adjust the position of the second pixel in the target object in the second image in the first direction to the same as the position of the first pixel in the target object in the first image, to obtain a third image.

[0093] In another embodiment, the homography transformation matrix H1 in the first direction may also satisfy the following conditions:

[0094]

[0095] i represents the i-th pair of matched feature points, that is, the i-th first feature point and the i-th second feature point, x i Indicates the i For the coordinate of the first feature point in the second direction, y i represents the coordinate of the first feature point in the i-th pair of feature points in the first direction, y i ' represents the coordinate of the second feature point in the i-th pair of feature points in the first direction.

[0096] When H1 meets the condition, the position of the second pixel in the target object in the second image can be adjusted in the first direction to be the same as the position of the first pixel in the target object in the first image according to H1, thereby obtaining a third image.

[0097] In this embodiment, the first direction is the vertical direction.

[0098] When the first direction is the horizontal direction, the position in the first direction is represented by x, and the position in the second direction is represented by y. The homography transformation matrix H1 may also satisfy the following conditions:

[0099]

[0100] x i ' represents the coordinate of the second feature point in the i-th pair of feature points in the first direction.

[0101] Refer to Figure 5, which is a schematic diagram of alignment in a first direction. As shown in Figure 5, the positions of each pixel of the target object in the second image shown in part (b) of Figure 4 in the first direction are aligned with the positions of the corresponding pixels in the first image. The first direction is the vertical direction, and the pixels at corresponding positions of the target object in the first and second images are located on the same row, achieving row alignment.

[0102] In another embodiment, referring to FIG6 , which is a schematic diagram of determining a target grid, step S300 is to determine a target grid in a third image based on a reference grid in a first image, including:

[0103] Step S301: Create M*N reference grids of a preset size in a first image.

[0104] Step S302 : determining K target pixel values ​​in the third image that meet a preset condition with the K reference pixel values ​​based on the reference pixel values ​​of the pixels corresponding to the K reference positions in the reference grid.

[0105] Step S303 : determining K target pixels corresponding to the target pixel values ​​in the third image according to the K target pixel values.

[0106] Step S304: Determine a target grid based on K target pixels; M, N, and K are all positive integers. Through step S300, a target grid matching each reference grid can be determined.

[0107] In step S301, the reference grid in the first image can be pre-created or created during step S300. The size of the reference grid can be determined based on actual needs. For example, the size, shape, and number of the preset grids can be determined based on preset configuration parameters. M and N can be equal.

[0108] The reference positions and the number of reference positions in the reference grid may also be determined according to actual needs. For example, when the reference grid is a square, the reference positions are the four corner points of the reference grid, that is, it includes four reference positions.

[0109] 7 , part (a) of FIG. 7 is a first image including a reference grid, and part (b) of FIG. 7 is a second image including a target grid.

[0110] Part (a) of Figure 7 shows a reference grid. The positive grid consisting of four dots surrounding the person is one of these reference grids, and the locations of the four dots are the reference positions of this reference grid. For other reference grids, the four corner points of each reference grid are the reference positions. The number of reference grids is 5*5, and the first image is divided into 5*5 reference grids.

[0111] For step S302, one of the reference grids is used as an example for explanation. According to the reference pixel values ​​of the pixels corresponding to the K reference positions in the reference grid, K target pixel values ​​that meet preset conditions with the K reference pixel values ​​are determined in the third image.

[0112] After determining the reference grid and the reference positions within the grid, the pixels corresponding to the reference positions in the first image can be determined, and the pixel values ​​of the pixels corresponding to the respective reference positions in the first image can be determined. Then, based on a preset condition and the pixel values ​​of the pixels corresponding to the K reference positions in the first image, K target pixel values ​​that meet the preset condition with the K reference pixel values ​​can be determined in the third image.

[0113] When the preset conditions are different, the determined target pixel value may also be different. The preset conditions can be determined according to actual needs.

[0114] For step S303, after the K target pixel values ​​are determined, K target pixels corresponding to the K target pixel values ​​can be determined in the third image according to the K target pixel values. One target pixel value can correspond to one target pixel.

[0115] In step S304, after determining K target pixels, a target grid is determined based on the K target pixels; M, N, and K are all positive integers. The position of the target pixel in the third image is the target position in the target grid, and the target grid can be determined based on the target position. For example, one target pixel corresponds to one target position, and K target pixels correspond to K target positions. The area enclosed by the K target positions is determined as the target grid.

[0116] Referring to part (b) of FIG7 , the pixel values ​​of the pixels corresponding to the four dots around the character are the target pixel values, the pixels corresponding to the four dots are the target pixels, and the positive direction surrounded by the target pixels is the target grid.

[0117] For other reference grids, the process of determining the corresponding target grid is the same as the above process, that is, the above process is applicable to each reference grid.

[0118] In another embodiment, one target position corresponds to one reference position, and in the corresponding target position and reference position, the position of the target pixel is the same as the position of the reference pixel in the first direction.

[0119] Referring to Figure 7, the reference position corresponding to the upper left corner of the reference grid in Figure 7 (a) is denoted as the first reference position, and the reference position corresponding to the upper right corner is denoted as the second reference position. The position corresponding to the upper left corner of the target grid in Figure 7 (b) is denoted as the first target position, and the position corresponding to the upper right corner is denoted as the second target position. When the first direction is vertical, the first target position and the first reference position are identical in the first direction, i.e., they are aligned in the same row. The second target position and the second reference position are identical in the first direction, i.e., they are aligned in the same row.

[0120] This can reduce the situation where the target position and the reference position deviate simultaneously in the first direction and the second direction. After the positions in the first direction are the same, only one dimension of variable in the second direction remains, which can facilitate the adjustment of the position in the second direction, thereby facilitating the adjustment of the target position and the reference position in the first direction and the second direction to be the same.

[0121] In addition, the problem of large amount of calculation and low accuracy of position adjustment when executing step S400 caused by the corresponding target position and reference position being different in the first direction and the second direction is also reduced, thereby reducing the amount of calculation and improving the accuracy of position adjustment.

[0122] In another embodiment, referring to FIG8 , which is a schematic diagram of another image processing method, the method includes:

[0123] Step S10, determining a first average pixel value of each pixel in a preset area; wherein the preset area includes: an area centered on the pixel corresponding to the reference position and having a preset number of pixels as a radius;

[0124] Step S20: determining the first average pixel value as the reference pixel value.

[0125] When determining a reference pixel value, the reference pixel value is determined by combining the pixel value of the pixel corresponding to the reference position and the pixel values ​​of the pixels surrounding the pixel corresponding to the reference position. The reference position may correspond to a pixel in the first image. The pixel values ​​of each pixel in a predetermined area with a predetermined number of pixels as a radius, centered on the pixel corresponding to the reference position, are used as a basis. The average value of each pixel in the predetermined area is determined as the reference pixel. Here, the average value of each pixel in the predetermined area is recorded as the first average pixel value.

[0126] This can improve the accuracy of determining the target position and reduce the problem of low precision caused by determining the target pixel value based on the pixel value of a single pixel corresponding to each reference position.

[0127] In another embodiment, referring to FIG. 9 , which is a schematic diagram of determining a target pixel value, step S302 includes:

[0128] Step S3021: Determine K candidate pixel values ​​in the third image.

[0129] Step S3022, determining a second average pixel value of each pixel in the candidate area; wherein the candidate area includes: an area with the pixel corresponding to the candidate pixel value as the center and a preset number of pixels as the radius.

[0130] Step S3023 : determining the candidate pixel values ​​corresponding to the K second average pixel values ​​that satisfy the preset conditions of the K reference pixel values ​​as target pixel values.

[0131] When determining the target pixel value, K candidate pixel values ​​are first determined in the third image. Then, the average pixel value of the pixel values ​​of each pixel in the candidate area corresponding to each candidate pixel value is determined, i.e., the second average pixel value. Each second average pixel value corresponds to one candidate pixel value, and the corresponding K candidate pixel values ​​are determined based on the K second average pixel values. Based on a preset condition and the K reference pixel values, the candidate pixel values ​​corresponding to the K second average pixel values ​​that meet the preset condition with the K reference pixel values ​​are determined as the target pixel value.

[0132] This method improves the accuracy of determining the target pixel value, reduces the problem of low accuracy caused by determining the target pixel value based on the pixel value corresponding to a single pixel, thereby improving the accuracy of determining the target grid, and further improving the accuracy of adjusting the position of each pixel of the target object in the third image in the second direction to the same as the position of the corresponding pixel of the target object in the first image in the first image.

[0133] In another embodiment, referring to FIG10 , which is a schematic diagram of adjustment in the second direction, step S400, based on the positional correspondence between the reference grid and the target grid, adjusts the position of a second pixel located in the target grid in the third image to be the same as the position of a first pixel located in the reference grid in the first image in the second direction, including:

[0134] Step S401, determining a position correspondence relationship based on a reference position and a target position, where the position correspondence relationship includes: a mapping transformation relationship in a second direction;

[0135] Step S402 : According to the mapping transformation relationship in the second direction, in the second direction, the position of the second pixel located in the target grid in the third image is adjusted to be the same as the position of the first pixel located in the reference grid in the first image.

[0136] For each reference grid and the target grid that matches it, a positional correspondence can be determined. A reference grid and a matching target grid form a grid pair. This method can determine the positional correspondence of each grid pair, i.e., the mapping transformation relationship in the second direction.

[0137] In this embodiment, after the target grid is determined, a position correspondence in the second direction is determined based on the target position in the target grid and the reference position in the reference grid. The position correspondence in the second direction includes a mapping transformation relationship in the second direction. The mapping transformation relationship in the second direction can be a transformation relationship in the form of a homography transformation matrix or the like.

[0138] The position of the second pixel of the target object within the target grid in the third image is adjusted using the mapping transformation relationship in the second direction. After the adjustment, the position of the second pixel of the target object within the target grid in the third image becomes identical in the second direction to the position of the corresponding first pixel within the reference grid in the first image. This aligns the positions of each pixel of the target object within the target grid with the positions of the corresponding pixels of the target object within the reference grid. In other words, the positions of each pixel within the target grid are adjusted in the second direction based on the mapping transformation relationship in the second direction.

[0139] By using the target grids corresponding to the M*N reference grids, the position of each pixel of the target object located in the target grid in the second direction can be aligned with the position of the corresponding pixel of the target object in the reference grid in the second direction.

[0140] For example, the target grid corresponding to the four dots where the person is shown in part (b) of Figure 7, the positions of each pixel in the target grid in the second direction are aligned with the positions of each corresponding pixel in the reference grid corresponding to the four dots where the person is shown in part (a) of Figure 7 in the second direction. According to the mapping transformation relationship in the second direction, the positions of the pixels corresponding to the fingertips of the left hand of the person shown in part (b) of Figure 7 are adjusted to be the same as the positions of the pixels corresponding to the fingertips of the left hand of the person shown in part (a) of Figure 7 in the second direction, that is, aligned. The positions of the pixels corresponding to the fingertips of the right hand of the person shown in part (b) of Figure 7 in the second direction are aligned with the positions of the pixels corresponding to the fingertips of the right hand of the person shown in part (a) of Figure 7 in the second direction.

[0141] According to the mapping transformation relationship in the second direction, the position of the corresponding pixels on the top of the person's head shown in part (b) of Figure 7 in the second direction is adjusted to be the same as the position of the corresponding pixels on the top of the person's head shown in part (a) of Figure 7 in the second direction, that is, aligned.

[0142] For example, the mapping transformation relationship in the first direction can be determined based on the positions of the four pairs of matching first feature points and second feature points. The mapping transformation relationship can be a homography transformation matrix. The homography transformation matrix can be a 3*3 matrix and can be expressed as follows:

[0143]

[0144] H2 represents the homography transformation matrix in the second direction; h0 to h8 represent elements in H2.

[0145] When the first direction is a vertical direction and the second direction is a horizontal direction, the homography transformation matrix H2 may also satisfy the following conditions:

[0146]

[0147] When H2 meets the condition, according to H2, the position of the second pixel located in the target grid in the third image can be adjusted in the second direction to be the same as the position of the first pixel located in the reference grid in the first image.

[0148] When the first direction is the horizontal direction, the position in the first direction is represented by x, and the position in the second direction is represented by y. The homography transformation matrix H2 may also satisfy the following conditions:

[0149]

[0150] In another embodiment, referring to FIG11 , which is a schematic diagram of an image processing device, the device includes:

[0151] An image acquisition module 1 is configured to acquire a first image and a second image; wherein both the first image and the second image include the same target object;

[0152] A first processing module 2 is configured to adjust, in a first direction, a position of a second pixel of the target object in the second image to be the same as a position of a first pixel of the target object in the first image, to obtain a third image; wherein the first pixel and the second pixel are pixels at the same position in the target object;

[0153] a target grid determination module 3, configured to determine a target grid in the third image based on a reference grid in the first image; wherein the target grid includes: a target position; a target pixel value of a pixel corresponding to the target position, and a reference pixel value of pixels corresponding to N reference positions in the reference grid that satisfies a preset condition;

[0154] The second processing module 4 is configured to adjust, in a second direction, the position of the second pixel located in the target grid in the third image to be the same as the position of the first pixel located in the reference grid in the first image based on the positional relationship between the reference grid and the target grid; wherein the second direction is perpendicular to the first direction.

[0155] In another embodiment, the target grid determination module 3 includes:

[0156] An establishing unit, configured to create M*N reference grids of a preset size in the first image;

[0157] a target pixel value determining unit, configured to determine, in the third image, K target pixel values ​​that satisfy the preset condition with the K reference pixel values, based on the reference pixel values ​​of the pixels corresponding to the K reference positions in the reference grid;

[0158] a target pixel determining unit, configured to determine, in the third image, K target pixels corresponding to the target pixel values ​​according to the K target pixel values;

[0159] The target grid determination unit is used to determine the target grid according to K target pixels; M, N and K are all positive integers.

[0160] In another embodiment, the apparatus further comprises:

[0161] A first average pixel value determination module is configured to determine a first average pixel value of each pixel in a preset area; wherein the preset area includes an area centered on the pixel corresponding to the reference position and having a preset number of pixels as a radius;

[0162] The reference pixel value determining module is configured to determine the first average pixel value as the reference pixel value.

[0163] In another embodiment, the target pixel value determining unit includes:

[0164] a candidate pixel value determination subunit, configured to determine K candidate pixel values ​​in the third image;

[0165] The second average pixel value determination subunit is used to determine the second average pixel value of each pixel in the candidate area; wherein the candidate area includes: an area with the pixel corresponding to the candidate pixel value as the center and the preset number of pixels as the radius;

[0166] The target pixel value determination subunit is configured to determine the candidate pixel values ​​respectively corresponding to the K second average pixel values ​​that satisfy the preset condition of the K reference pixel values ​​as the target pixel values.

[0167] In another embodiment, the reference grid includes at least a square grid or a triangular grid; and the reference position includes at least a corner point of the grid or a midpoint of a sideline of the grid.

[0168] In another embodiment, the preset condition at least includes: the sum of the absolute values ​​of the differences between the reference pixel value and the target pixel value is less than a target threshold.

[0169] In another embodiment, the second processing module 4 includes:

[0170] A first position correspondence determination unit is configured to determine the position correspondence according to the reference position and the target position; the position correspondence includes a mapping transformation relationship in the second direction;

[0171] The first processing unit is configured to adjust, in the second direction, the position of the second pixel located in the target grid in the third image to be the same as the position of the first pixel located in the reference grid in the first image based on the mapping transformation relationship in the second direction.

[0172] In another embodiment, the mapping transformation relationship in the second direction includes: a homography transformation matrix in the second direction.

[0173] In another embodiment, the first processing module 2 includes:

[0174] A first feature point extraction unit, configured to extract a first feature point of the target object in the first image;

[0175] A second feature point extraction unit is configured to extract a second feature point of the target object in the second image; wherein the first feature point and the second feature point are a pair of matching feature points at the same position in the target object;

[0176] a second position correspondence determination unit, configured to determine a mapping transformation relationship in a first direction according to the first feature point and the second feature point;

[0177] The second processing unit is used to adjust the position of the second pixel in the target object in the second image in the first direction to be the same as the position of the first pixel in the target object in the first image according to the mapping transformation relationship in the first direction, so as to obtain the third image.

[0178] In another embodiment, the mapping transformation relationship in the first direction includes: a homography transformation matrix in the first direction.

[0179] In another embodiment, an electronic device is provided, including:

[0180] A processor and a memory for storing executable instructions capable of running on the processor, wherein:

[0181] When the processor is used to run the executable instructions, the executable instructions execute the method described in any one of the above embodiments.

[0182] In another embodiment, a non-transitory computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, the method described in any one of the above embodiments is implemented.

[0183] It should be noted that the “first” and “second” in the embodiments of the present disclosure are only for the convenience of expression and distinction and have no other specific meanings.

[0184] In another embodiment, another image processing method is provided. Referring to FIG12 , it is a schematic diagram of the method.

[0185] Generally speaking, global image alignment methods based on homography transformation have difficulty aligning objects that are not in the same plane; methods based on dense optical flow fields have a large amount of computation and have difficulty processing large images; methods based on grid alignment basically use feature points to constrain the displacement of the grid, and the alignment effect is poor in scenes where image feature points cannot be extracted.

[0186] This disclosed example can be applied to the field of image alignment. It can be used to align images of the same scene captured by a single camera from different perspectives, or to align images of the same scene captured by multiple cameras. The aligned images can be used as input for multi-frame HDR and super-resolution algorithms, as well as for multi-camera image fusion algorithms.

[0187] For example, a first image captured by a mobile phone camera at 1x magnification and a second image captured at 3x magnification are used. As shown in Figure 4, when the 1x and 3x lenses of a mobile phone capture the same scene, the first and second images have significant differences in field of view due to different shooting angles and lens configurations, such as field of view and focal length. To perform image fusion using the images from the two lenses, an algorithm must be used to align the first and second images before proceeding with the subsequent algorithm processing.

[0188] Referring to FIG12 , the method includes:

[0189] Step 1: Extract feature points of the target object in the first image and the second image. The same feature of the target object forms a feature point pair in the first image and the second image.

[0190] Step 2: By determining the pairs of feature points, the positions of the pixels of the target object in the second image and the positions of the corresponding pixels of the target object in the first image are adjusted to the same homography transformation matrix in the first direction. For example, the first direction is the vertical direction, and the homography matrix of the image is globally row-aligned, so as to achieve row alignment between the target object in the first image and the target object in the second image, and obtain a third image.

[0191] Step 3: Determine the corresponding target grid in the third image based on the reference grid in the first image. This includes: dividing the grid in the third image (i.e., the row-aligned image), and obtaining the target grid corresponding to the grid point in the image by row search and matching of the grid points.

[0192] Step 4: For each reference grid and the corresponding target grid, calculate the position transformation relationship of the target object in the first image and the third image in the second direction using the correspondence between the preset position and the target position in the grid, and adjust the position of the target object in the third image according to the position transformation relationship.

[0193] Step 1: Extract matching feature points of the target object in the first image and the target object in the second image.

[0194] Because the first and second images may differ in rotation, translation, and field of view, performing a global homography to align the first and second images requires obtaining feature matching point pairs between the target object in the first and second images. Common feature point matching methods such as SIFT and ORB can be used. See Figure 4.

[0195] Step 2: Calculate the global row alignment homography matrix and align the image rows.

[0196] The global homography transformation matrix H is a 3*3 matrix with the following form:

[0197]

[0198] h0 to h8 represent elements of H.

[0199] The homography matrix contains a total of 8 degrees of freedom, and the set of matching feature point pairs in step 1 can provide two sets of constraints. Therefore, at least four pairs of matching feature point pairs are required to obtain the row-aligned homography matrix. At the same time, in order to achieve row alignment of the image, the obtained H matrix must also satisfy the requirement that the coordinates of the matching feature point pairs in the row direction (i.e., the y coordinates) of the transformed image are equal. The H matrix satisfies:

[0200]

[0201] That is, for the position (x, y) of the corresponding pixel of the feature point A shown in part (b) of Figure 4 and the position (x1, y1) of the corresponding pixel of the feature point A' shown in part (a) of Figure 4, after the H transformation, the position of the corresponding pixel of point A is equal to y'.

[0202] Step 3: On the third image after row alignment, use grid point row search to determine the target grid.

[0203] After achieving row alignment of the images using the global homography matrix H, a uniform reference grid is created on the first image. The target grid is then searched on the row-aligned third image using the sum of the absolute differences in pixel values, thereby obtaining the correspondence between each grid point in the left and right images. See Figure 7 for a schematic diagram of grid search matching.

[0204] Step 4: Determine the position transformation relationship between the target object in the first image and the target object in the third image using the reference position of the reference grid and the target position of the target grid.

[0205] After step three, the correspondence between the reference position in each reference grid and the target position in the target grid is obtained. Since each grid has four grid points, a homography can be used to calculate the positional correspondence between the pixels in each reference grid and the corresponding target grid. The form of the homography is similar to the H matrix in step two. By mapping the pixels in each target grid using the homography, the third image is aligned with the pixel positions of the target object in the first image.

[0206] 1. Using global homography for row alignment, we limit image misalignment to the column direction and restrict the search range of grid points to one dimension. This reduces the amount of computation required in the alignment process compared to the optical flow-based grid alignment method.

[0207] 2. Grid division is performed on the image, and grid matching points are directly searched. This increases the reference information available for grid alignment. This makes the constraints between grid points based on the grid alignment method independent of the extraction of feature points, thus improving the alignment effect.

[0208] Figure 13 is a block diagram of a terminal device according to an exemplary embodiment. For example, the terminal device can be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0209] 13 , the terminal device may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output (I / O) interface 812 , a sensor component 814 , and a communication component 816 .

[0210] The processing component 802 generally controls the overall operation of the terminal device, such as operations associated with presentations, phone calls, data communications, camera operation, and recording. The processing component 802 may include one or more processors 820 to execute instructions to perform all or part of the steps of the aforementioned method. Furthermore, the processing component 802 may include one or more modules to facilitate interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate interaction between the multimedia component 808 and the processing component 802.

[0211] The memory 804 is configured to store various types of data to support operations on the terminal device. Examples of such data include instructions for any application or method operating on the terminal device, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0212] The power component 806 provides power to various components of the terminal device. The power component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the terminal device.

[0213] The multimedia component 808 includes a screen that provides an output interface between the terminal device and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor can not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the terminal device is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera can receive external multimedia data. Each front camera and rear camera can be a fixed optical lens system or have a focal length and optical zoom capability.

[0214] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), which is configured to receive external audio signals when the terminal device is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signal can be further stored in the memory 804 or transmitted via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0215] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as a keyboard, click wheel, buttons, etc. These buttons may include but are not limited to: a home button, volume buttons, a start button, and a lock button.

[0216] The sensor assembly 814 includes one or more sensors for providing status assessments of various aspects of the terminal device. For example, the sensor assembly 814 can detect the open / closed state of the terminal device, the relative positioning of components, such as the display and keypad of the terminal device. The sensor assembly 814 can also detect changes in the position of the terminal device or a component of the terminal device, the presence or absence of user contact with the terminal device, the orientation or acceleration / deceleration of the terminal device, and temperature changes of the terminal device. The sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 814 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0217] The communication component 816 is configured to facilitate wired or wireless communication between the terminal device and other devices. The terminal device can access a wireless network based on a communication standard, such as WiFi, 4G or 5G, or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0218] In an exemplary embodiment, the terminal device may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above method.

[0219] Other embodiments of the present invention will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of the invention that follow from the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the invention being indicated by the following claims.

[0220] It should be understood that the present invention is not limited to the exact construction described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. An image processing method, wherein: The method comprises: Acquire a first image and a second image; wherein the first image and the second image both include the same target object; In a first direction, adjusting the position of a second pixel of the target object in the second image to be the same as the position of a first pixel of the target object in the first image, to obtain a third image; wherein the first pixel and the second pixel are pixels at the same position in the target object; Determining a target grid in the third image based on a reference grid in the first image; wherein the target grid includes: a target position; a target pixel value of a pixel corresponding to the target position, and a reference pixel value of a pixel corresponding to a reference position in the reference grid meeting a preset condition; According to the position correspondence between the reference grid and the target grid, in a second direction, the position of the second pixel in the third image is adjusted to be the same as the position of the first pixel in the first image; wherein the second direction is perpendicular to the first direction.

2. The method according to claim 1, wherein The determining of the target grid in the third image according to the reference grid in the first image includes: Creating M*N reference grids of a preset size in the first image; Determining, in the third image, K target pixel values ​​that satisfy the preset condition with the K reference pixel values ​​based on the reference pixel values ​​of the K reference positions in the reference grid; According to the K target pixel values, determining K target pixels corresponding to the target pixel values ​​in the third image; The target grid is determined according to K target pixels; M, N and K are all positive integers.

3. The method according to claim 2, wherein: The method further comprises: Determine a first average pixel value of each pixel in a preset area; wherein the preset area includes: an area with a pixel corresponding to the reference position as the center and a preset number of pixels as the radius; The first average pixel value is determined as the reference pixel value.

4. The method according to claim 3, wherein: The determining, in the third image, K target pixel values ​​that meet the preset condition with the K reference pixel values ​​according to the reference pixel values ​​of the K reference positions in the reference grid, includes: Determining K candidate pixel values ​​in the third image; Determine a second average pixel value of each pixel in a candidate area; wherein the candidate area includes: an area with a pixel corresponding to the candidate pixel value as the center and the preset number of pixels as the radius; The candidate pixel values ​​respectively corresponding to the K second average pixel values ​​whose K reference pixel values ​​meet the preset condition are determined as the target pixel values.

5. The method according to any one of claims 1 to 4, wherein: The reference grid includes at least: a square grid or a triangular grid; The reference position includes at least a corner point of a grid or a middle point of a grid edge line.

6. The method according to any one of claims 1 to 4, wherein: The preset condition at least includes: the sum of the absolute values ​​of the differences between the reference pixel value and the target pixel value is less than a target threshold.

7. The method according to claim 1, wherein Adjusting, in a second direction, the position of the second pixel in the third image to be the same as the position of the first pixel in the first image according to the positional correspondence between the reference grid and the target grid, includes: Determining the position correspondence relationship according to the reference position and the target position; the position correspondence relationship includes: a mapping transformation relationship in the second direction; According to the mapping transformation relationship in the second direction, in the second direction, the position of the second pixel located in the target grid in the third image is adjusted to be the same as the position of the first pixel located in the reference grid in the first image.

8. The method according to claim 7, wherein: The mapping transformation relationship in the second direction includes: a homography transformation matrix in the second direction.

9. The method according to claim 1, wherein The step of adjusting, in the first direction, a position of a second pixel of the target object in the second image to be the same as a position of a first pixel of the target object in the first image to obtain a third image includes: extracting a first feature point of the target object in the first image; Extracting a second feature point of the target object in the second image; wherein the first feature point and the second feature point are a matching feature point pair at the same position in the target object; Determining a mapping transformation relationship in a first direction according to the first feature point and the second feature point; According to the mapping transformation relationship in the first direction, the position of the second pixel in the target object in the second image is adjusted in the first direction to be the same as the position of the first pixel in the target object in the first image, so as to obtain the third image.

10. The method according to claim 9, wherein: The mapping transformation relationship in the first direction includes: a homography transformation matrix in the first direction.

11. An image processing device, wherein: The device comprises: An image acquisition module, configured to acquire a first image and a second image; wherein the first image and the second image both include the same target object; a first processing module, configured to adjust, in a first direction, a position of a second pixel of the target object in the second image to be the same as a position of a first pixel of the target object in the first image, to obtain a third image; wherein the first pixel and the second pixel are pixels at the same position in the target object; a target grid determination module, configured to determine a target grid in the third image based on a reference grid in the first image; wherein the target grid includes: a target position; a target pixel value of a pixel corresponding to the target position, and a reference pixel value of a pixel corresponding to a reference position in the reference grid meeting a preset condition; A second processing module is configured to adjust, in a second direction, a position of a second pixel in the third image to be the same as a position of the first pixel in the first image based on a positional correspondence between the reference grid and the target grid; wherein the second direction is perpendicular to the first direction.

12. An electronic device comprising a processor, a transceiver, a memory, and an executable program stored in the memory and capable of being run by the processor, wherein: When the processor runs the executable program, the method according to any one of claims 1 to 10 is performed.

13. A computer storage medium storing an executable program; after being executed by a processor, the executable program can implement the method provided in any one of claims 1 to 10.

Citation Information

Patent Citations

  • Image processing method and apparatus, electronic device, and computer-readable storage medium

    CN109087238A

  • Image alignment method and device, equipment and storage medium

    CN110827336A

  • Image processing method and device, electronic equipment and medium

    CN112102374A

  • Aligning digital images

    CN113781369A