Image processing method, device, equipment and storage medium

Abstract

The present application discloses an image processing method, device, equipment and storage medium, belonging to the field of communication technology. The method mainly includes obtaining rotational motion information of a first image, a second image and an electronic device, wherein the first image and the second image are adjacent frame images; performing an affine transformation on the first image according to the rotational motion information to obtain a third image, wherein the similarity between the third image and the second image is higher than a first preset similarity; determining the displacement vector information of the third image relative to the second image according to the first target image projection information of the second image and the second target image projection information of the third image; generating motion estimation confidence information according to the second image, the third image and the displacement vector information, wherein the motion estimation confidence information is used to evaluate the confidence value of the motion vector information of each pixel block in the first image in the second image.

Description

Technical Field

[0001] The present application belongs to the field of communication technology, and specifically relates to an image processing method, device, equipment and storage medium. Background Art

[0002] With the development of electronic equipment technology, the functions of electronic equipment are becoming more and more diverse. For example, electronic equipment can record videos or take pictures, and process the videos or pictures to meet the needs of users to record videos or take pictures.

[0003] Currently, the processing of videos or images includes motion estimation, which is used to determine the motion vector information describing the transformation of two-dimensional images. However, traditional motion estimation, whether based on optical flow or block matching, cannot cope with the complex motion of electronic devices, resulting in inaccurate determination of motion vector information, making image registration ineffective and affecting the imaging quality of the final image. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide an image processing method, apparatus, device and storage medium, which can solve the current problem of inaccurate determination of motion vector information.

[0005] In a first aspect, an embodiment of the present application provides an image processing method, which is applied to an electronic device. The method may include:

[0006] Acquire a first image, a second image, and rotational motion information of the electronic device, wherein the first image and the second image are adjacent frame images;

[0007] Performing an affine transformation on the first image according to the rotational motion information to obtain a third image, wherein the similarity between the third image and the second image is higher than a first preset similarity;

[0008] Determine displacement vector information of the third image relative to the second image according to the first target image projection information of the second image and the second target image projection information of the third image;

[0009] Motion estimation confidence information is generated according to the second image, the third image and the displacement vector information, where the motion estimation confidence information is used to evaluate the confidence value of the motion vector information of each pixel block in the first image in the second image.

[0010] In a second aspect, an embodiment of the present application provides an image processing device, which is applied to an electronic device, and the device may include:

[0011] An acquisition module, used to acquire a first image, a second image and rotational motion information of the electronic device, wherein the first image and the second image are adjacent frame images;

[0012] A processing module, configured to perform an affine transformation on the first image according to the rotational motion information to obtain a third image, wherein the similarity between the third image and the second image is higher than a first preset similarity;

[0013] A determination module, used to determine displacement vector information of the third image relative to the second image according to the first target image projection information of the second image and the second target image projection information of the third image;

[0014] The generation module is used to generate motion estimation confidence information according to the second image, the third image and the displacement vector information, wherein the motion estimation confidence information is used to evaluate the confidence value of the motion vector information of each pixel block in the first image in the second image.

[0015] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor, a memory, and a program or instruction stored in the memory and executable on the processor, and when the program or instruction is executed by the processor, the steps of the image processing method shown in the first aspect are implemented.

[0016] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the image processing method shown in the first aspect are implemented.

[0017] In a fifth aspect, an embodiment of the present application provides a chip, the chip including a processor and a communication interface, the communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the steps of the image processing method shown in the first aspect.

[0018] In the embodiment of the present application, after the first image is affine transformed by the rotational motion information, the third image obtained can be regarded as the front and rear frame images no longer having the axial camera rotational motion, which is more in line with the motion state that only includes horizontal and vertical plane motions. Then, the first target image projection information of the second image and the second target image projection information of the third image are used to estimate the global motion, and the displacement vector information of the third image relative to the second image is calculated, which can be used to solve the problem of weak anti-noise ability in traditional motion estimation and no effective determination of the accuracy of motion vector information.

[0019] In addition, the image processing method provided in the embodiment of the present application also generates motion estimation confidence information for evaluating the confidence value of the motion vector information of each pixel block in the first image in the second image based on the second image, the third image and the displacement vector information, which is conducive to more flexible subsequent algorithm operation and more robust use of motion estimation results, thereby improving the final image noise reduction, super-resolution, HDR effect, etc. Therefore, the image processing method provided in the embodiment of the present application can perform noise reduction processing on the image while improving the accuracy of determining the motion vector information, thereby improving the imaging quality of the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 One of the processing architecture diagrams provided in the embodiment of the present application;

[0021] Figure 2 A second schematic diagram of a processing architecture provided in an embodiment of the present application;

[0022] Figure 3 A flowchart of an image processing method provided in an embodiment of the present application;

[0023] Figure 4 A schematic diagram of obtaining a three-axis rotation motion vector provided in an embodiment of the present application;

[0024] Figure 5 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application;

[0025] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0026] Figure 7 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0028] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0029] Motion estimation is a process used to determine the motion vectors that describe the transformation of a two-dimensional image. Motion estimation is an important component of image processing technologies such as video noise reduction, multi-frame fusion, sub-pixel super-resolution, high-dynamic range (HDR), and video stabilization.

[0030] Traditional motion estimation can be divided into motion estimation based on optical flow or motion estimation based on block matching. Taking motion estimation based on block matching as an example, the main process of motion estimation based on block matching is to divide each frame of the image sequence into many non-overlapping macroblocks, and assume that the displacement of all pixels in the macroblock is the same. Then, for each macroblock to a given specific search range of the reference frame, the block most similar to the current block is found according to a certain matching criterion, that is, the matching block, and the relative displacement between the matching block and the current block is the motion vector. When compressing the video, only the motion vector and residual data need to be saved to completely restore the current block. However, whether it is motion estimation based on optical flow or motion estimation based on block matching, traditional motion estimation cannot cope with the complex motion of electronic devices, that is, the motion estimation ability of the axial rotation of the shooting device body is weak, and the simple two-dimensional horizontal or vertical estimation cannot describe the axial rotation motion. In this way, there is a lack of effective estimation of the global overall motion, resulting in inaccurate determination of motion vector information, failure of subsequent registration, affecting the clarity, noise and tailing defects of the final image processing, and reducing the imaging quality of the final image.

[0031] Based on the above-mentioned problems, the embodiment of the present application provides an image processing method based on motion estimation. By obtaining the three-axis rotational motion vector of the electronic device, the rotational motion information of the electronic device can be determined more accurately, and the accuracy of the determined motion vector information can be improved. In addition, before the global motion estimation, i.e., generating the motion estimation confidence information, the first image can be affine transformed according to the rotational motion information to obtain the third image, and the displacement vector information of the third image relative to the second image can be determined according to the first target image projection information of the second image and the second target image projection information of the third image. In this way, the influence of the axial rotational motion of the electronic device on the global motion estimation can be removed, which is conducive to the subsequent algorithm operation to be more flexible, and the motion estimation results can be used more robustly, and the final image noise reduction, super-resolution, HDR effect, etc. can be improved. Therefore, the image processing method provided by the embodiment of the present application can perform noise reduction processing on the image while improving the accuracy of the determined motion vector information, thereby improving the imaging quality of the image.

[0032] Based on this, the following Figure 1-Figure 4 , the image processing method provided in the embodiment of the present application is described in detail through specific embodiments and their application scenarios.

[0033] The present application embodiment proposes a processing architecture, such as Figure 1As shown, the processing architecture may include an electronic device. The electronic device 10 may include a camera and a gyroscope, etc., which are angular motion detection devices for detecting the angular motion of a moment-sensitive housing of a high-speed rotating body relative to the inertial space around one or two axes orthogonal to the axis of rotation. The camera, such as the front camera and the rear camera in the electronic device, is used to capture the first image and the second image. The angular motion detection device, such as a gyroscope, is used to obtain the three-axis rotational motion vector of the electronic device, so that the image processing device in the electronic device determines the rotational motion information of the electronic device according to the three-axis rotational motion vector obtained by the gyroscope.

[0034] Based on this processing architecture, the application scenario of the image processing method provided in the embodiment of the present application is explained.

[0035] like Figure 2 As shown, when a user records a video through the electronic device 10, the electronic device obtains at least two adjacent frame images, namely the first image and the second image, in the video recorded by the shooting device, and obtains the three-axis rotational motion vector of the electronic device measured by the gyroscope. Based on the first image, the second image and the three-axis rotational motion vector, the rotational motion affine matrix is ​​calculated, and the rotational motion affine matrix is ​​determined as the rotational motion information.

[0036] Next, an affine transformation is performed on the first image according to the rotational motion information to obtain a third image. The similarity between the third image and the second image is higher than the first preset similarity, that is, the third image is closer to the second image, so as to later calculate the displacement vector information of the third image relative to the second image in order to reduce the noise of the image.

[0037] Furthermore, grayscale projection can be used to estimate the global motion vector, i.e., displacement vector information, corresponding to the image after affine transformation, and determine the reference item for subsequent motion estimation, i.e., motion estimation confidence information. Here, the grayscale projection in the embodiment of the present application may include image mapping, projection filtering, and related calculations. The processes included in the grayscale projection are described in detail below.

[0038] Image mapping: performing image mapping on the second image and the third image respectively to obtain a first projection sequence corresponding to the second image and a second projection sequence corresponding to the third image.

[0039] Projection filtering, calculating a first image projection value corresponding to the first projection sequence and a second image projection value corresponding to the second projection sequence. And filtering the first image projection value and the second image projection value respectively to obtain first target image projection information corresponding to the first image projection value and second target image projection information corresponding to the second image projection value.

[0040] Correlation calculation, when the first target image projection information includes a first row and column projection curve, and the second target image projection information includes a second row and column projection curve, cross-correlation calculation is performed on the first row and column projection curve and the second row and column projection curve to obtain a curve peak of the correlation curve between the first row and column projection curve and the second row and column projection curve; and, the curve peak of the correlation curve is determined as the displacement vector information of the third image relative to the second image.

[0041] Then, in order to facilitate more flexible operation of subsequent algorithms and more robust use of motion estimation results, and to improve the final image noise reduction, super-resolution, HDR effects, etc., in an embodiment of the present application, a motion estimation confidence system can be constructed based on the global motion vector as a reference item for subsequent image processing, that is, motion estimation confidence information can be generated based on the second image, the third image and the displacement vector information, and the motion estimation confidence information is used to evaluate the confidence value of the motion vector information of each pixel block in the first image in the second image.

[0042] It should be noted that, in addition to being applicable to the above-mentioned scenario of processing images in a recorded video (or continuously taken photos) in real time during the user recording a video, the image processing method provided in the embodiment of the present application can also be used in scenarios of performing image processing on already recorded videos (or continuously taken photos). Here, the image processing method provided in the embodiment of the present application can be applied to any scenario of performing image processing on at least two images (recording videos or taking continuous images through electronic devices).

[0043] Therefore, in the embodiment of the present application, after the first image is affine transformed by the rotational motion information, the third image obtained can be regarded as the front and rear frame images no longer having the axial camera rotational motion, which is more in line with the motion state that only includes horizontal and vertical plane motions. Then, the first target image projection information of the second image and the second target image projection information of the third image are used to estimate the global motion, and the displacement vector information of the third image relative to the second image is calculated, which can be used to solve the problem of weak anti-noise ability in traditional motion estimation and no effective determination of the accuracy of motion vector information.

[0044] In addition, the image processing method provided in the embodiment of the present application also generates motion estimation confidence information for evaluating the confidence value of the motion vector information of each pixel block in the first image in the second image based on the second image, the third image and the displacement vector information, which is conducive to more flexible subsequent algorithm operation and more robust use of motion estimation results, thereby improving the final image noise reduction, super-resolution, HDR effect, etc. Therefore, the image processing method provided in the embodiment of the present application can perform noise reduction processing on the image while improving the accuracy of determining the motion vector information, thereby improving the imaging quality of the image.

[0045] According to the above application scenarios, the following Figure 3 The image processing method provided in the embodiment of the present application is described in detail.

[0046] Figure 3 A flowchart of an image processing method provided in an embodiment of the present application.

[0047] like Figure 3 As shown, the image processing method can be applied to Figure 1 and Figure 2 The electronic device shown, based on this, can specifically include the following steps:

[0048] Step 310, obtaining the rotational motion information of the first image, the second image and the electronic device, wherein the first image and the second image are adjacent frame images. Step 320, performing an affine transformation on the first image according to the rotational motion information to obtain a third image, wherein the similarity between the third image and the second image is higher than a first preset similarity. Step 330, determining the displacement vector information of the third image relative to the second image according to the first target image projection information of the second image and the second target image projection information of the third image. Step 340, generating motion estimation confidence information according to the second image, the third image and the displacement vector information, wherein the motion estimation confidence information is used to evaluate the confidence value of the motion vector information of each pixel block in the first image in the second image.

[0049] In this way, after the first image is affine transformed with rotational motion information, the third image obtained can be regarded as the front and rear frame images no longer having axial camera rotational motion, which is more in line with the motion state that only contains horizontal and vertical plane motions. Then, the first target image projection information of the second image and the second target image projection information of the third image are used to estimate the global motion, and the displacement vector information of the third image relative to the second image is calculated, which can be used to solve the problem of weak anti-noise ability and no effective determination of the accuracy of motion vector information in traditional motion estimation.

[0050] In addition, the image processing method provided in the embodiment of the present application also generates motion estimation confidence information for evaluating the confidence value of the motion vector information of each pixel block in the first image in the second image based on the second image, the third image and the displacement vector information. This is conducive to more flexible subsequent algorithm operation and more robust use of motion estimation results, thereby improving the final image noise reduction, super-resolution, HDR effects, etc.

[0051] Therefore, the image processing method provided in the embodiment of the present application can improve the accuracy of determining motion vector information while performing noise reduction processing on the image to improve the imaging quality of the image.

[0052] The above steps are described in detail below:

[0053] First, referring to step 310, in one or more optional embodiments, before step 310, the image processing method may further include steps 3101 to 3103, as specifically described below.

[0054] Step 3101, obtaining the three-axis rotation motion vector of the electronic device.

[0055] Step 3102, calculating the rotational motion affine matrix based on the first image, the second image and the three-axis rotational motion vector.

[0056] Step 3103, determining the rotational motion affine matrix as the rotational motion information.

[0057] For example, Figure 4 As shown, in the embodiment of the present application, the three-axis rotational motion vector of the electronic device measured by the gyroscope can be obtained, that is, the X value in the horizontal direction, the Y value in the vertical direction, and the Z value in the vertical direction. Based on this, according to the three-axis rotational motion vector, the rotational motion affine matrix of the electronic device is calculated by a direct method, and the rotational motion affine matrix can be used to characterize the posture information of the electronic device, wherein the posture information may include that the position has not changed, but only the motion information such as the shooting height and direction has changed, and may also include information that the position, shooting height, direction, etc. have all changed, and of course, may also include information that the position has changed, but the shooting height, etc. have not changed.

[0058] In this way, if the posture information of the electronic device changes by T, so that the relative position of the pixel block A in the first image and the pixel block A1 in the second image changes, then T1 that satisfies the preset change can be selected from the different values ​​of the change T to minimize the error between the grayscale values ​​of the pixel blocks. At this time, this T1 is the posture transformation matrix of the shooting device in the electronic device, that is, the rotational motion affine matrix.

[0059] Furthermore, with respect to step 320, in one or more optional embodiments, an affine transformation is performed on the first image according to the rotational motion information to obtain a third image, and a similarity between the third image and the second image is higher than a first preset similarity.

[0060] The affine transformation in the embodiment of the present application refers to a linear transformation of a vector space followed by a translation in geometry to transform it into another vector space. Here, the affine transformation mainly transforms the first image after the affine transformation into a third image, making the third image closer to the second image. In this way, the first image after the affine transformation can be regarded as the front and rear frames no longer having axial camera rotation motion, which is more in line with the motion state of only horizontal and vertical planar motions in the grayscale projection involved in step 330, which is conducive to more flexible operation of the subsequent algorithm.

[0061] Next, step 330 is involved, in which the global motion vector corresponding to the image after affine transformation is estimated by grayscale projection, and the reference item for subsequent motion estimation, i.e., motion estimation confidence information, is determined. Here, the grayscale projection in the embodiment of the present application may include image mapping, projection filtering, and related calculations. The processes included in the grayscale projection are described in detail below. Before step 330, the image processing method may also include steps 3301-3303, as shown below.

[0062] Image mapping, namely step 3301, performs image mapping on the second image and the third image respectively to obtain a first projection sequence corresponding to the second image and a second projection sequence corresponding to the third image.

[0063] Exemplarily, image mapping is to map the two-dimensional grayscale information of each frame input image, namely the first image and the third image, into two independent one-dimensional projection sequences. The projection sequences, namely the first projection sequence and the second projection sequence, can be calculated by the following formula (1).

[0064] G k (i)=∑ j G k (i, j), G k (j) ) =∑ i G k (i, j), (1)

[0065] Among them, G k (i) is the gray value of the i-th row of the k-th frame image, G k (j) is the gray value of the jth column of the kth frame image, G k (i, j) is the grayscale value of the pixel at position (i, j) in the k-th frame image.

[0066] Projection filtering, namely step 3302, calculates a first image projection value corresponding to the first projection sequence and a second image projection value corresponding to the second projection sequence.

[0067] Further, the sequence values ​​corresponding to each column in the first projection sequence are added to obtain a first image projection value corresponding to the first projection sequence; and the sequence values ​​corresponding to each column in the second projection sequence are added to obtain a second image projection value corresponding to the second projection sequence.

[0068] Step 3303: Filter the first image projection value and the second image projection value respectively to obtain first target image projection information corresponding to the first image projection value and second target image projection information corresponding to the second image projection value.

[0069] For example, projection filtering is to filter the image projection value, because when the offset is large, the edge information will have an adverse effect on the peak value of the cross-correlation during the cross-correlation operation, which requires removing the projection value at the edge. Cosine filtering is usually used, which can retain the projection value of the middle area and reduce the information amplitude of the edge area, thereby ensuring the correctness of the following correlation calculation and improving the correction accuracy.

[0070] Based on this, in one or more optional embodiments of the relevant calculation process, when the first target image projection information includes a first row and column projection curve, and the second target image projection information includes a second row and column projection curve, step 330 may specifically include:

[0071] Perform cross-correlation calculation on the first row and column projection curves and the second row and column projection curves to obtain a curve peak value of a correlation curve between the first row and column projection curves and the second row and column projection curves;

[0072] A curve peak of the correlation curve is determined as displacement vector information of the third image relative to the second image.

[0073] Exemplarily, the correlation calculation is to perform cross-correlation calculation on the row and column projection curves of the kth frame image such as the third image and the reference frame image such as the second image, and the row and column displacement vectors of the current frame image relative to the reference frame image can be determined based on the peak values ​​of the two correlation curves.

[0074] Then, step 340 is involved. In one or more optional embodiments, step 340 may specifically include steps 3401 to 3403, as shown below:

[0075] Step 3401 , segmenting the third image based on the image sequence to obtain a plurality of mutually non-overlapping first pixel blocks, wherein the plurality of first pixel blocks include a first target pixel block.

[0076] Step 3402: Determine a matching pixel block in the second image according to the first target pixel block and the displacement vector information.

[0077] Further, taking the first target pixel block as a base point and combining the displacement vector information, determining a target area on the second image, the target area includes a plurality of second target pixel blocks, and the displacement vector information between each second target pixel block and the first target pixel block in the plurality of second target pixel blocks satisfies a preset offset threshold;

[0078] A pixel block that meets a preset matching condition among the plurality of second target pixel blocks is determined as a matching pixel block.

[0079] Step 3403: Generate motion estimation confidence information according to the first target pixel block and the matching pixel block.

[0080] Exemplarily, the third image is divided into a plurality of non-overlapping pixel blocks in a raster scanning order, and the sizes of these pixel blocks are often fixed. Then, taking the first target pixel block in the third image as the base point, in combination with the relative offset vector, i.e., the range provided by the displacement vector information of the first target pixel block, such as the range corresponding to the preset offset threshold, multiple second target pixel blocks are searched, and in combination with the preset matching condition, the optimal matching pixel block is determined among the multiple second target pixel blocks. In this way, when the selected first target pixel block is small, the complexity of the calculation can be simplified.

[0081] In addition, video sequences usually have strong temporal correlation, so the motion vector field has strong spatial correlation. In this way, since in the embodiment of the present application, after the first image is affine transformed, the front and rear frames can be regarded as no longer having axial camera rotation motion, which is more consistent with the motion state involving only horizontal and vertical planar motion in the grayscale projection involved in step 330, at this time, the global motion vector calculated in step 303, i.e., the displacement vector information, is used as the initial value of the local motion estimation, which can improve the accuracy of determining the motion vector information.

[0082] Based on this, after step 3403, the image processing method may further include:

[0083] Detecting a range of target motion estimation confidence values ​​included in the motion estimation confidence information;

[0084] When the range of the target motion estimation confidence value is within the first confidence value range, determining that the confidence value of the motion vector information of each pixel block in the first image in the second image is not credible, and performing noise reduction on the first image without matching the pixel blocks and the displacement vector information;

[0085] When the range of the target motion estimation confidence value is within the second confidence value range, it is determined that the confidence value of the motion vector information of each pixel block in the first image in the second image is credible, and the first image is denoised by matching the pixel blocks and the displacement vector information.

[0086] Exemplarily, a motion estimation confidence value to assist subsequent modules can be generated based on the global motion estimation result. The range of the motion estimation confidence value in the embodiment of the present application can be set to [0,1]. If the target motion estimation confidence value ranges from the first confidence value, i.e., between 0 and 0.45, it means that the displacement vector information is completely unreliable. In this way, there is no need to match the pixel block and the displacement vector information to perform noise reduction on the first image. On the contrary, if the target motion estimation confidence value ranges from the second confidence value, i.e., between 0.5 and 1, it means that the displacement vector information is credible. This value can effectively assist subsequent modules to flexibly use the motion estimation results and improve the overall performance of the system. In this way, the first image can be noise reduced by matching the pixel block and the displacement vector information.

[0087] Therefore, in the embodiment of the present application, after the first image is affine transformed by the rotational motion information, the third image obtained can be regarded as the front and rear frame images no longer having the axial camera rotational motion, which is more in line with the motion state that only includes horizontal and vertical plane motions. Then, the first target image projection information of the second image and the second target image projection information of the third image are used to estimate the global motion, and the displacement vector information of the third image relative to the second image is calculated, which can be used to solve the problem of weak anti-noise ability in traditional motion estimation and no effective determination of the accuracy of motion vector information.

[0088] In addition, the image processing method provided in the embodiment of the present application also generates motion estimation confidence information for evaluating the confidence value of the motion vector information of each pixel block in the first image in the second image based on the second image, the third image and the displacement vector information, which is conducive to more flexible subsequent algorithm operation and more robust use of motion estimation results, thereby improving the final image noise reduction, super-resolution, HDR effect, etc. Therefore, the image processing method provided in the embodiment of the present application can perform noise reduction processing on the image while improving the accuracy of determining the motion vector information, thereby improving the imaging quality of the image.

[0089] It should be noted that the image processing method provided in the embodiment of the present application can be executed by an image processing device or a control module in the image processing device for executing the image processing method. In the embodiment of the present application, an image processing device executing the image processing method is taken as an example to illustrate the image processing device provided in the embodiment of the present application.

[0090] Based on the same inventive concept, the present application also provides an image processing device. Figure 5 Provide detailed explanation.

[0091] Figure 5 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application.

[0092] like Figure 5 As shown, the image processing device 50 is applied to Figure 1-Figure 2 The electronic device shown may specifically include:

[0093] An acquisition module 501 is used to acquire a first image, a second image and rotational motion information of an electronic device, wherein the first image and the second image are adjacent frame images;

[0094] A processing module 502 is used to perform an affine transformation on the first image according to the rotational motion information to obtain a third image, wherein the similarity between the third image and the second image is higher than a first preset similarity;

[0095] A determination module 503, configured to determine displacement vector information of the third image relative to the second image according to the first target image projection information of the second image and the second target image projection information of the third image;

[0096] The generating module 504 is used to generate motion estimation confidence information according to the second image, the third image and the displacement vector information, wherein the motion estimation confidence information is used to evaluate the confidence value of the motion vector information of each pixel block in the first image in the second image.

[0097] The image processing device 120 is described in detail below, as follows:

[0098] In one or more possible embodiments, the image processing device 50 may further include a first calculation module; wherein,

[0099] The acquisition module 501 can also be used to acquire the three-axis rotational motion vector of the electronic device. The first calculation module is used to calculate the rotational motion affine matrix based on the first image, the second image and the three-axis rotational motion vector. Based on this, the determination module 503 can also be used to determine the rotational motion affine matrix as the rotational motion information.

[0100] In another or more possible embodiments, the image processing device 50 may further include a mapping module, a second calculation module and a filtering module; wherein,

[0101] A mapping module, used to perform image mapping on the second image and the third image respectively, to obtain a first projection sequence corresponding to the second image and a second projection sequence corresponding to the third image;

[0102] A second calculation module, used for calculating a first image projection value corresponding to the first projection sequence and a second image projection value corresponding to the second projection sequence;

[0103] The filtering module is used to filter the first image projection value and the second image projection value respectively to obtain first target image projection information corresponding to the first image projection value and second target image projection information corresponding to the second image projection value.

[0104] In one or more possible embodiments, the second calculation module can be specifically used to add the sequence values ​​corresponding to each column in the first projection sequence to obtain a first image projection value corresponding to the first projection sequence; and add the sequence values ​​corresponding to each column in the second projection sequence to obtain a second image projection value corresponding to the second projection sequence.

[0105] In one or more possible embodiments, the determination module 503 may be specifically configured to, when the first target image projection information includes a first row-column projection curve and the second target image projection information includes a second row-column projection curve, perform cross-correlation calculation on the first row-column projection curve and the second row-column projection curve to obtain a curve peak value of a correlation curve between the first row-column projection curve and the second row-column projection curve;

[0106] A curve peak of the correlation curve is determined as displacement vector information of the third image relative to the second image.

[0107] In one or more possible embodiments, the generating module 504 may be specifically configured to segment the third image based on the image sequence to obtain a plurality of mutually non-overlapping first pixel blocks, wherein the plurality of first pixel blocks include the first target pixel block;

[0108] Determine a matching pixel block in the second image according to the first target pixel block and the displacement vector information;

[0109] Motion estimation confidence information is generated according to the first target pixel block and the matching pixel block.

[0110] Further, the generating module 504 can be specifically used to determine a target area on the second image based on the first target pixel block as a base point and in combination with the displacement vector information, wherein the target area includes a plurality of second target pixel blocks, and the displacement vector information between each second target pixel block and the first target pixel block in the plurality of second target pixel blocks satisfies a preset offset threshold;

[0111] A pixel block that meets a preset matching condition among the plurality of second target pixel blocks is determined as a matching pixel block.

[0112] In one or more further possible embodiments, the image processing device 50 may further include a detection module for detecting the range of target motion estimation confidence values ​​included in the motion estimation confidence information; based on this, the determination module 503 may further be used to, when the range of the target motion estimation confidence values ​​is a first confidence value range, determine that the confidence value of the motion vector information of each pixel block in the first image in the second image is not credible, and do not perform noise reduction on the first image by matching the pixel blocks and the displacement vector information; and, when the range of the target motion estimation confidence values ​​is a second confidence value range, determine that the confidence value of the motion vector information of each pixel block in the first image in the second image is credible, and perform noise reduction on the first image by matching the pixel blocks and the displacement vector information.

[0113] The image processing device in the embodiment of the present application can be a device, or a component, integrated circuit, or chip in an electronic device. The device can be a mobile electronic device or a non-mobile electronic device. Exemplarily, the mobile electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, an in-vehicle electronic device, a wearable device, an ultra-mobile personal computer (UMPC), a netbook, or a personal digital assistant (PDA), etc. The non-mobile electronic device can be a server, a network attached storage (NAS), a personal computer (PC), a television (TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0114] The image processing device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0115] The image processing device provided in the embodiment of the present application can achieve Figures 1 to 4 To avoid repetition, the various processes implemented by the method embodiment are not described here.

[0116] In the embodiment of the present application, after the first image is affine transformed by the rotational motion information, the third image obtained can be regarded as the front and rear frame images no longer having the axial camera rotational motion, which is more in line with the motion state that only includes horizontal and vertical plane motions. Then, the first target image projection information of the second image and the second target image projection information of the third image are used to estimate the global motion, and the displacement vector information of the third image relative to the second image is calculated, which can be used to solve the problem of weak anti-noise ability in traditional motion estimation and no effective determination of the accuracy of motion vector information.

[0117] In addition, the image processing method provided in the embodiment of the present application also generates motion estimation confidence information for evaluating the confidence value of the motion vector information of each pixel block in the first image in the second image based on the second image, the third image and the displacement vector information. This is conducive to more flexible subsequent algorithm operation and more robust use of motion estimation results, thereby improving the final image noise reduction, super-resolution, HDR effects, etc.

[0118] Therefore, the image processing method provided in the embodiment of the present application can improve the accuracy of determining motion vector information while performing noise reduction processing on the image to improve the imaging quality of the image.

[0119] Optional, such as Figure 6 As shown, an embodiment of the present application also provides an electronic device 60, including a processor 601, a memory 602, and a program or instruction stored in the memory 602 and executable on the processor 601. When the program or instruction is executed by the processor 601, each process of the above-mentioned image processing method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0120] It should be noted that the electronic devices in the embodiments of the present application include the above-mentioned mobile electronic devices and non-mobile electronic devices.

[0121] Figure 7 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application.

[0122] The electronic device 700 includes but is not limited to: a radio frequency unit 701, a network module 702, an audio output unit 703, an input unit 704, a sensor 705, a display unit 706, a user input unit 707, an interface unit 708, a memory 709, a processor 710 and a receiver 711.

[0123] Those skilled in the art will appreciate that the electronic device 700 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 710 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Figure 7 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0124] Among them, in the embodiment of the present application, the processor 710 is used to obtain the rotational motion information of the first image, the second image and the electronic device, the first image and the second image are adjacent frame images; perform an affine transformation on the first image according to the rotational motion information to obtain a third image, and the similarity between the third image and the second image is higher than the first preset similarity; determine the displacement vector information of the third image relative to the second image according to the first target image projection information of the second image and the second target image projection information of the third image; and generate motion estimation confidence information according to the second image, the third image and the displacement vector information, and the motion estimation confidence information is used to evaluate the confidence value of the motion vector information of each pixel block in the first image in the second image.

[0125] The electronic device 700 is described in detail below, as follows:

[0126] In one or more possible embodiments, the processor 710 can also be used to obtain a three-axis rotational motion vector of the electronic device; calculate a rotational motion affine matrix based on the first image, the second image and the three-axis rotational motion vector; and determine the rotational motion affine matrix as rotational motion information.

[0127] In another or more possible embodiments, the processor 710 can also be used to perform image mapping on the second image and the third image, respectively, to obtain a first projection sequence corresponding to the second image and a second projection sequence corresponding to the third image; calculate a first image projection value corresponding to the first projection sequence and a second image projection value corresponding to the second projection sequence; and filter the first image projection value and the second image projection value, respectively, to obtain first target image projection information corresponding to the first image projection value and second target image projection information corresponding to the second image projection value.

[0128] In one or more possible embodiments, the processor 710 can be specifically used to add the sequence values ​​corresponding to each column in the first projection sequence to obtain a first image projection value corresponding to the first projection sequence; and add the sequence values ​​corresponding to each column in the second projection sequence to obtain a second image projection value corresponding to the second projection sequence.

[0129] In one or more further possible embodiments, the processor 710 may be specifically configured to, when the first target image projection information includes a first row and column projection curve and the second target image projection information includes a second row and column projection curve, perform cross-correlation calculation on the first row and column projection curve and the second row and column projection curve to obtain a curve peak value of a correlation curve between the first row and column projection curve and the second row and column projection curve; and determine the curve peak value of the correlation curve as the displacement vector information of the third image relative to the second image.

[0130] In one or more further possible embodiments, the processor 710 may be specifically configured to segment the third image based on the image sequence to obtain a plurality of mutually non-overlapping first pixel blocks, wherein the plurality of first pixel blocks include a first target pixel block; determine a matching pixel block in the second image based on the first target pixel block and the displacement vector information; and generate motion estimation confidence information based on the first target pixel block and the matching pixel block.

[0131] Further, the processor 710 can be specifically used to determine a target area on the second image based on the first target pixel block as a base point and in combination with the displacement vector information, the target area including multiple second target pixel blocks, and the displacement vector information between each second target pixel block and the first target pixel block in the multiple second target pixel blocks satisfies a preset offset threshold; and determine the pixel blocks in the multiple second target pixel blocks that meet the preset matching conditions as matching pixel blocks.

[0132] In one or more further possible embodiments, the processor 710 may also be used to detect a range of target motion estimation confidence values ​​included in the motion estimation confidence information; when the range of the target motion estimation confidence values ​​is a first confidence value range, determine that the confidence values ​​of the motion vector information of each pixel block in the first image in the second image are not credible, and do not perform noise reduction on the first image by matching the pixel blocks and the displacement vector information; and, when the range of the target motion estimation confidence values ​​is a second confidence value range, determine that the confidence values ​​of the motion vector information of each pixel block in the first image in the second image are credible, and perform noise reduction on the first image by matching the pixel blocks and the displacement vector information.

[0133] It should be understood that the input unit 704 may include a graphics processor (GPU) 7041 and a microphone 7042, and the graphics processor 7041 processes the image data of a static image or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 706 may include a display panel 7061, and the display panel 7061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 707 includes a touch panel 7071 and other input devices 7072. The touch panel 7071 is also called a touch screen. The touch panel 7071 may include two parts: a touch detection device and a touch controller. Other input devices 7072 may include, but are not limited to, a physical keyboard, a function key (such as a volume control button, a switch button, etc.), a trackball, a mouse, and a joystick, which will not be repeated here. The memory 709 may be used to store software programs and various data, including but not limited to applications and operating systems. The processor 710 may integrate an application processor and a modem processor, wherein the application processor mainly processes an operating system, a user interface, and applications, etc., and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 710.

[0134] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned image processing method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0135] The processor is the processor in the electronic device in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0136] In addition, an embodiment of the present application further provides a chip, which includes a processor and a communication interface, the communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned image processing method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0137] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0138] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0139] In addition, it should be noted that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in a reverse order according to the functions involved. For example, the described methods may be performed in an order different from that described, and various steps may be added, omitted, or combined. In addition, features described with reference to certain examples may be combined in other examples.

[0140] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods of each embodiment of the present application.

[0141] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. An image processing method, characterized in that: include: Acquire a first image, a second image, and rotational motion information of an electronic device, wherein the first image and the second image are adjacent frame images; Performing an affine transformation on the first image according to the rotational motion information to obtain a third image, wherein a similarity between the third image and the second image is higher than a first preset similarity; Determining displacement vector information of the third image relative to the second image according to the first target image projection information of the second image and the second target image projection information of the third image; Motion estimation confidence information is generated according to the second image, the third image and the displacement vector information, wherein the motion estimation confidence information is used to evaluate the confidence value of the motion vector information of each pixel block in the first image in the second image.

2. The method according to claim 1, characterized in that Before acquiring the first image, the second image, and the rotational motion information of the electronic device, the method further includes: Acquire a three-axis rotation motion vector of the electronic device; Calculate a rotational motion affine matrix based on the first image, the second image and the three-axis rotational motion vector; The rotational motion affine matrix is ​​determined as the rotational motion information.

3. The method according to claim 1 or 2, characterized in that: Before determining the displacement vector information of the third image relative to the second image according to the first target image projection information of the second image and the second target image projection information of the third image, the method further includes: performing image mapping on the second image and the third image respectively to obtain a first projection sequence corresponding to the second image and a second projection sequence corresponding to the third image; calculating a first image projection value corresponding to the first projection sequence and a second image projection value corresponding to the second projection sequence; The first image projection value and the second image projection value are filtered respectively to obtain first target image projection information corresponding to the first image projection value and second target image projection information corresponding to the second image projection value.

4. The method according to claim 3, characterized in that The calculating a first image projection value corresponding to the first projection sequence and a second image projection value corresponding to the second projection sequence comprises: The sequence values ​​corresponding to each column in the first projection sequence are added to obtain a first image projection value corresponding to the first projection sequence; and the sequence values ​​corresponding to each column in the second projection sequence are added to obtain a second image projection value corresponding to the second projection sequence.

5. The method according to claim 1, characterized in that The first target image projection information includes a first row and column projection curve, and the second target image projection information includes a second row and column projection curve; and determining the displacement vector information of the third image relative to the second image according to the first target image projection information of the second image and the second target image projection information of the third image includes: Performing cross-correlation calculation on the first row and column projection curve and the second row and column projection curve to obtain a curve peak value of a correlation curve between the first row and column projection curve and the second row and column projection curve; A curve peak of the correlation curve is determined as displacement vector information of the third image relative to the second image.

6. The method according to claim 1, characterized in that The generating motion estimation confidence information according to the second image, the third image and the displacement vector information comprises: Segmenting the third image based on the image sequence to obtain a plurality of mutually non-overlapping first pixel blocks, wherein the plurality of first pixel blocks include a first target pixel block; Determining a matching pixel block in the second image according to the first target pixel block and the displacement vector information; Motion estimation confidence information is generated according to the first target pixel block and the matching pixel block.

7. The method according to claim 6, characterized in that The determining, according to the first target pixel block and the displacement vector information, a matching pixel block in the second image includes: Determine a target area on the second image based on the first target pixel block as a base point and in combination with the displacement vector information, wherein the target area includes a plurality of second target pixel blocks, and the displacement vector information between each second target pixel block and the first target pixel block in the plurality of second target pixel blocks satisfies a preset offset threshold; A pixel block that meets a preset matching condition among the plurality of second target pixel blocks is determined as the matching pixel block.

8. The method according to claim 6 or 7, characterized in that: After generating motion estimation confidence information according to the first target pixel block and the matching pixel block, the method further includes: Detecting a range of target motion estimation confidence values ​​included in the motion estimation confidence information; When the range of the target motion estimation confidence value is within the first confidence value range, determining to evaluate that the motion vector information of each pixel block in the first image in the second image is not credible, and not using the matching pixel block and the displacement vector information to perform noise reduction on the first image; When the range of the target motion estimation confidence value is within the second confidence value range, it is determined that the motion vector information of each pixel block in the first image in the second image is credible, and the first image is denoised by matching the pixel blocks and the displacement vector information.

9. An image processing device, characterized in that: include: An acquisition module, used to acquire a first image, a second image and rotational motion information of the electronic device, wherein the first image and the second image are adjacent frame images; a processing module, configured to perform an affine transformation on the first image according to the rotational motion information to obtain a third image, wherein a similarity between the third image and the second image is higher than a first preset similarity; a determination module, configured to determine displacement vector information of the third image relative to the second image according to the first target image projection information of the second image and the second target image projection information of the third image; A generation module is used to generate motion estimation confidence information based on the second image, the third image and the displacement vector information, wherein the motion estimation confidence information is used to evaluate the confidence value of the motion vector information of each pixel block in the first image in the second image.

10. An electronic device, characterized in that: include: A processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the image processing method according to any one of claims 1 to 8.

11. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by a processor, the steps of the image processing method according to any one of claims 1 to 8 are implemented.

Images