Image processing method, electronic equipment and storage medium

Abstract

The invention provides an image processing method, electronic equipment and a storage medium, relates to the technical field of image processing, and is used for improving a video anti-shake effect. The method comprises the following steps: acquiring a (K + 1) th frame of image, wherein K is a positive integer; based on a plurality of first image blocks corresponding to two adjacent frames of images from the (K-T) th frame of image to the (K + 1) th frame of image after first image segmentation, determining a first offset between the two adjacent frames of images from the (K-T) th frame of image to the (K + 1) th frame of image, T being a positive integer smaller than K; based on a plurality of second image blocks corresponding to two adjacent frames of images from the K-Tth frame of image to the (K + 1) th frame of image after second image segmentation, determining a second offset between the two adjacent frames of images from the K-Tth frame of image to the (K + 1) th frame of image; determining a third offset based on the second offset and the first offset between two adjacent frames of images from the (K-T) th frame of image to the (K + 1) th frame of image; and compensating the (K + 1) th frame of image based on the third offset to obtain the compensated (K + 1) th frame of image.

Description

Technical Field

[0001] The present disclosure relates to the technical field of image processing, and in particular to an image processing method, an electronic device, and a storage medium. Background Art

[0002] When users shoot while in motion, or when the hands shake during shooting, the captured images or videos may be shaken and blurred. In order to solve the problem of image shake, optical image stabilization (OIS) technology and electronic image stabilization (EIS) technology have emerged. OSI technology uses a variety of sensors, such as gyroscopes, accelerometers and other devices to collect displacement information to compensate for displacement. It is expensive and limited by the structure and design specifications, so its practicality is limited. EIS technology uses a complex optical flow algorithm to obtain jitter variables by calculating good features (significant corresponding corners, points and other features between image frames), and then implements anti-shake based on the obtained jitter variables. It has the problems of high computational complexity and poor practicality. Summary of the invention

[0003] The embodiments of the present disclosure provide an image processing method, an electronic device, and a storage medium, which are at least used to achieve a simple and efficient video stabilization effect.

[0004] In order to at least achieve the above-mentioned purpose, the present disclosure adopts the following technical solutions: In a first aspect, an image processing method is provided, the method comprising: acquiring the K+1th frame image, K being a positive integer; determining a first offset between two adjacent frames of images from the KTth frame image to the K+1th frame image based on a plurality of first image blocks respectively corresponding to each other after two adjacent frames of images from the KTth frame image to the K+1th frame image are segmented through the first image, T being a positive integer less than K; determining a second offset between two adjacent frames of images from the KTth frame image to the K+1th frame image based on a plurality of second image blocks respectively corresponding to each other after two adjacent frames of images from the KTth frame image to the K+1th frame image are segmented through the second image; determining a third offset based on the second offset and the first offset between two adjacent frames of images from the KTth frame image to the K+1th frame image; and compensating the K+1th frame image based on the third offset to obtain the compensated K+1th frame image.

[0005] In a second aspect, an image processing device is provided, the device comprising: an acquisition unit, used to acquire a K+1th frame image, K is a positive integer; a processing unit, used to: determine a first offset between two adjacent frames of images from the KTth frame image to the K+1th frame image based on a plurality of first image blocks respectively corresponding to each other after two adjacent frames of images from the KTth frame image to the K+1th frame image are segmented through a first image, T is a positive integer less than K; determine a second offset between two adjacent frames of images from the KTth frame image to the K+1th frame image based on a plurality of second image blocks respectively corresponding to each other after two adjacent frames of images from the KTth frame image to the K+1th frame image are segmented through a second image; determine a third offset based on the second offset and the first offset between two adjacent frames of images from the KTth frame image to the K+1th frame image; and compensate the K+1th frame image based on the third offset to obtain a compensated K+1th frame image.

[0006] According to a third aspect, an electronic device is provided, comprising: one or more processors and one or more memories for storing one or more processor executable instructions; wherein, when the one or more processors are configured to execute instructions, the electronic device implements the method provided in the first aspect above.

[0007] According to a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and when the computer instructions are executed on a computer, the computer executes the method provided in the first aspect.

[0008] According to a fifth aspect, a computer program product comprising computer instructions is provided. When the computer instructions are executed on a computer, the computer executes the method provided in the first aspect.

[0009] In the embodiment of the present disclosure, after acquiring the K+1th frame image, the K+1th frame image is compensated for a third offset obtained based on the first offset and the second offset between two adjacent frame images from the KTth frame image to the K+1th frame image to obtain a compensated K+1th frame image. On the one hand, compensating the K+1th frame image based on the third offset can eliminate the jitter in the video shooting process, obtain a relatively stable K+1th frame image, that is, obtain the compensated K+1th frame image, and achieve the video stabilization effect. On the other hand, the third offset is obtained based on the processing of the KTth frame image to the K+1th frame image, which can be understood as being obtained based on the processing of the T-frame delayed frame image before the K+1th frame image, that is, an image processing method provided by an embodiment of the present disclosure, which can determine the third offset for compensating the K+1th frame image during the video shooting process, and then compensate the K+1th frame image based on the third offset to achieve the video stabilization effect. The method has the advantages of simple algorithm, no reliance on preprocessing and other sensors, strong robustness to hardware differences, efficient video stabilization effect, and improved practicality of video stabilization technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The accompanying drawings are used to provide further understanding of the technical solution of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the technical solution of the present invention and do not constitute a limitation on the technical solution of the present invention.

[0011] Figure 1 A flowchart of an image processing method provided by an embodiment of the present disclosure;

[0012] Figure 2 A flowchart of another image processing method provided by an embodiment of the present disclosure;

[0013] Figure 3 A median filtering comparison schematic diagram provided in an embodiment of the present disclosure;

[0014] Figure 4 A flowchart of another image processing method provided by an embodiment of the present disclosure;

[0015] Figure 5 A flowchart of another image processing method provided by an embodiment of the present disclosure;

[0016] Figure 6 A flowchart of another image processing method provided by an embodiment of the present disclosure;

[0017] Figure 7 A flowchart of another image processing method provided by an embodiment of the present disclosure;

[0018] Figure 8A schematic diagram of image processing and segmentation provided in an embodiment of the present disclosure;

[0019] Fig. 9 Another image processing segmentation schematic diagram provided by an embodiment of the present disclosure;

[0020] Fig.10 A schematic diagram of the composition of an image processing device provided by an embodiment of the present disclosure;

[0021] Fig.11 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0022] The following will be combined with the drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present disclosure.

[0023] Unless the context requires otherwise, throughout the specification and claims, the term "comprise" and other forms thereof, such as the third person singular form "comprise" and the present participle form "compriKing", are to be interpreted as open, inclusive, that is, "including, but not limited to". In the description of the specification, the terms "one embodiment", "some embodiments", "exemplary embodiment", "example", "specific example" or "some examples" and the like are intended to indicate that specific features, structures, materials or characteristics associated with the embodiment or example are included in at least one embodiment or example of the present disclosure. The schematic representation of the above terms does not necessarily refer to the same embodiment or example. In addition, the specific features, structures, materials or characteristics described may be included in any one or more embodiments or examples in any appropriate manner.

[0024] The terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the present disclosure, unless otherwise specified, "plurality" means two or more.

[0025] In the embodiments of the present disclosure, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present disclosure should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0026] Additionally, the use of “based on” is meant to be open and inclusive, as a process, step, calculation, or other action “based on” one or more stated conditions or values ​​may, in practice, be based on additional conditions or values ​​beyond those stated.

[0027] During video shooting, when the camera zooms (optical zoom or digital zoom) to a larger magnification, slight shaking will further aggravate the effect of video shaking. Optical image stabilization and mechanical image stabilization in hardware are not effective for jitters exceeding a certain threshold due to limitations in design specifications. In the software EIS technology image stabilization, the cached preview video is used for image stabilization, which has the disadvantages of high computational complexity and poor real-time performance. In summary, the practicality of video image stabilization technology in related technologies is poor. Therefore, how to achieve efficient video image stabilization effect is a technical problem that needs to be solved urgently.

[0028] Based on this, the embodiments of the present disclosure provide an image processing method, an electronic device, and a storage medium. After acquiring the K+1th frame image, each frame image from the KTth frame image to the K+1th frame image is segmented twice to obtain a plurality of first image blocks and a plurality of second image blocks corresponding to each frame image, and the size of the first image block is larger than the size of the second image block, that is, a plurality of large image blocks and a plurality of small image blocks corresponding to each frame image are obtained. It should be understood that the image detail information contained in image blocks of different sizes is different, and there is a delay between image acquisition and image display, which can be understood as the current display The image is a frame image before the currently acquired image, and the T frame image before the K+1th frame image can be understood as a delayed frame image. Therefore, based on the multiple large image blocks and the multiple small image blocks corresponding to each frame image from the KTth frame image to the K+1th frame image, the first offset and the second offset between two adjacent frame images from the KTth frame image to the K+1th frame image are determined, and then the K+1th frame image is compensated for the third offset obtained based on the first offset and the second offset between two adjacent frame images from the KTth frame image to the K+1th frame image, so as to obtain the compensated K+1th frame image. On the one hand, the third offset obtained based on the large and small block images containing different image detail information is used to compensate for the K+1th frame image, which can eliminate the jitter in the video shooting process and obtain a relatively stable K+1th frame image, that is, the compensated K+1th frame image, to achieve the video anti-shake effect. On the other hand, the third offset is obtained based on the processing of the KTth frame image to the K+1th frame image, which can be understood as being obtained based on the processing of the T-frame delayed frame image before the K+1th frame image, that is, an image processing method provided by an embodiment of the present disclosure, which can determine the third offset for compensating the K+1th frame image during the video shooting process, and then compensate the K+1th frame image based on the third offset to achieve the video anti-shake effect. The method has a simple algorithm, does not rely on preprocessing and other sensors, has strong robustness to hardware differences, achieves efficient video anti-shake effect, and improves the practicality of video anti-shake technology.

[0029] The image processing method provided by the embodiments of the present disclosure can be applied to electronic devices with a camera function, for example, the electronic device can be a mobile phone, a tablet computer, a desktop, a laptop, a handheld computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, as well as a cellular phone, a personal digital assistant (PDA), an augmented reality (AR)\virtual reality (VR) device, a camera, etc. The embodiments of the present disclosure do not limit the specific form of the electronic device.

[0030] Next, if Figure 1 As shown, the present disclosure provides an image processing method, which may include the following steps:

[0031] S101, obtaining the K+1th frame image.

[0032] The K+1th frame image may be the K+1th frame image in a video stream obtained by the electronic device shooting the shooting area, and K is a positive integer. The video stream may be a real-time video stream.

[0033] In some embodiments, after the video stabilization function is turned on, the electronic device executes an image processing method provided by an embodiment of the present disclosure. Correspondingly, after the video stabilization function is turned off, the electronic device does not execute or stops executing an image processing method provided by an embodiment of the present disclosure.

[0034] As a possible example, the video stabilization function is turned on by default in electronic devices.

[0035] As another possible example, the electronic device determines whether to turn on / off the video stabilization function according to a user instruction.

[0036] In some embodiments, after the electronic device turns on the video stabilization function, the electronic device acquires K+1 frames of images.

[0037] In some embodiments, the electronic device obtains the K+1th frame image by receiving a shooting instruction, and then responding to the shooting instruction, the electronic device shoots a shooting area to obtain a video stream, and then intercepts the video stream to obtain the K+1th frame image.

[0038] In some embodiments, the K+1th frame image is obtained by processing the K+1th frame original image based on the K frame images before the K+1th frame image. For the description of how to process the K+1th frame original image based on the K frame images before the K+1th frame image to obtain the K+1th frame image, please refer to the corresponding descriptions in the following steps S201 to S203, which will not be repeated here.

[0039] S102: Determine a first offset between two adjacent image frames from the KTth image frame to the K+1th image frame based on a plurality of first image blocks respectively corresponding to two adjacent image frames from the KTth image frame to the K+1th image frame after being segmented by the first image.

[0040] Wherein, T is a positive integer less than K.

[0041] It should be understood that there is a delay between image acquisition and image display. For example, the current image captures the K+1th frame image, but the current image display may be the KTth frame image. The T-frame images between the Kth frame image and the KTth frame image can be understood as delayed frame images. During the image acquisition process, that is, during the video shooting process, the fluctuation of video jitter should be within a certain range within a period of time. Therefore, the KTth frame image to the K+1th frame image can be processed to obtain the offset between the KTth frame image and the K+1th frame image, that is, the third offset described below, and then the K+1th frame image is compensated based on the third offset to eliminate the jitter during the video shooting process and obtain a relatively stable K+1th frame image, that is, the compensated K+1th frame image.

[0042] However, the image detail information contained in image blocks of different sizes is different. In order to determine the third offset, the first offset between two adjacent frames of the KT-th frame image to the K+1-th frame image can be determined based on a plurality of first image blocks that correspond to each other after the two adjacent frames of the KT-th frame image to the K+1-th frame image are segmented through the first image, and the second offset between two adjacent frames of the KT-th frame image to the K+1-th frame image can be determined based on a plurality of second image blocks that correspond to each other after the two adjacent frames of the KT-th frame image to the K+1-th frame image are segmented through the second image, and then the third offset can be determined based on the second offset and the first offset between two adjacent frames of the KT-th frame image to the K+1-th frame image. The image segmentation parameters used for the first image segmentation are different from the image segmentation parameters used for the second image segmentation, and the image segmentation parameters include quantity or size.

[0043] Regarding how to perform a first image segmentation on two adjacent frames of images from the KTth frame image to the K+1th frame image to obtain multiple first image blocks corresponding to each frame of images from the KTth frame image to the K+1th frame image, and perform a second image segmentation on two adjacent frames of images from the KTth frame image to the K+1th frame image to obtain multiple second image blocks corresponding to each frame of images from the KTth frame image to the K+1th frame image, reference can be made to the following description of step S301 and step S304, which will not be repeated here.

[0044] In some embodiments, Figure 2 As shown, the first offset between any two adjacent frames of the KT-th frame image to the K+1-th frame image, for example, the first offset between the i-th frame image and the i+1-th frame image, can be determined based on the following steps:

[0045] X1. Determine a plurality of image block pairs based on a plurality of first image blocks corresponding to the i-th frame image and a plurality of first image blocks corresponding to the (i+1)-th frame image.

[0046] Each image block pair includes a third image block and a fourth image block, the third image block is a first image block in the i-th frame image, the fourth image block is a first image block matching the third image block in the i+1-th frame image, and i is an integer less than or equal to K.

[0047] As an example, step X1 may include the following steps:

[0048] X11. Use each first image block in the i-th frame image as the third image block.

[0049] X12. For each third image block, determine a fourth image block that matches the third image block from multiple first image blocks in the target area of ​​the (i+1)th frame image, and take the third image block and the fourth image block that matches the third image block as an image block pair.

[0050] The target area of ​​the i+1th frame image includes a first area corresponding to the position of the third image block in the i+1th frame image, and a second area adjacent to the first area. That is, for each third image block, based on the position of the third image block, a fourth image block matching the third image block is determined from a larger range of the position corresponding to the position of the third image block in the i+1th frame image.

[0051] In some embodiments, determining a fourth image block matching the third image block from a plurality of first image blocks in the target area of ​​the (i+1)th frame image may include the following steps:

[0052] A1. Obtain color feature information of each of a plurality of first image blocks in a target area of ​​an (i+1)th frame image, and color feature information of a third image block.

[0053] The color feature information of an image block may be the color component of the image block, and the color component of an image block may be the color value of a pixel of the image block, and the color value of the pixel includes a red value, a green value, and a blue value.

[0054] Exemplarily, the attribute information of a third image block may be as shown in the following formulas (1) to (4):

[0055] block m1,m2 ,m1∈[1,M],m2∈[1,M] Formula (1);

[0056] center m1,m2 =(x1,y1),x1∈[0,rows-1],y1∈[0,cols-1] Formula (2);

[0057] blkWidth=int(cols / M) formula (3);

[0058] blkHeight=int(rows / M) formula (4);

[0059] In the above formulas (1) to (4), block represents the third image block, (m1, m2) represents the position of the third image block in the i-th frame image, M represents the number of rows and columns of the i-th frame image into multiple third image blocks, that is, the i-th frame image is divided into M*M third image blocks, and block m1,m2 represents the coordinates of the third image block in the M*M third image blocks, center m1,m2 Indicates the position of the center point of the third image block, rows indicates the row (height) of the i-th frame image, cols indicates the column (width) of the i-th frame image, blkWidth indicates the width of the third image block, and blkHeight indicates the height of the third image block.

[0060] In some embodiments, after the position of the third image block in the i-th frame image is determined based on the above formula (1), the target area of ​​the (i+1)-th frame image can be determined based on the position of the third image block in the i-th frame image.

[0061] Exemplarily, the target area corresponding to the position of the third image block in the (i+1)th frame image can be expressed as the following formula (5):

[0062] (blkHeight+Δheight,blkWidth+Δwidth) formula (5);

[0063] Among them, Δheight and Δwidth are both preset constants, which can be understood as two preset offsets, (blkHeight+Δheight,blkWidth+Δwidth) is the target area, (blkHeight,blkWidth) can be understood as the above-mentioned first area, and (Δheight,Δwidth) can be understood as the above-mentioned second area. It can be seen from the above formula (5) that in order to reduce the influence of video jitter on the matching accuracy of the fourth image block matching the third image block in the i+1 frame image, after determining the position of the third image block, the fourth image block matching the third image block is determined based on the position of the third image block within a larger range of the position of the first image block in the i+1 frame image.

[0064] Exemplarily, the attribute information of a first image block in the target area may be as shown in the following formulas (6) to (8):

[0065] rgb′ i′,j′ =[r′ i′,j′ ,g′ i′,j′,b′ i′,j′ ] Formula (6);

[0066] block′ m1,m2 ,m1∈[1,M],m2∈[1,M] Formula (7);

[0067] center′ m1,m2 =(x1,y1),x1∈[0,rows-1],y1∈[0,cols-1] Formula (8);

[0068] Among them, rgb′ i,j represents the color component of a pixel in the first image block in the target area, that is, the color feature information of the first image block in the target area, (i′, j′) represents the position of the pixel in the target area, and i′∈(0,blkHeight+Δheight),j′∈(0,blkWidth+Δwidth), r i′,j′ represents the red value of the pixel in the first image block, g i′,j′ represents the green value of the pixel in the first image block, b i′,j′ Indicates the blue value of the pixel in the first image block, block′ m1,m2 Indicates the position of the first image block in the i+1th frame image, center′ m1,m2 Indicates the position of the center point of the first image block in the (i+1)th frame image.

[0069] Exemplarily, the color feature information of the third image block may be expressed as follows:

[0070] RGB i,j =[r i,j ,g i,j ,b i,j ] Formula (9);

[0071] Among them, rgb i,j represents the color component of the pixel in the third image block, that is, the color feature information of the third image block, (i, j) represents the position of the pixel in the third image block in the third image block, and i∈(0,blkHeight),j∈(0,blkWidth), r i,j represents the red value of the pixel in the third image block, g i,j represents the green value of the pixel in the third image block, b i,j Represents the blue value of the pixel in the third image block.

[0072] A2. Determine, from a plurality of first image blocks in the target area of ​​the (i+1)th frame image, a first image block that matches the color feature information of the third image block as a fourth image block that matches the third image block.

[0073] As an example, the first image block among multiple first image blocks in the target area that matches the color feature information of the third image block may be the first image block among multiple first image blocks in the target area that has the smallest difference in color feature information with the third image block.

[0074] It should be understood that for an object in one frame of image, the color of the object should not change or will not change significantly in the next frame of image. Therefore, for each first image block in multiple first image blocks, if the color feature information of a first image block matches the color feature information of the third image block, it means that the first image block matches the third image block. Therefore, the first image block that matches the color feature information of the third image block can be determined from the multiple first image blocks in the target area of ​​the i+1th frame of image as the fourth image block that matches the third image block.

[0075] As an example, the first image block among multiple first image blocks in the target area that matches the color feature information of the third image block may be the first image block among multiple first image blocks in the target area that has the smallest difference in color feature information with the third image block.

[0076] As an example, the first image block among multiple first image blocks in the target area that matches the color feature information of the third image block may be the first image block among multiple first image blocks in the target area that has the highest similarity with the color feature information of the third image block.

[0077] Exemplarily, taking the first image block among multiple first image blocks in the target area that matches the color feature information of the third image block as the first image block among the multiple first image blocks in the target area with the smallest difference in color feature information with the third image block as an example, in some embodiments, after obtaining the color feature information of each of the multiple first image blocks in the target area of ​​the (i+1)th frame image and the color feature information of the third image block, the color feature information of each of the multiple first image blocks can be compared with the color feature information of the third image block to obtain the color component difference corresponding to each first image block, and then the first image block with the smallest color component difference is used as the fourth image block that matches the third image block.

[0078] Exemplarily, taking the color characteristic information of an image block as the color component of the image block as an example, the color components of multiple first image blocks can be subtracted from the color components of the third image block to obtain the color component differences corresponding to the first image blocks, and then the first image block corresponding to the minimum color component difference is used as the fourth image block that matches the third image block.

[0079] In some embodiments, the fourth image block that matches the third image block in the i+1th frame image has the same size (same height and width) as the third image block. That is, the fourth image block that matches the third image block in the i+1th frame image is an image block that has the smallest color component difference with the third image block in the i+1th frame image and has the same size as the third image block.

[0080] Exemplarily, determining the first image block having the smallest color feature information difference with the third image block from among the multiple first image blocks in the target area of ​​the (i+1)th frame image as the fourth image block matching the third image block can be as shown in the following formula (10):

[0081] matchedBlock=Min||block m1,m2 -block′ m1,m2 ||=

[0082] ∑ i,j ((r i,j -r i′,j′ ) 2 +(g i,j -g i′,j′ ) 2 +(b i,j -b i′,j′ ) 2 ) 1 / 2 Formula (10);

[0083] Among them, matchedBlock is the fourth image block that matches the third image block.

[0084] After determining a fourth image block that matches a third image block, the third image block and the fourth image block that matches the third image block can be taken as an image block pair. By performing the above-mentioned processing of step X11 to step X12 on each of the multiple first image blocks corresponding to the i-th frame image, multiple image block pairs can be determined.

[0085] X2. Determine an offset of the image block pair based on a position difference between the third image block and the fourth image block in the image block pair.

[0086] The position difference between the third image block and the fourth image block may refer to the difference between the position of the center point of the third image block and the position of the center point of the fourth image block.

[0087] As an example, the difference between the position of the center point of the third image block and the position of the center point of the fourth image block in an image block pair may be used as the offset of the image block pair.

[0088] Exemplarily, based on the position difference between the third image block and the fourth image block in the image block pair, the offset of the image block pair is determined as shown in the following formula (11):

[0089] tV k =(center' i,j -center i,j )=[x i -x' i ,y j -y' j ] Formula (11);

[0090] Among them, center i,j Indicates the position of the center point of the fourth image block, center i,j represents the position of the center point of the third image block, tV k Represents the offset of an image patch pair.

[0091] By calculating the above formula (11) for each of the multiple image block pairs, the offsets corresponding to the multiple image block pairs can be obtained.

[0092] X3. Determine a first offset between the i-th frame image and the (i+1)-th frame image based on the offsets corresponding to the multiple image block pairs.

[0093] As an example, based on the offsets corresponding to each of the multiple image block pairs, the first offset between the i-th frame image and the i+1-th frame image is determined. The offsets corresponding to each of the multiple image block pairs can be filtered to obtain the first offset between the i-th frame image and the i+1-th frame image.

[0094] As an example, the filtering process includes median filtering, mean filtering, bilateral filtering and Gaussian filtering. It should be understood that the above filtering processes including median filtering, mean filtering, bilateral filtering and Gaussian filtering are only exemplary, and other filtering processes for eliminating noise by other users should also fall within the protection scope of the solution disclosed in the present invention.

[0095] Among them, median filtering is a nonlinear signal processing technology based on sorting statistics theory that can effectively suppress noise. The basic principle of median filtering is to replace the value of a point in a digital image or digital sequence with the median value of each point in a neighborhood of the point, so that the surrounding pixel values ​​are close to the true value, thereby eliminating isolated noise points.

[0096] Exemplarily, taking the filtering process as a median filtering process as an example, median filtering is performed on the offsets corresponding to multiple image blocks to obtain the first offset between the i-th frame image and the i+1-th frame image, that is, the median of the offsets corresponding to the multiple image block pairs is used as the first offset between the i-th frame image and the i+1-th frame image.

[0097] It should be noted that the above example directly performs filtering processing on the offsets corresponding to the multiple image blocks to obtain the first offset between the i-th frame image and the i+1-th frame image. As another example, in order to improve the accuracy of determining the first offset, and then improve the accuracy of compensation for the K+1-th frame image to improve the video stabilization effect, based on the offsets corresponding to the multiple image blocks, determining the first offset between the i-th frame image and the i+1-th frame image may include the following steps:

[0098] B1. Remove abnormal image block pairs from multiple image block pairs and retain non-abnormal image block pairs.

[0099] The abnormal image block pair may be an image block pair corresponding to a third image block with less local significant information, or may be an image block pair caused by motion or noise of local non-global offset.

[0100] In some embodiments, in order to determine an abnormal image block pair, for each of the plurality of image block pairs, it is possible to determine whether the image block pair is an abnormal image block pair based on image parameters of the i-th frame image and image parameters of the i+1-th frame image. The image parameters of a frame image may include at least one or more of the following: zoom ratio, ISO sensitivity, image detail information (texture), and color values ​​of pixels in each image block in the frame image.

[0101] In some embodiments, based on the image parameters of the i-th frame image and the image parameters of the i+1-th frame image, abnormal image block pairs caused by local non-integrated offset motion or noise can be screened out, that is, image block pairs that satisfy the following formula (12) are regarded as abnormal image block pairs.

[0102]

[0103] Among them, r mean is the red value of the color value of the pixel in the third image block in an image block pair, r mean ′ is the red value of the color value of the pixel in the fourth image block in an image block pair, g mean is the green value of the color value of the pixel in the third image block in an image block pair, g mean ′ is the green value of the color value of the pixel in the fourth image block in an image block pair, b mean is the blue value of the color value of the pixel in the third image block in an image block pair, b mean ′ is the blue value in the color value of the pixel in the fourth image block in an image block pair, ξ1 is the first threshold, which is a constant, and ξ1 is related to the zoom ratio, sensitivity and image detail information of the i-th frame image.

[0104] In some embodiments, local salient information of each third image block in the i-th frame image may be determined based on the image detail information in the image parameters of the i-th frame image, thereby filtering out the third image blocks with less local salient information. Exemplarily, local salient information of each third image block in the i-th frame image may be determined based on the image detail information in the image parameters of the i-th frame image, as shown in the following formula (13):

[0105]

[0106] Wherein, Texture(block) is the local salient information of a third image block in the first image block, and sub is used to represent the size of the first image block, which can be understood as a third image block including (subRows*subCols) pixels.

[0107] After determining the local saliency information of a third image block, an image block pair corresponding to the third image block whose value is less than or equal to the second threshold value may be regarded as an abnormal image block pair.

[0108] Exemplarily, the image block pair corresponding to the third image block satisfying the following formula (14) may be regarded as an abnormal image block pair.

[0109] Texture(block)≤ξ2 Formula (14);

[0110] Wherein, ξ2 is the second threshold value, which is a constant, and ξ2 is related to the zoom ratio, sensitivity and image detail information of the i-th frame image.

[0111] It should be understood that the i-th frame image and the i+1-th frame image are both captured by an electronic device, and the zoom ratio or sensitivity corresponding to the i-th frame image and the i+1-th frame image is the zoom ratio or sensitivity of the electronic device. When the electronic device zooms to a larger zoom ratio or a larger sensitivity, due to the influence of noise or blur, when the first threshold or the second threshold is too small, it will cause the first image block to match the second image block abnormally. By increasing the first threshold and the second threshold, a suitable image block pair can be found, that is, a non-abnormal image block pair can be obtained.

[0112] In some embodiments, the first threshold is obtained based on the following formula (15), and the second threshold is obtained based on the following formula (16).

[0113] ξ1=ISO factor *Zoom factor *ξ1 formula (15);

[0114] ξ2=ISO factor *Zoom factor *ξ2 formula (16);

[0115] Among them, ISO factor is the sensitivity factor, Zoom factor is the zoom factor.

[0116] In some embodiments, only the image block pair satisfying the above formula (12) can be regarded as an abnormal image block pair; or only the image block pair corresponding to the third image block satisfying the above formula (14) can be regarded as an abnormal image block pair; or both the image block pair satisfying the above formula (12) and the image block pair corresponding to the third image block satisfying the above formula (14) can be regarded as an abnormal image block pair, which is not limited in the embodiments of the present disclosure.

[0117] In some embodiments, after the abnormal image block pairs are determined, the abnormal image block pairs may be eliminated from the plurality of image block pairs to obtain and retain non-abnormal image block pairs.

[0118] B2. Determine a first offset between the i-th image frame and the (i+1)-th image frame based on the respective offsets of the non-abnormal image block pairs.

[0119] In some embodiments, based on the respective offsets of non-abnormal image blocks, the first offset between the i-th frame image and the i+1-th frame image is determined. The respective offsets of the non-abnormal image blocks can be filtered to obtain the first offset between the i-th frame image and the i+1-th frame image.

[0120] For the description of the filtering process, reference may be made to the description of the filtering process in the above step X3, which will not be repeated here.

[0121] Exemplarily, assuming that the number of multiple image block pairs is M*M, abnormal image block pairs are eliminated from the multiple image block pairs, and the number of non-abnormal image block pairs retained is I, where M is a positive integer and I is a positive integer less than or equal to M*M.

[0122] As an example, taking the filtering process as a median filtering process as an example, the median filtering process is performed on the respective offsets of the non-abnormal image blocks to obtain the first offset between the i-th frame image and the i+1-th frame image, which can be shown as the following formula (17):

[0123] V i =Median(tV0,tV1,tV2,...,tV I ) formula (17);

[0124] Among them, V i Indicates the first offset between the i-th frame image and the i+1-th frame image, tV0-tV I Used to represent the offset corresponding to the first non-abnormal image block pair among I non-abnormal image block pairs to the offset corresponding to the Ith non-abnormal image block pair.

[0125] Figure 3 A median filtering comparison diagram provided in an embodiment of the present disclosure is shown in FIG. Figure 3 The solid line in is the offset between the i-th frame image and the i+1-th frame image before median filtering. Figure 3 The dotted line in is the first offset between the i-th frame image and the i+1-th frame image after median filtering, Figure 3 It can be seen that by performing median filtering on the offsets of the non-abnormal image blocks, the first offset between the i-th frame image and the (i+1)-th frame image is more gentle.

[0126] It should be noted that, when the electronic device is equipped with a sensor such as a gyroscope, the median filtering is performed on the offsets of the non-abnormal image blocks to obtain the original first offset between the i-th frame image and the i+1-th frame image, and then the offset detected by the gyroscope and other sensors is subtracted from the original first offset to obtain the first offset between the i-th frame image and the i+1-th frame image, so as to improve the accuracy of determining the first offset between the i-th frame image and the i+1-th frame image, thereby improving the subsequent video anti-shake effect. For example, taking the electronic device as a handheld device as an example, when the handheld device moves slowly to shoot a video, the gyroscope and other sensors detect the nearly uniform offset in the horizontal / vertical direction. In the V calculated in the above steps, k The offset detected by the sensor is subtracted from the offset to obtain the first offset between the i-th frame image and the i+1-th frame image.

[0127] By performing the above processing on any two adjacent frames from the KTth frame image to the K+1th frame image, the first offset between any two adjacent frames from the KTth frame image to the K+1th frame image can be obtained, that is, T first offsets can be obtained.

[0128] S103 , determining a second offset between two adjacent image frames from the KT-th to the K+1-th image frames based on a plurality of second image blocks respectively corresponding to two adjacent image frames from the KT-th to the K+1-th image frames after being segmented by the second image.

[0129] Among them, the size of the first image block is larger than the size of the second image block, the first image block can be understood as a large image block, and the second image block can be understood as a small image block. That is to say, in the above step S102, the first offset between two adjacent frames of images from the KT frame image to the K+1 frame image is determined based on a plurality of large image blocks that correspond to each other after the two adjacent frames of images from the KT frame image to the K+1 frame image are segmented through the first image, and in step S103, the second offset between two adjacent frames of images from the KT frame image to the K+1 frame image is determined based on a plurality of small image blocks that correspond to each other after the two adjacent frames of images from the KT frame image to the K+1 frame image are segmented through the second image. For the description of step S103, reference can be made to the corresponding description of the above step S102, which will not be repeated here.

[0130] S104: Determine a third offset based on the second offset and the first offset between two adjacent frames of images from the KTth frame image to the K+1th frame image.

[0131] As an example, based on the second offset and the first offset between two adjacent frames of the KT-th frame image to the K+1-th frame image, the third offset can be determined by performing a difference operation on the second offset and the first offset between two adjacent frames of the KT-th frame image to the K+1-th frame image to obtain T offset differences, and then filtering the T offset differences to obtain the third offset. The filtering process includes median filtering, mean filtering, bilateral filtering, and Gaussian filtering.

[0132] Exemplarily, based on the second offset and the first offset between two adjacent frames of the KT-th frame image to the K+1-th frame image, the third offset is determined as shown in the following formulas (18) to (19):

[0133] delayFramsVectors=[(v k-T+1 -V k-T+1 ),(v k-T+2 -V k-T+2 ),...,(vk -V k )] Formula (18);

[0134] f k =MedianFilter(delayFramsVectors) formula (19);

[0135] Where delayFramsVector represents T offset differences, v k-T+1 represents the second offset between the KTth frame image and the K-T+1th frame image, V k-T+1 represents the first offset between the KTth frame image and the K-T+1th frame image, v k-T+2 represents the second offset between the K-T+1th frame image and the K-T+2th frame image, V k-T+2 represents the first offset between the K-T+1th frame image and the K-T+2th frame image, v k represents the second offset between the Kth frame image and the K+1th frame image, V k represents the first offset between the Kth frame image and the K+1th frame image, fv k Indicates the third offset.

[0136] S105 . Compensate the K+1th frame image based on the third offset to obtain a compensated K+1th frame image.

[0137] As an example, based on the third offset, the K+1th frame image is compensated to obtain the compensated K+1th frame image. The K+1th frame image can be affine transformed based on the third offset to obtain the compensated K+1th frame image.

[0138] For example, Figure 4 As shown, performing affine transformation processing on the K+1th frame image based on the third offset to obtain the compensated K+1th frame image may include the following steps:

[0139] C1. Based on the third offset and the K+1th frame image, obtain a second affine transformation matrix.

[0140] Among them, an affine transformation is geometrically defined as an affine transformation between two vector spaces or an affine mapping consisting of a non-singular linear transformation (a transformation performed using a linear function) followed by a translation transformation. In the finite-dimensional case, each affine transformation can be given by a matrix A and a vector b. The affine transformation can be written as A and an additional column b. An affine transformation corresponds to the multiplication of a matrix and a vector, while the composition of affine transformations corresponds to ordinary matrix multiplication, as long as an extra row is added to the bottom of the matrix. This row is all 0 except for a 1 on the rightmost side, and a 1 is added to the bottom of the column vector.

[0141] As an example, based on the third offset and the K+1th frame image, the second affine transformation matrix is ​​obtained. The third offset can be added to the K+1th frame image to obtain the K+1th frame image after adding the third offset, and then the second affine transformation matrix is ​​obtained based on the K+1th frame image and the K+1th frame image after adding the third offset.

[0142] Exemplarily, based on the third offset and the K+1th frame image, a second affine transformation matrix is ​​obtained, which can be shown as the following formula (20):

[0143] affineMatrix=getAffineMatrix(image1,image2) formula (20);

[0144] Wherein, affineMatrix is ​​the second affine transformation matrix, image1 represents the K+1th frame image, image2 represents the K+1th frame image after adding the third offset, and for the description of getAffineMatrix, please refer to the corresponding description in the related technology, which will not be repeated here.

[0145] As a possible implementation manner, the second affine transformation matrix may also be obtained based only on the third offset.

[0146] C2. Obtain a compensated K+1th frame image based on the second affine transformation matrix and the K+1th frame image.

[0147] As an example, based on the second affine transformation matrix and the K+1th frame image, the compensated K+1th frame image is obtained. The K+1th frame image can be added with a third offset to obtain the K+1th frame image after adding the third offset, and then the K+1th frame image after adding the third offset is affine transformed based on the second affine transformation matrix to obtain the compensated K+1th frame image.

[0148] Exemplarily, based on the second affine transformation matrix and the K+1th frame image, the compensated K+1th frame image is obtained, which can be shown as the following formula (21):

[0149] image2′=WarpAffine(image2,affineMatrix) formula (21);

[0150] Wherein, image2′ represents the compensated K+1th frame image, and WarpAffine represents the affine transformation.

[0151] It should be understood that there will be black edges in the K+1th frame image after adding the third offset. In order to remove the black edges in the K+1th frame image after adding the third offset and restore the image size, it is necessary to perform affine transformation processing on the K+1th frame image after adding the third offset.

[0152] based on Figure 1 In the embodiment shown, after acquiring the K+1 frame image, each frame image from the KT frame image to the K+1 frame image is segmented twice to obtain a plurality of first image blocks and a plurality of second image blocks corresponding to each frame image, and the size of the first image block is larger than the size of the second image block, that is, a plurality of large image blocks and a plurality of small image blocks corresponding to each frame image are obtained. It should be understood that the image detail information contained in image blocks of different sizes is different, and there is a delay between image acquisition and image display. It can be understood that the currently displayed image is an image frame before the currently acquired image, and the T-frame image before the K+1-th frame image can be understood as a delayed frame image. Therefore, based on the multiple large image blocks and multiple small image blocks corresponding to each frame image from the KT-th frame image to the K+1-th frame image, the first offset and the second offset between the two adjacent frames of the KT-th frame image to the K+1-th frame image are determined, and then the K+1-th frame image is compensated for the third offset obtained based on the first offset and the second offset between the two adjacent frames of the KT-th frame image to the K+1-th frame image, so as to obtain the compensated K+1-th frame image.

[0153] In this way, on the one hand, the third offset obtained based on the large and small block images containing different image detail information is used to compensate for the K+1th frame image, which can eliminate the jitter in the video shooting process and obtain a relatively stable K+1th frame image, that is, the compensated K+1th frame image, to achieve the video anti-shake effect. On the other hand, the third offset is obtained based on the processing of the KTth frame image to the K+1th frame image, which can be understood as being obtained based on the processing of the T-frame delayed frame image before the K+1th frame image, that is, an image processing method provided by an embodiment of the present disclosure, which can determine the third offset for compensating the K+1th frame image during the video shooting process, and then compensate the K+1th frame image based on the third offset to achieve the video anti-shake effect. The algorithm is simple, does not rely on preprocessing and other sensors, has strong robustness to hardware differences, achieves efficient video anti-shake effect, and improves the practicality of video anti-shake technology.

[0154] In some embodiments, the K+1th frame image is obtained by processing the K+1th frame original image based on the K frame images before the K+1th frame image. For example, Figure 5 As shown, obtaining the K+1th frame image in the above step S101 may include the following steps:

[0155] S201 : Determine a first offset between two adjacent image frames from the 1st to the Kth image frames based on a plurality of first image blocks respectively corresponding to two adjacent image frames from the 1st to the Kth image frames after being segmented by the first image.

[0156] The first frame image is the first frame image in the K+1 frame image. For the description of how to determine the first offset between two adjacent frames of the first frame image to the Kth frame image, reference can be made to the corresponding description in the above step S102, which will not be repeated here.

[0157] S202 , predicting a first offset between two adjacent frames of images from the Kth frame image to the 1st frame image to obtain a fourth offset.

[0158] The fourth offset may also be called by other names, for example, the predicted offset of the K+1th frame image.

[0159] It should be understood that during the video shooting process, the fluctuation of video jitter should be within a certain range within a period of time. Therefore, after obtaining the first offset between two adjacent frames of images from the 1st frame to the Kth frame, the first offset between two adjacent frames of images from the 1st frame to the Kth frame can be predicted to obtain the predicted offset of the K+1th frame image, that is, the fourth offset.

[0160] As an example, as an example, a preset offset prediction model based on a time series algorithm that has been trained is pre-stored in the memory of the electronic device, and a first offset between two adjacent frames of images from the 1st frame image to the Kth frame image is predicted to obtain a fourth offset, which can be obtained by inputting the first offset between two adjacent frames of images from the 1st frame image to the Kth frame image into the preset offset prediction model based on the time series algorithm. The preset offset prediction model includes a Kalman filter model or an autoregressive integrated moving average (ARIMA) model.

[0161] It should be noted that the Kalman filter model provided in the embodiment of the present disclosure is a Kalman filter model after the image parameters of the Kth frame image are updated. It should be understood that the Kth frame image is collected by an electronic device, that is, the Kalman filter model provided in the embodiment of the present disclosure is a Kalman filter model after the shooting parameters of the electronic device during the video shooting process are updated. In the embodiment of the present disclosure, the image parameters of the Kth frame image are applied to the update process of the Kalman filter model. Exemplarily, the parameters of the updated Kalman filter model are shown in the following formula:

[0162] predictor:

[0163]

[0164] p' k =Ap k-1 A T +Q

[0165] update:

[0166] K k =p' k H T (Hp' k H T +R) -1

[0167]

[0168] p k =(IK k H)p' k

[0169] The above Q is an update factor, which is related to the zoom ratio and sensitivity of the electronic device during the shooting process. When the electronic device is zoomed to a larger zoom ratio or a larger sensitivity during the shooting process, Q is updated to Q=zoom factor *ISO facto *r σ 2 Among them, zoom factor is the zoom factor, ISO factor is the sensitivity factor, and σ is a constant. For the description of other parameters of the updated Kalman filter model, for example, p' k , K k For the description of parameters such as A and A, please refer to the description of each parameter in the Kalman filter model in the relevant technology, which will not be repeated here.

[0170] S203 , obtaining the K+1th frame of original image, and compensating the K+1th frame of original image based on the fourth offset to obtain the K+1th frame of image.

[0171] As an example, the K+1th frame original image is compensated based on the fourth offset to obtain the K+1th frame image, and the K+1th frame image is obtained by performing affine transformation processing on the K+1th frame original image based on the fourth offset.

[0172] For example, Figure 6 As shown, performing affine transformation processing on the K+1th frame original image based on the fourth offset to obtain the K+1th frame image may include the following steps:

[0173] D1. Obtain a first affine transformation matrix based on the fourth offset and the K+1th frame of the original image.

[0174] As an example, based on the fourth offset and the K+1th frame original image, the first affine transformation matrix is ​​obtained, and the fourth offset can be added to the K+1th frame original image to obtain the K+1th frame original image after adding the fourth offset, and then based on the K+1th frame original image and the K+1th frame original image after adding the fourth offset, the first affine transformation matrix is ​​obtained.

[0175] Exemplarily, based on the fourth offset and the original image of the K+1th frame, a first affine transformation matrix is ​​obtained, which can be shown as the following formula (22):

[0176] affineMatrix′=getAffineMatrix(image3,image4) formula (22);

[0177] Among them, affineMatrix' is the first affine transformation matrix, image3 represents the K+1th frame original image, image2 represents the K+1th original image after adding the fourth offset, and for the description of getAffineMatrix, please refer to the corresponding description in the related technology, which will not be repeated here.

[0178] As a possible implementation method, the first affine transformation matrix can also be obtained based on the fourth offset.

[0179] D2. Obtain the K+1th frame image based on the first affine transformation matrix and the K+1th frame original image.

[0180] As an example, based on the first affine transformation matrix and the K+1th frame original image, the K+1th frame image is obtained. The K+1th frame original image may be added with a fourth offset to obtain the K+1th frame original image after adding the fourth offset, and then the K+1th frame original image after adding the fourth offset is affine transformed based on the first affine transformation matrix to obtain the K+1th frame image.

[0181] Exemplarily, based on the first affine transformation matrix and the K+1th frame original image, the K+1th frame image is obtained, which can be shown as the following formula (23):

[0182] image2=WarpAffine(image4,affineMatrix) formula (23);

[0183] Among them, image2 represents the K+1th frame image, and WarpAffine represents affine transformation.

[0184] It should be understood that there will be black edges in the K+1th frame original image after adding the fourth offset. In order to remove the black edges in the K+1th frame original image after adding the fourth offset and restore the image size, it is necessary to perform affine transformation processing on the K+1th frame original image after adding the fourth offset.

[0185] based on Figure 5 In the embodiment shown, a first offset between two adjacent frames of images from the 1st frame image to the Kth frame image is determined based on a plurality of first image blocks respectively corresponding to two adjacent frames of images from the 1st frame image to the Kth frame image after segmentation of the first image, and then a fourth offset is obtained based on a prediction process performed on the first offset between two adjacent frames of images from the Kth frame image to the 1st frame image. That is, before obtaining the K+1th frame original image, a prediction process is performed in advance based on the first offset between two adjacent frames of images from the Kth frame image to the 1st frame image to obtain a predicted offset of the K+1th frame image, that is, the fourth offset, and then after obtaining the K+1th frame original image, the K+1th frame original image is compensated based on the fourth offset to obtain the K+1th frame image.

[0186] From the above description of the first image block, it can be known that the first image block can be understood as a large image block, that is, the K frame images before the K+1 frame image are each divided into a plurality of large image blocks, and then the first offset between two adjacent frame images is determined based on the plurality of large image blocks corresponding to each of the K frame images, and then the K+1 frame original image is compensated based on the fourth offset obtained from the K first offsets, so as to obtain a relatively stable K+1 frame image. In this way, it can be understood that the first compensation of the K+1 frame original image is performed based on the K frame images before the K+1 frame being divided into large image blocks, and then the second compensation of the K+1 frame image is performed based on the T frame delayed frame images before the K+1 frame being divided into large image blocks and small image blocks, so as to eliminate the jitter in the video shooting process and obtain a relatively stable K+1 frame image, that is, the compensated K+1 frame image.

[0187] In some embodiments, Figure 7 As shown, an image processing method provided by an embodiment of the present disclosure may further include the following steps before the above step S102:

[0188] S301: Obtain salient features of the Kth frame image.

[0189] The salient features of the K-th frame image are salient features of the K-th frame image in the shooting area.

[0190] In some embodiments, after obtaining the K+1 frame image, in order to perform image segmentation on the K+1 frame image to obtain a plurality of first image blocks and a plurality of second image blocks corresponding to each frame image, it is necessary to determine the size of the first image block and the size of the second image block. In order to determine the size of the first image block and the size of the second image block, the electronic device obtains the salient features of the Kth frame image.

[0191] S302: Determine feature parameters of the first image block and feature parameters of the second image block based on the salient features of the Kth frame image.

[0192] The characteristic parameter includes size or quantity. That is, based on the significant feature of the Kth frame image, the size or quantity of the first image block is determined, and the size or quantity of the second image block is determined.

[0193] For example, the salient features of the K-th frame image can be expressed as the following formula (24):

[0194]

[0195] Among them, Texture k is the salient feature of the K-th frame image.

[0196] As an example, determining the characteristic parameters of the first image block and the characteristic parameters of the second image block based on the salient features of the K-th frame image may be determining the number of first image blocks and the number of second image blocks based on the salient features of the K-th frame image, and further determining the size of the first image block and the size of the second image block based on the number of first image blocks, the number of second image blocks and the K-th frame image.

[0197] In some embodiments, the size of the first image block and the size of the second image block are positively correlated with the salient features of the Kth frame image, and the number of the first image blocks and the number of the second image blocks are negatively correlated with the salient features of the Kth frame image.

[0198] For example, when Texture k When the Texture is smaller, the size of the first image block and the size of the second image block are reduced, that is, the number of the first image blocks and the number of the second image blocks are increased to make the image segmentation more detailed. k When the size of the first image block and the size of the second image block are larger, the size of the first image block and the size of the second image block are increased, that is, the number of the first image blocks and the number of the second image blocks are reduced, so that the image is segmented more roughly.

[0199] S303 . Perform first image segmentation on each image frame from the KTth image frame to the K+1th image frame based on the characteristic parameters of the first image block to obtain a plurality of first image blocks corresponding to each image frame.

[0200] S304 , performing second image segmentation on each image frame from the KTth image frame to the K+1th image frame based on the characteristic parameters of the second image block, to obtain a plurality of second image blocks corresponding to each image frame.

[0201] Exemplarily, taking the characteristic parameter as size as an example, first image segmentation can be performed on each frame image from the 1st frame image to the K+1th frame image based on the size of the first image block to obtain multiple first image blocks corresponding to each frame image after the first image segmentation, and second image segmentation can be performed on each frame image from the 1st frame image to the K+1th frame image based on the size of the second image block to obtain multiple second image blocks corresponding to each frame image after the second image segmentation, so that the first offset between two adjacent frames of images from the KTth frame image to the K+1th frame image can be determined based on the multiple first image blocks corresponding to each adjacent two frames of images from the KTth frame image to the K+1th frame image after the first image segmentation, and the second offset between two adjacent frames of images from the KTth frame image to the K+1th frame image can be determined based on the multiple second image blocks corresponding to each adjacent two frames of images from the KTth frame image to the K+1th frame image after the second image segmentation.

[0202] For example, Figure 8As shown in FIG. 1 , a schematic diagram of image segmentation provided by an embodiment of the present disclosure is shown in FIG. Fig. 9 FIG. 1 is another image segmentation schematic diagram provided by an embodiment of the present disclosure. Figure 8 Correspondingly, a first image segmentation is performed on a frame of image to obtain a plurality of first image blocks, Fig. 9 Correspondingly, a second image segmentation is performed on a frame of image to obtain a plurality of second image blocks. Taking the feature parameter as an example, assuming that the number of first image blocks is 4*4 and the number of second image blocks is 8*8, Figure 8 It can be seen that the first image segmentation is performed on the frame image based on the number of first image blocks to obtain 4*4 first image blocks. Fig. 9 It can be seen that the second image segmentation is performed on the frame image based on the number of second image blocks to obtain 8*8 second image blocks. Figure 8 as well as Fig. 9 It can be seen that Figure 8 A size of the first image block is greater than Fig. 9 The size of a second image block in .

[0203] In some embodiments, for each frame image from the 1st frame image to the K+1th frame image, a first image segmentation can be performed on the frame image based on the characteristic parameters of the first image block to obtain multiple first image blocks corresponding to the frame image, and a second image segmentation can be performed on the frame image based on the characteristic parameters of the second image block to obtain multiple second image blocks corresponding to the frame image.

[0204] In some embodiments, the electronic device has a display function, and after obtaining the compensated K+1th frame image, the electronic device may subsequently display the compensated K+1th frame image.

[0205] The above mainly introduces the solution provided by the present disclosure from the perspective of interaction between various nodes. It is understandable that each node, such as an electronic device, includes a hardware structure and / or software module corresponding to the execution of each function in order to realize the above functions. It should be easily appreciated by those skilled in the art that the present disclosure can be implemented in the form of hardware or a combination of hardware and computer software in combination with the algorithm steps of each example described in the embodiments disclosed herein. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present disclosure.

[0206] Fig.10 FIG. 1 is a schematic diagram showing the composition of an image processing device provided by an embodiment of the present disclosure. Fig.10As shown, the image processing device 40 may include an acquisition unit 401 and a processing unit 402. In some embodiments, the image processing device 40 may further include a display unit 403.

[0207] The image processing device 40 may be the electronic device or a chip in the electronic device. When the electronic device 40 is used to implement the functions of the electronic device in the above embodiment, each unit is specifically used to implement the following functions.

[0208] The acquisition unit 401 is used to acquire the K+1th frame image, where K is a positive integer.

[0209] The processing unit 402 is used to determine a first offset between two adjacent frames of images from the KT-th frame image to the K+1-th frame image based on a plurality of first image blocks that respectively correspond to two adjacent frames of images from the KT-th frame image to the K+1-th frame image after being segmented through the first image, where T is a positive integer less than K; determine a second offset between two adjacent frames of images from the KT-th frame image to the K+1-th frame image based on a plurality of second image blocks that respectively correspond to two adjacent frames of images from the KT-th frame image to the K+1-th frame image after being segmented through the second image; determine a third offset based on the second offset and the first offset between two adjacent frames of images from the KT-th frame image to the K+1-th frame image; and compensate the K+1-th frame image based on the third offset to obtain a compensated K+1-th frame image.

[0210] In some embodiments, the processing unit 402 is specifically used to: determine a first offset between two adjacent frames of images from the 1st frame to the Kth frame based on a plurality of first image blocks that correspond to each other after two adjacent frames of images from the 1st frame to the Kth frame are segmented through the first image; perform prediction processing on the first offset between two adjacent frames of images from the Kth frame to the 1st frame to obtain a fourth offset; obtain the K+1th frame original image, and compensate the K+1th frame original image based on the fourth offset to obtain the K+1th frame image.

[0211] In some embodiments, the processing unit 402 is specifically used to: determine a plurality of image block pairs based on a plurality of first image blocks corresponding to the i-th frame image and a plurality of first image blocks corresponding to the i+1-th frame image; wherein each image block pair includes a third image block and a fourth image block, the third image block is a first image block in the i-th frame image, the fourth image block is a first image block in the i+1-th frame image that matches the third image block, and i is an integer less than or equal to K; determine an offset of the image block pair based on a position difference between the third image block and the fourth image block in the image block pair; determine a first offset between the i-th frame image and the i+1-th frame image based on the offsets corresponding to each of the plurality of image block pairs.

[0212] In some embodiments, the processing unit 402 is specifically used to: use each first image block in the i-th frame image as a third image block; for each third image block, determine a fourth image block matching the third image block from multiple first image blocks in the target area of ​​the i+1-th frame image, and use the third image block and the fourth image block matching the third image block as an image block pair; wherein the target area of ​​the i+1-th frame image includes a first area in the i+1-th frame image corresponding to the position of the third image block, and a second area adjacent to the first area.

[0213] In some embodiments, the processing unit 402 is specifically used to: obtain color feature information of each of multiple first image blocks in the target area of ​​the i+1th frame image, and color feature information of the third image block; determine the first image block with the smallest difference in color feature information with the third image block from the multiple first image blocks in the target area of ​​the i+1th frame image as the fourth image block matching the third image block.

[0214] In some embodiments, the processing unit 402 is specifically used to: eliminate abnormal image block pairs from multiple image block pairs and retain non-abnormal image block pairs; determine a first offset between the i-th frame image and the i+1-th frame image based on the offsets of the non-abnormal image block pairs.

[0215] In some embodiments, the processing unit 402 is specifically configured to: perform filtering processing on the respective offsets of the non-abnormal image blocks to obtain a first offset between the i-th frame image and the (i+1)-th frame image.

[0216] In some embodiments, the processing unit 402 is specifically configured to: perform affine transformation processing on the K+1th frame image based on the fourth offset to obtain the K+1th frame image.

[0217] In some embodiments, the acquisition unit 401 is further configured to acquire the salient features of the Kth frame image.

[0218] The processing unit 402 is further used to determine the characteristic parameters of the first image block and the characteristic parameters of the second image block based on the significant features of the Kth frame image, wherein the characteristic parameters include size or quantity; perform first image segmentation on each frame image from the KTth frame image to the K+1th frame image based on the characteristic parameters of the first image block to obtain a plurality of first image blocks corresponding to each frame image; perform second image segmentation on each frame image from the KTth frame image to the K+1th frame image based on the characteristic parameters of the second image block to obtain a plurality of second image blocks corresponding to each frame image.

[0219] In some embodiments, the processing unit 402 is specifically used to: perform difference calculation on the second offset and the first offset between two adjacent frame images from the KT frame image to the K+1 frame image to obtain T offset differences; and perform filtering processing on the T offset differences to obtain a third offset.

[0220] In some embodiments, the processing unit 402 is specifically configured to: perform affine transformation processing on the K+1th frame image based on the third offset to obtain the compensated K+1th frame image.

[0221] In some embodiments, the display unit 403 is used to display the compensated K+1th frame image.

[0222] It should be noted that Fig.10 The units in can also be called modules, for example, the acquisition unit can be called an acquisition module. Fig.10 In the illustrated embodiment, the names of the various units may not be the names shown in the figure. For example, the acquisition unit may also be called a communication unit.

[0223] Fig.10 If each unit in the embodiment is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present disclosure is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (processor) to perform all or part of the steps of the methods of each embodiment of the present disclosure. The storage medium for storing computer software products includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program codes.

[0224] When the image processing device 40 implements the functions of the integrated modules in the form of hardware, the present disclosure provides a schematic diagram of the structure of an electronic device. Fig.11 As shown, the electronic device 50 includes: a processor 502 , a communication interface 503 , and a bus 504 . Optionally, the electronic device 50 may further include a memory 501 .

[0225] The processor 502 may be a processor that implements or executes various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of the present disclosure. The processor 502 may be a central processing unit, a general-purpose processor, a digital signal processor, an application-specific integrated circuit, a field programmable gate array, or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of the present disclosure. The processor 502 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like.

[0226] The communication interface 503 is used to connect with other devices via a communication network, such as Ethernet, wireless access network, wireless local area network (WLAN), etc.

[0227] The memory 501 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program codes in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto.

[0228] As a possible implementation, the memory 501 may exist independently of the processor 502, and the memory 501 may be connected to the processor 502 via a bus 504 to store instructions or program codes. When the processor 502 calls and executes the instructions or program codes stored in the memory 501, the image processing method provided in the embodiment of the present disclosure can be implemented.

[0229] In another possible implementation, the memory 501 may also be integrated with the processor 502 .

[0230] The bus 504 may be an extended industry standard architecture (EISA) bus, etc. The bus 504 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.11 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0231] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the electronic device can be divided into different functional modules to complete all or part of the functions described above.

[0232] The disclosed embodiments also provide a computer-readable storage medium. All or part of the processes in the above method embodiments can be completed by computer instructions to instruct the relevant hardware, and the program can be stored in the above computer-readable storage medium. When the program is executed, it can include the processes of the above method embodiments. The above computer-readable storage medium can also be an external storage device of the above electronic device, such as a plug-in hard disk, a smart memory card (smart media card, SMC), a secure digital (secure digital, SD) card, a flash card (flash card), etc. equipped on the above electronic device. Further, the above computer-readable storage medium can also include both the internal storage unit of the above electronic device and an external storage device. The above computer-readable storage medium is used to store the above computer program and other programs and data required by the above electronic device. The above computer-readable storage medium can also be used to temporarily store data that has been output or is to be output.

[0233] The embodiments of the present disclosure also provide a computer program product, which includes a computer program. When the computer program product is run on a computer, the computer is enabled to execute any one of the image processing methods provided in the above embodiments.

[0234] Although the present disclosure is described herein in conjunction with various embodiments, in the process of implementing the claimed disclosure, those skilled in the art may understand and implement other variations of the disclosed embodiments by viewing the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other components or steps, and "one" or "an" does not exclude multiple situations. A single processor or other unit may implement several functions listed in the claims. Certain measures are recorded in different dependent claims, but this does not mean that these measures cannot be combined to produce good results.

[0235] Although the present disclosure has been described in conjunction with specific features and embodiments thereof, it is apparent that various modifications and combinations may be made thereto without departing from the spirit and scope of the present disclosure. Accordingly, this specification and the drawings are merely exemplary illustrations of the present disclosure as defined by the appended claims, and are deemed to have covered any and all modifications, variations, combinations or equivalents within the scope of the present disclosure. Obviously, those skilled in the art may make various modifications and variations to the present disclosure without departing from the spirit and scope of the present disclosure. Thus, if these modifications and variations of the present disclosure fall within the scope of the claims of the present disclosure and their equivalents, the present disclosure is also intended to include these modifications and variations.

[0236] The above is only a specific implementation of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present disclosure should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. An image processing method, characterized in that: The method comprises: Get the K+1th frame image, where K is a positive integer; Based on a plurality of first image blocks respectively corresponding to two adjacent frames of images from the KT-th frame image to the K+1-th frame image after being segmented by the first image, a first offset between two adjacent frames of images from the KT-th frame image to the K+1-th frame image is determined, where T is a positive integer less than K; Determine a second offset between two adjacent frames of images from the KTth frame image to the K+1th frame image based on a plurality of second image blocks respectively corresponding to two adjacent frames of images from the KTth frame image to the K+1th frame image after being segmented by the second image; Determine a third offset based on the second offset and the first offset between two adjacent frames of images from the KTth frame image to the K+1th frame image; Based on the third offset, the K+1th frame image is compensated to obtain a compensated K+1th frame image.

2. The method according to claim 1, characterized in that The step of acquiring the K+1th frame of image comprises: Determine a first offset between two adjacent frames of images from the first frame of image to the Kth frame of image based on a plurality of first image blocks respectively corresponding to two adjacent frames of image from the first frame of image to the Kth frame of image after being segmented by the first image; Predicting a first offset between two adjacent frames of the Kth frame image and the first frame image to obtain a fourth offset; The K+1th frame of original image is acquired, and the K+1th frame of original image is compensated based on the fourth offset to obtain the K+1th frame of image.

3. The method according to claim 1 or 2, characterized in that: The first offset between two adjacent frames of images is determined by the following steps: Based on a plurality of first image blocks corresponding to the i-th frame image and a plurality of first image blocks corresponding to the i+1-th frame image, a plurality of image block pairs are determined; wherein each of the image block pairs comprises a third image block and a fourth image block, the third image block is a first image block in the i-th frame image, the fourth image block is a first image block in the i+1-th frame image that matches the third image block, and i is an integer less than or equal to K; determining an offset of the image block pair based on a position difference between the third image block and the fourth image block in the image block pair; Based on the offsets corresponding to the plurality of image block pairs, a first offset between the i-th frame image and the (i+1)-th frame image is determined.

4. The method according to claim 3, characterized in that The determining of a plurality of image block pairs based on a plurality of first image blocks corresponding to the i-th frame image and a plurality of first image blocks corresponding to the (i+1)-th frame image comprises: Taking each first image block in the i-th frame image as the third image block; For each of the third image blocks, a fourth image block matching the third image block is determined from a plurality of first image blocks in a target area of ​​the i+1th frame image, and the third image block and the fourth image block matching the third image block are taken as an image block pair; wherein the target area of ​​the i+1th frame image includes a first area corresponding to a position of the third image block in the i+1th frame image, and a second area adjacent to the first area.

5. The method according to claim 4, characterized in that The step of determining a fourth image block matching the third image block from a plurality of first image blocks in the target area of ​​the (i+1)th frame image includes: Acquire color feature information of each of a plurality of first image blocks in the target area of ​​the (i+1)th frame image, and color feature information of the third image block; A first image block matching the color feature information of the third image block is determined from a plurality of first image blocks in the target area of ​​the (i+1)th frame image as a fourth image block matching the third image block.

6. The method according to claim 4, characterized in that The determining, based on the offsets corresponding to the plurality of image block pairs, a first offset between the i-th frame image and the (i+1)-th frame image comprises: Eliminate abnormal image block pairs from the multiple image block pairs and retain non-abnormal image block pairs; Based on the respective offsets of the non-abnormal image block pairs, a first offset between the i-th frame image and the (i+1)-th frame image is determined.

7. The method according to claim 6, characterized in that The determining, based on the respective offsets of the non-abnormal image block pairs, a first offset between the i-th frame image and the i+1-th frame image comprises: The respective offsets of the non-abnormal image blocks are filtered to obtain a first offset between the i-th frame image and the (i+1)-th frame image.

8. The method according to claim 2, characterized in that: The compensating the K+1th frame original image based on the fourth offset to obtain the K+1th frame image includes: Affine transformation is performed on the K+1th frame original image based on the fourth offset to obtain the K+1th frame image.

9. The method according to claim 1, characterized in that: The method further comprises: Obtain the salient features of the K-th frame image; Determining a feature parameter of the first image block and a feature parameter of the second image block based on the significant features of the Kth frame image, wherein the feature parameters include size or quantity; Performing the first image segmentation on each frame of the image from the KTth frame to the K+1th frame of the image based on the characteristic parameters of the first image block, to obtain a plurality of first image blocks corresponding to each frame of the image; The second image segmentation is performed on each image frame from the KTth image frame to the K+1th image frame based on the characteristic parameters of the second image block to obtain a plurality of second image blocks corresponding to each image frame.

10. The method according to claim 1, characterized in that The determining of the third offset based on the second offset and the first offset between two adjacent frames of images from the KTth frame image to the K+1th frame image includes: Performing a difference operation on the second offset and the first offset between two adjacent frames of images from the KTth frame image to the K+1th frame image to obtain T offset differences; The T offset differences are filtered to obtain the third offset.

11. The method according to claim 1, characterized in that: The compensating the K+1th frame image based on the third offset to obtain the compensated K+1th frame image includes: Affine transformation is performed on the K+1th frame image based on the third offset to obtain the compensated K+1th frame image.

12. An electronic device, characterized in that: include: one or more processors and one or more memories for storing instructions executable by the one or more processors; The one or more processors are configured to execute the instructions so that the electronic device performs the method as described in any one of claims 1-11.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and when the computer instructions are executed on a computer, the computer is enabled to execute the method according to any one of claims 1 to 11.

Images