Video processing method, device, electronic device and storage medium

Abstract

The present disclosure relates to a video processing method, device, electronic device and storage medium, including: obtaining at least one feature point of each frame of a video to be processed; determining the feature point motion difference between each feature point of each frame of the video to be processed and the corresponding feature point of the previous frame of each frame; determining effective feature points based on the feature point motion difference between each feature point of each frame of the video to be processed and the corresponding feature point of the previous frame of each frame; repairing the previous frame of each frame of the video to be processed according to the correspondence between the effective feature points of each frame of the video to be processed and the effective feature points of the previous frame of each frame. The effective feature points of each frame can be extracted to repair each frame, that is, the invalid feature points in at least one feature point of each frame can be eliminated, and each frame can be repaired based on the stable feature points of each frame, so that the distortion of the frame can be avoided, and the naturalness and coordination of the frame after the jitter repair can be better.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and more specifically, to a video processing method, device, electronic device, and storage medium. Background Art

[0002] With the development of technology, people can use a variety of devices to record videos. For example, digital cameras and smart terminals are important tools for video recording. Video recording can meet people's needs in many aspects, such as recording information and sharing life. However, when shooting videos, the captured video frames often shake. If the video shooting device is in motion, the shaking of the video frames will be more severe, which strongly affects the appearance and quality of the video frames.

[0003] In the related art, the motion offset between video frames is estimated to adjust the position of the target in the video frame, thereby repairing the jittery video frame and stabilizing the video frame. However, this method will cause the video frame to be distorted, resulting in the deterioration of the naturalness and coordination of the video frame after the jitter repair. Summary of the invention

[0004] The present disclosure provides a video processing method, device, electronic device and storage medium to at least solve the problem in the above-mentioned related technology that when repairing a jittery video frame, the video frame will be distorted, resulting in the naturalness and coordination of the video frame after the jitter repair being deteriorated.

[0005] According to a first aspect of an embodiment of the present disclosure, a video processing method is provided, comprising: obtaining at least one feature point of each frame of a video to be processed; determining a feature point motion difference between each feature point of each frame of the video to be processed and a corresponding feature point of a previous frame of the frame; determining a valid feature point from the at least one feature point based on the feature point motion difference between each feature point of each frame of the video to be processed and a corresponding feature point of a previous frame of the frame, wherein the feature point motion difference of a feature point is inversely correlated with a degree of effectiveness of the feature point; and repairing a previous frame of each frame of the video to be processed according to the correspondence between the valid feature points of each frame of the video to be processed and the valid feature points of the previous frame of the frame.

[0006] Optionally, the at least one feature point of each frame is a corner point of each area among the multiple areas divided into each frame.

[0007] Optionally, for the first frame of the video to be processed, at least one feature point of the first frame is a corner point of each area among the multiple areas divided by the first frame; for each subsequent frame after the first frame of the video to be processed, the feature point of each subsequent frame is a tracking feature point obtained based on the feature point of the previous frame and corresponding to the feature point of the previous frame.

[0008] Optionally, determining a valid feature point from the at least one feature point based on the feature point motion difference between each feature point of each frame of the video to be processed and the corresponding feature point of the previous frame of each frame includes: determining the number of valid tracking frames for a first feature point among the at least one feature point, wherein the valid tracking frame is a frame after the first frame of the video to be processed and including a second feature point, the feature point motion difference between the second feature point and the first feature point is less than a first preset threshold, and the first feature point is any feature point among the at least one feature point; when the number is greater than the second preset threshold, determining the first feature point as a valid feature point.

[0009] Optionally, the valid tracking frame has a corresponding relationship with the first preset threshold, and the first preset threshold corresponding to any valid tracking frame is determined based on a preset distance coefficient and an average value of feature point motion differences of any valid tracking frame, and the average value of feature point motion differences of any valid tracking frame is an average value of feature point motion differences between feature points of any valid tracking frame and corresponding feature points of a previous frame of any valid tracking frame.

[0010] Optionally, the second preset threshold is determined according to a preset amplitude coefficient and the number of frames included in the video to be processed.

[0011] Optionally, the repairing of the previous frame of each frame of the video to be processed according to the correspondence between the effective feature points of each frame of the video to be processed and the effective feature points of the previous frame of each frame includes: based on the correspondence between the effective feature points of each frame of the video to be processed and the effective feature points of the previous frame of each frame, smoothing the correspondence to obtain a smoothed correspondence; based on the smoothed correspondence, performing motion compensation on the previous frame of each frame.

[0012] Optionally, based on the correspondence between the effective feature points of each frame of the video to be processed and the effective feature points of the previous frame of each frame, the correspondence is smoothed to obtain a smoothed correspondence, including: obtaining a motion trajectory according to the correspondence between the effective feature points of each frame of the video to be processed and the effective feature points of the previous frame of each frame; determining the degree of picture jitter based on the motion trajectory; determining a smoothing parameter according to the degree of picture jitter; and using the smoothing parameter to smooth the correspondence to obtain a smoothed correspondence.

[0013] Optionally, it also includes: dividing all frames of the video to be processed into at least one frame group; repairing the previous frame of each frame of the video to be processed according to the correspondence between the valid feature points of each frame of the video to be processed and the valid feature points of the previous frame of each frame, including: repairing the previous frame of each frame contained in each frame group according to the correspondence between the valid feature points of each frame contained in each frame group in the at least one frame group and the valid feature points of the previous frame of each frame.

[0014] Optionally, the feature point motion difference is the Euclidean distance between the feature points.

[0015] According to a second aspect of an embodiment of the present disclosure, a video processing device is provided, comprising: an acquisition module, configured to acquire at least one feature point of each frame of a video to be processed; a feature point motion difference determination module, configured to determine the feature point motion difference between each feature point of each frame of the video to be processed and the corresponding feature point of the previous frame of the frame; an effective feature point determination module, configured to determine effective feature points from the at least one feature point based on the feature point motion difference between each feature point of each frame of the video to be processed and the corresponding feature point of the previous frame of the frame, wherein the feature point motion difference of the feature point is inversely correlated with the effectiveness of the feature point; and a repair module, configured to repair the previous frame of each frame of the video to be processed according to the correspondence between the effective feature points of each frame of the video to be processed and the effective feature points of the previous frame of the frame.

[0016] Optionally, the at least one feature point of each frame is a corner point of each area among the multiple areas divided into each frame.

[0017] Optionally, for the first frame of the video to be processed, at least one feature point of the first frame is a corner point of each area among the multiple areas divided by the first frame; for each subsequent frame after the first frame of the video to be processed, the feature point of each subsequent frame is a tracking feature point obtained based on the feature point of the previous frame and corresponding to the feature point of the previous frame.

[0018] Optionally, the effective feature point determination module is configured to: determine the number of effective tracking frames for a first feature point among the at least one feature point, wherein the effective tracking frame is a frame after the first frame of the video to be processed and including a second feature point, and a feature point motion difference between the second feature point and the first feature point is less than a first preset threshold, and the first feature point is any feature point among the at least one feature point; when the number is greater than the second preset threshold, the first feature point is determined as a effective feature point.

[0019] Optionally, the valid tracking frame has a corresponding relationship with the first preset threshold, and the first preset threshold corresponding to any valid tracking frame is determined based on a preset distance coefficient and an average value of feature point motion differences of any valid tracking frame, and the average value of feature point motion differences of any valid tracking frame is an average value of feature point motion differences between feature points of any valid tracking frame and corresponding feature points of a previous frame of any valid tracking frame.

[0020] Optionally, the second preset threshold is determined according to a preset amplitude coefficient and the number of frames included in the video to be processed.

[0021] Optionally, the repair module is configured to: based on the correspondence between the effective feature points of each frame of the video to be processed and the effective feature points of the previous frame of each frame, smooth the correspondence to obtain a smoothed correspondence; based on the smoothed correspondence, perform motion compensation on the previous frame of each frame.

[0022] Optionally, the repair module is configured to: obtain a motion trajectory based on the correspondence between the effective feature points of each frame of the video to be processed and the effective feature points of the previous frame of each frame; determine the degree of picture jitter based on the motion trajectory; determine a smoothing parameter based on the degree of picture jitter; and use the smoothing parameter to smooth the correspondence to obtain a smoothed correspondence.

[0023] Optionally, the video processing device also includes: a division module, configured to divide all frames of the video to be processed into at least one frame group; the repair module is configured to repair the previous frame of each frame contained in each frame group according to the correspondence between the valid feature points of each frame contained in each frame group in the at least one frame group and the valid feature points of the previous frame of each frame.

[0024] Optionally, the feature point motion difference is the Euclidean distance between the feature points.

[0025] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement a video processing method according to the present disclosure.

[0026] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the video processing method according to the present disclosure.

[0027] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the video processing method according to the present disclosure is implemented.

[0028] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0029] The effective feature points of each frame can be extracted to repair each frame, that is, the invalid feature points in at least one feature point of each frame can be eliminated, and the picture motion can be analyzed based on the stable feature points of each frame, and then each frame can be repaired. This can avoid frame distortion and make the naturalness and coordination of the frame after jitter repair better.

[0030] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0032] Figure 1 is a flowchart illustrating a video processing method according to an exemplary embodiment of the present disclosure;

[0033] Figure 2 is a specific implementation flow chart showing a video processing method according to an exemplary embodiment of the present disclosure;

[0034] Figure 3 is a block diagram illustrating a video processing apparatus according to an exemplary embodiment of the present disclosure;

[0035] Figure 4 is a block diagram illustrating an electronic device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0036] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0037] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The implementation methods described in the following examples do not represent all implementation methods consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the attached claims.

[0038] It should be noted that the phrase "at least one of the items" in the present disclosure includes three types of parallel situations: "any one of the items", "a combination of any number of the items", and "all of the items". For example, "including at least one of A and B" includes the following three parallel situations: (1) including A; (2) including B; (3) including A and B. Another example is "executing at least one of step 1 and step 2" which means the following three parallel situations: (1) executing step 1; (2) executing step 2; (3) executing step 1 and step 2.

[0039] At present, multimedia technology has greatly enriched people's lives in modern society, especially video technology, which meets people's various needs. It is an indispensable tool in social development and production life, and is also a very common means of recording and sharing life. Digital cameras and smart terminals are important tools for video recording. In the absence of professional recording equipment, such as stabilizers, the video images shot often shake, especially when the video recording equipment is in motion, the shaking of the video images shot will be more severe, which strongly affects the perception and quality of the video images.

[0040] To solve the jitter problem of video images, you can use professional equipment to record videos to avoid jitter in the video images. However, using professional equipment to record videos will bring expensive usage costs; alternatively, you can use post-processing methods to repair the jittery video images to make the video images stable. However, in the related art, the motion of the image is adjusted only by estimating the motion offset between frames. This method simply estimates the offset between frames, and then corrects the target position in the video image based on the estimation result, which often causes the video image to be distorted, resulting in the naturalness and coordination of the video image after the jitter repair deteriorating.

[0041] In order to solve the problem in the related art that when repairing a jittery video image, the video image may be distorted, resulting in the deterioration of the naturalness and coordination of the video image after the jitter repair, the video processing method proposed in the present invention can extract the effective feature points of each frame to repair each frame. For example, the invalid feature points in at least one feature point of each frame can be eliminated, and the picture motion can be analyzed based on the stable feature points of each frame, and then each frame can be repaired. This can avoid the distortion of the frame and make the naturalness and coordination of the frame after the jitter repair better.

[0042] Figure 1 Detailed description is a flowchart illustrating a video processing method according to an exemplary embodiment of the present disclosure, which can be applied to various devices, such as mobile phones, desktops, laptops, tablets, etc.

[0043] Reference Figure 1 In step 101, at least one feature point of each frame of the video to be processed is obtained. The video to be processed may include multiple frames.

[0044] According to an exemplary embodiment of the present disclosure, at least one feature point of each frame is a corner point of each of the multiple regions divided into each frame. In one example, a video frame can be divided into multiple regions, and the corner point of each of the multiple regions can be determined as at least one feature point of the video frame. For example, each frame can be divided into multiple grid regions of the same size, and the Harris corner points H of the grid regions can be determined as the at least one feature point of the video frame. j As the feature points of the grid area, 1≤j≤M, M is the total number of grid areas divided in each frame, and all the feature points of each frame can be recorded as All feature points By M H j composition.

[0045] In one example, a "Harris corner point" may be an extreme point, i.e., a point with particularly prominent attributes in some aspect. For example, it may be an isolated point with the maximum or minimum intensity in some attribute, the end point of a line segment, or a point with the maximum local curvature on a curve, etc. Of course, the feature points disclosed in the present invention are not limited to regional corner points, and feature points may also be determined in other ways. In this way, since the corner points have particularly prominent attributes in some aspect, i.e., the corner points are points with relatively distinct features, it is more convenient to identify and track the corner points as feature points of the video frame.

[0046] According to an exemplary embodiment of the present disclosure, for the first frame of the video to be processed, at least one feature point of the first frame may be a corner point of each of the multiple regions divided by the first frame; for each subsequent frame after the first frame of the video to be processed, the feature point of each subsequent frame is a tracking feature point obtained based on the feature point of the previous frame and corresponding to the feature point of the previous frame. Further, for each subsequent frame, an optical flow method may be used to track at least one feature point of the previous frame to obtain at least one feature point of the subsequent frame.

[0047] For example, for the second frame, the optical flow method can be used to track at least one feature point (feature point 11, feature point 12) of the first frame to obtain at least one feature point (feature point 21 corresponding to feature point 11, feature point 22 corresponding to feature point 12) of the second frame; for the third frame, the optical flow method can be used to track at least one feature point (feature point 21, feature point 22) of the second frame to obtain at least one feature point (feature point 31 corresponding to feature point 21, feature point 32 corresponding to feature point 22) of the third frame. By analogy, at least one feature point of each frame in the video to be processed can be obtained.

[0048] In step 102, a feature point motion difference between each feature point of each frame of the video to be processed and a corresponding feature point of a previous frame of each frame may be determined.

[0049] According to an exemplary embodiment of the present disclosure, the feature point motion difference may be the Euclidean distance between the feature points.

[0050] For example, for the second frame I2 of the video to be processed, each feature point can be calculated Its tracked feature points The Euclidean distance d between l It should be understood that the feature point p 1,l is the feature point in the first frame I1; for the third frame I3 of the video to be processed, each feature point can be calculated Its tracked feature points The Euclidean distance d between l And so on. Wherein, 1≤l≤L, L is the total number of feature points in each frame. Of course, the feature point motion difference is not limited to the Euclidean distance, but can also be other forms of distance, and the above implementation is only an exemplary description.

[0051] In step 103, a valid feature point can be determined from at least one feature point based on the feature point motion difference between each feature point of each frame of the video to be processed and the corresponding feature point of the previous frame of each frame. The feature point motion difference of the feature point is inversely correlated with the effectiveness of the feature point, that is, the greater the feature point motion difference of the feature point, the lower the effectiveness of the feature point. The effectiveness of the feature point can indicate the quality of the feature point tracking, that is, it can indicate the stability of the feature point. The higher the effectiveness of the feature point, the more accurate the tracking of the feature point is, and the better the stability of the feature point is.

[0052] According to an exemplary embodiment of the present disclosure, for a first feature point in at least one feature point, the number of valid tracking frames can be determined. The valid tracking frame is a frame after the first frame of the video to be processed and includes a second feature point, the feature point motion difference between the second feature point and the first feature point is less than a first preset threshold, and the first feature point is any feature point in the at least one feature point. When the number of valid tracking frames is greater than the second preset threshold, the first feature point can be determined as a valid feature point, wherein the valid feature point is a feature point with a higher degree of validity, that is, the valid feature point is a feature point with relatively good stability.

[0053] In one embodiment, for each feature point in at least one feature point, the following operations may be performed: the number of frames after the first frame of the video to be processed and including the current feature point whose feature point motion difference is less than a first preset threshold is determined, and when the number is greater than a second preset threshold, the current feature point may be determined as a valid feature point. In this way, when the number of valid tracking frames is greater than the second preset threshold, it means that the number of valid tracking times of the corresponding first feature point is sufficient, and there is sufficient reason to believe that the first feature point is a valid feature point, and the accuracy of determining the valid feature point is good.

[0054] For example, assuming that the video to be processed contains 5 frames, the lth feature point can be compared Its tracked feature points The Euclidean distance d between l The magnitude relationship between the first preset threshold value of the second frame. l >When the first preset threshold is reached, the lth feature point p contained in the second frame can be determined 2,l Relative to the tracked feature point p in the first frame 1,l Invalid tracking; when d l ≤ the first preset threshold, the lth feature point p contained in the second frame can be determined 2,l Relative to the tracked feature point p in the first frame 1,lFor effective tracking, that is, for the lth feature point, the second frame belongs to the effective tracking frame. Similarly, for the third frame to the fifth frame, the total number of effective tracking frames corresponding to the lth feature point can be determined by traversing in sequence according to the above method. When the total number of effective tracking frames corresponding to the lth feature point is greater than the second preset threshold, the lth feature point can be determined as a valid feature point.

[0055] Of course, the method of determining effective feature points is not limited to this. Other possible methods can be used to determine effective feature points based on the principle that the feature point motion difference is inversely correlated with its effectiveness. For example, for each feature point, the sum of its feature point motion differences between any two adjacent frames can be counted. The sum is inversely correlated with the effectiveness of the feature point, that is, the larger the sum, the worse the effectiveness of the feature point. When the sum does not exceed the set threshold, the feature point can be determined to be a valid feature point, and so on.

[0056] According to an exemplary embodiment of the present disclosure, there is a corresponding relationship between the effective tracking frame and the first preset threshold value, and the first preset threshold value corresponding to any effective tracking frame can be determined based on the preset distance coefficient g and the average value ε of the feature point motion difference of any effective tracking frame. Among them, the average value ε of the feature point motion difference of any effective tracking frame is the average value of the feature point motion difference between the feature point of any effective tracking frame and the corresponding feature point of the previous frame of any effective tracking frame. In this way, since the average value of the feature point motion difference of the video frame can reflect the average level of the corresponding feature points of the previous frame tracked by all feature points contained in the video frame, therefore, the first preset threshold value of the video frame is determined based on the preset distance coefficient and the average value of the feature point motion difference of the video frame, and the method of determining the effective feature points in the video frame based on the first preset threshold value is relatively objective, which can ensure the accuracy of determining the effective feature points.

[0057] In one example, the first preset threshold may be set for each frame after the first frame of the video to be processed, that is, there is a corresponding relationship between the video frame and the first preset threshold, and the first preset thresholds of different video frames may be the same or different.

[0058] For example, suppose the video to be processed contains 5 frames in total. For frame 2 of the video to be processed, the nth feature point can be calculated Its tracked feature points The Euclidean distance d between n; The Euclidean distance between each feature point of frame 2 and the corresponding feature point of frame 1 may be averaged, that is, the sum of the Euclidean distances between each feature point of frame 2 and the corresponding feature point of frame 1 may be calculated, and the sum may be divided by the total number of feature points of frame 2 to obtain the average Euclidean distance ε2 of frame 2. Further, the first preset threshold of frame 2 may be the product g*ε2 of the preset distance coefficient g and the average Euclidean distance ε2 of frame 2.

[0059] Next, we can compare the nth feature point Its tracked feature points The Euclidean distance d between n The magnitude relationship between d and the first preset threshold g*ε2 of frame 2. n >g*ε2, the nth feature point p contained in frame 2 can be determined 2,n The tracked feature point p relative to frame 1 1,n Invalid tracking; when d n When ≤g*ε2, the nth feature point p contained in frame 2 can be determined 2,n The tracked feature point p relative to frame 1 1,n For effective tracking, at this time, for the nth feature point, frame 2 is the effective tracking frame.

[0060] Similarly, for frame 3 of the video to be processed, the nth feature point can be calculated Its tracked feature points The Euclidean distance d between n ; The Euclidean distance between each feature point of frame 3 and the corresponding feature point of frame 2 may also be averaged, that is, the sum of the Euclidean distances between each feature point of frame 3 and the corresponding feature point of frame 2 may be calculated, and the sum may be divided by the total number of at least one feature point to obtain the average Euclidean distance ε3 of frame 3. Further, the first preset threshold of frame 3 may be the product g*ε3 of the preset distance coefficient g and the average Euclidean distance ε3 of frame 3.

[0061] Similarly, the nth feature point can be compared Its tracked feature points The Euclidean distance d between n The magnitude relationship between d and the first preset threshold g*ε3 of frame 3. n >g*ε3, the nth feature point p contained in frame 3 can be determined 3,n The tracked feature point p relative to frame 2 2,n Invalid tracking; when d n When ≤g*ε3, the nth feature point p contained in frame 3 can be determined 3,n The tracked feature point p relative to frame 2 2,nFor effective tracking, at this time, for the nth feature point, frame 3 is the effective tracking frame. And so on, for the nth feature point, it can be determined whether frame 4 and frame 5 are effective tracking frames.

[0062] Based on the above embodiments, it is possible to determine the Euclidean distance d between the nth feature point of a frame and the nth feature point of the previous frame of this frame after frame 1 of the video to be processed n The number of frames whose Euclidean distance is less than the first preset threshold of this frame, that is, for the nth feature point, it is possible to determine the number of effective tracking frames after frame 1 of the video to be processed. When the number of effective tracking frames is greater than the second preset threshold, the nth feature point can be determined as an effective feature point.

[0063] According to an exemplary embodiment of the present disclosure, the second preset threshold can be determined according to a preset amplitude coefficient and the number of frames included in the video to be processed. In one example, the second preset threshold can be the product of the preset amplitude coefficient q and the number of frames included in the video to be processed. Where 0 < q < 1. As described above, assuming that the video to be processed includes a total of 5 frames, the second preset threshold can be 5q. For the nth feature point, if the number of effective tracking frames after frame 1 of the video to be processed is greater than the second preset threshold 5q, the nth feature point can be determined as an effective feature point.

[0064] If the number of effective tracking frames after frame 1 of the video to be processed is less than or equal to the second preset threshold 5q, the nth feature point can be determined as an invalid feature point. Next, the nth feature point p of the invalid feature point i,n can be removed from all the feature points of the ith frame to obtain the set of effective feature points of the ith frame That is, the nth feature point can be removed from all the feature points of each frame. At this time, a corresponding set of effective feature points can be obtained for each frame. In this way, since the number of effective tracking frames cannot exceed the total number of video frames included in the video to be processed, the second preset threshold can be determined according to the preset amplitude coefficient and the number of frames included in the video to be processed, which can avoid errors in the confirmation of effective feature points caused by setting the second preset threshold too small or too large, and can ensure the accuracy of determining effective feature points.

[0065] In step 104, the previous frame of each frame of the video to be processed can be repaired according to the corresponding relationship between the effective feature points of each frame of the video to be processed and the effective feature points of the previous frame of each frame.

[0066] According to an exemplary embodiment of the present disclosure, it is possible to based on the effective feature points of each frame Determine the corresponding relationship of each frame to the previous frame. The corresponding relationship is used to describe the relationship between three-dimensional points on the same plane in space in two video frames. In an example, the corresponding relationship can be a homography matrix O i-1 Furthermore, we can use the effective feature points of each frame And the valid feature points of the previous frame of each frame The minimum median value method (LMEDS) is used to determine the homography matrix O corresponding to the previous frame of each frame. i-1 .

[0067] Next, for each frame of the video to be processed, the previous frame can be repaired based on the homography matrix corresponding to the current frame. For example, as mentioned above, assuming that the video to be processed contains a total of 5 frames, then for frame 1, frame 2, frame 3, and frame 4, they can all be repaired based on their corresponding homography matrices, that is:

[0068] The homography matrix O1 corresponding to frame 1 can be used to repair frame 1;

[0069] The homography matrix O2 corresponding to frame 2 can be used to repair frame 2;

[0070] The homography matrix O3 corresponding to frame 3 can be used to repair frame 3;

[0071] Frame 4 can be repaired using the homography matrix O4 corresponding to frame 4.

[0072] It should be noted that, for the last frame of the video to be processed, its corresponding homography matrix can be a unit matrix, that is, the present disclosure may not perform jitter repair on the last frame of the video to be processed, but only perform jitter repair on the first frame to the second to last frame of the video to be processed. Alternatively, the above unit matrix may be used to perform jitter repair on the last frame of the video to be processed.

[0073] According to an exemplary embodiment of the present disclosure, based on the correspondence between the effective feature points of each frame of the video to be processed and the effective feature points of the previous frame of each frame, the correspondence can be smoothed to obtain a smoothed correspondence. Then, based on the smoothed correspondence, motion compensation can be performed on the previous frame of each frame, thereby repairing the previous frame of each frame. In this way, repairing the previous frame of each frame based on the smoothed correspondence can achieve a better effect of jitter repair.

[0074] For example, based on the homography matrix corresponding to the previous frame of each frame of the video to be processed, the homography matrix corresponding to the current frame can be smoothed to obtain the homography matrix after smoothing corresponding to the current frame.

[0075] Based on the four homography matrices O1, O2, O3 and O4 corresponding to frames 1, 2, 3 and 4 of the video to be processed, the homography matrix O1 corresponding to frame 1 can be smoothed to obtain the smoothed homography matrix corresponding to frame 1; the homography matrix O2 corresponding to frame 2 can be smoothed to obtain the smoothed homography matrix corresponding to frame 2; the homography matrix O3 corresponding to frame 3 can be smoothed to obtain the smoothed homography matrix corresponding to frame 3; the homography matrix O4 corresponding to frame 4 can be smoothed to obtain the smoothed homography matrix corresponding to frame 4. Next, the previous frame of each frame can be repaired based on the smoothed homography matrix corresponding to the previous frame of each frame, that is:

[0076] Frame 1 can be repaired based on the homography matrix O1 after smoothing corresponding to frame 1;

[0077] Frame 2 can be repaired based on the homography matrix O2 after smoothing corresponding to frame 2;

[0078] Frame 3 can be repaired based on the homography matrix O3 after smoothing corresponding to frame 3;

[0079] Frame 4 may be repaired based on the homography matrix O4 corresponding to frame 4 after smoothing.

[0080] According to an exemplary embodiment of the present disclosure, the motion trajectory can also be obtained based on the correspondence between the effective feature points of each frame of the video to be processed and the effective feature points of the previous frame of each frame. It should be noted that the Gaussian smoothing method can be used to smooth the homography matrix, that is, a low-pass filter can be used to low-pass filter the homography matrix. Further, the homography matrix corresponding to the previous frame of each frame of the video to be processed can be multiplied to obtain the motion trajectory. For example, the four homography matrices O1, O2, O3 and O4 corresponding to the frames 1, 2, 3 and 4 of the video to be processed can be multiplied to obtain the motion trajectory.

[0081] Based on the above embodiment, the degree of picture jitter can be determined based on the motion trajectory, wherein the greater the difference in coordinates of the corresponding feature points in two adjacent frames of the video to be processed indicated by the motion trajectory, the greater the degree of picture jitter, that is, the difference in coordinates of the corresponding feature points is positively correlated with the degree of picture jitter. Next, a smoothing parameter can be determined according to the degree of picture jitter, wherein the greater the degree of picture jitter, the greater the smoothing strength indicated by the smoothing parameter, that is, the degree of picture jitter is positively correlated with the smoothing strength.

[0082] In one embodiment, the correspondence relationship can be smoothed using a smoothing parameter to obtain a smoothed correspondence relationship. In one possible embodiment, for the previous frame of each frame of the video to be processed, the homography matrix corresponding to the current frame can be smoothed based on a determined smoothing strength. For example, for the homography matrix O1 corresponding to frame 1 of the video to be processed, the homography matrix O1 can be smoothed using a determined smoothing strength; for the homography matrix O2 corresponding to frame 2 of the video to be processed, the homography matrix O2 can be smoothed using a determined smoothing strength; for the homography matrix O3 corresponding to frame 3 of the video to be processed, the homography matrix O3 can be smoothed using a determined smoothing strength; for the homography matrix O4 corresponding to frame 4 of the video to be processed, the homography matrix O4 can be smoothed using a determined smoothing strength.

[0083] Based on the above embodiments, the homography matrix after smoothing corresponding to each frame can be used to perform coordinate transformation on the frame, so as to obtain the frame after the jitter is repaired. For example, the homography matrix after smoothing corresponding to frame 1 can be used to perform coordinate transformation on frame 1, so as to obtain the frame after the jitter is repaired in frame 1; the homography matrix after smoothing corresponding to frame 2 can be used to perform coordinate transformation on frame 2, so as to obtain the frame after the jitter is repaired in frame 2, and so on. In this way, the degree of picture jitter can be analyzed first, and then the homography matrix can be adaptively smoothed according to the degree of picture jitter, so as to achieve a better effect of jitter repair.

[0084] According to an exemplary embodiment of the present disclosure, all frames of the video to be processed can also be divided into at least one frame group. Next, the previous frame of each frame contained in each frame group can be repaired based on the correspondence between the valid feature points of each frame contained in each frame group in the at least one frame group and the valid feature points of the previous frame of each frame.

[0085] For example, assuming that the video to be processed contains N frames, the N frames can be divided into K groups, and the number of frames contained in each group is N / K. The following operations can be performed for each frame group: determine at least one feature point of each frame in the current frame group, and the feature point motion difference between each feature point of each frame of the current frame group and the corresponding feature point of the previous frame of each frame. Next, based on the feature point motion difference between each feature point of each frame of the current frame group and the corresponding feature point of the previous frame of each frame, a valid feature point can be determined from at least one feature point. Among them, the feature point motion difference of each feature point is negatively correlated with its effectiveness. Finally, the previous frame of each frame of the current frame group can be repaired according to the corresponding relationship between the valid feature points of each frame of the current frame group and the valid feature points of the previous frame of each frame. In this way, by grouping the video frames contained in the video to be processed, the picture motion between the frame groups can be made relatively small, which is conducive to the tracking of feature points.

[0086] Figure 2 is a flowchart showing a specific implementation of a video processing method according to an exemplary embodiment of the present disclosure.

[0087] Reference Figure 2 In step 201, all frames of the video to be processed are divided into at least one frame group. For example, assuming that the video to be processed contains N frames, the N frames can be divided into K groups, and the number of frames contained in each group is N / K.

[0088] In step 202, at least one feature point of each frame in the first set of frames is determined.

[0089] For example, first, frame 1 can be divided into multiple grid regions of equal size, and the Harris corner points of each grid region are As the feature points of the grid area, 1≤j≤M, M is the total number of grid areas divided in frame 1, and all feature points of frame 1 can be recorded as

[0090] For frame 2, we can use the optical flow method to track all the feature points in frame 1. Thus, all the feature points of frame 2 are obtained And so on, each time the optical flow method can be used to track the feature points of the previous frame to obtain the feature points of the current frame.

[0091] In step 203, for each frame in the first set of frames, a valid feature point among at least one feature point of the frame is determined.

[0092] For example, for the i-th frame in the first set of frames, the l-th feature point can be calculated Its tracked feature points The Euclidean distance d between l , where 1≤l≤L, L is the total number of feature points in each frame. The Euclidean distance between each feature point in the i-th frame and the corresponding feature point in the (i-1)-th frame can also be averaged, that is, the sum of the Euclidean distances between each feature point in the i-th frame and the corresponding feature point in the (i-1)-th frame can be calculated, and the sum can be divided by the total number of at least one feature point to obtain the average Euclidean distance ε of the i-th frame. i Furthermore, the first preset threshold value of the i-th frame may be the average value ε of the Euclidean distance between the preset distance coefficient g and the i-th frame. i The product of g*ε i , where g>1.

[0093] It should be noted that for each feature point, the number of valid tracking times can also be counted, and the “number of valid tracking times” is the “number of valid tracking frames” in the previous embodiment. i,l Indicates the cumulative number of valid tracking of the lth feature point in the i-th frame, and s can be set 1,l =0, that is, the cumulative number of effective tracking of the lth feature point in the first frame is zero.

[0094] Next, we can compare the lth feature point Its tracked feature points The Euclidean distance d between l and the first preset threshold g*ε of the i-th frame i When d l >g*ε i When , the lth feature point p contained in the i-th frame can be determined i,l Relative to the tracked feature point p in the (i-1)th frame i-1,l Invalid tracking, at this time, s i,l =s i-1,l +0=s i-1,l When d l ≤g*ε i When , the lth feature point p contained in the i-th frame can be determined i,l Relative to the tracked feature point p in the (i-1)th frame i-1,l For effective tracking, at this time, s i,l =s i-1,l +1.

[0095] For example, assuming that the video to be processed contains 5 frames, the lth feature point can be compared Its tracked feature points The Euclidean distance d between l The relationship between the value of g*ε2 and the first preset threshold value g*ε2 of the second frame. l >g*ε2, the lth feature point p contained in the second frame can be determined 2,l Relative to the tracked feature point p in the first frame 1,l Invalid tracking, at this time, s 2,l =s 1,l +0=0+0=0, that is, the cumulative number of effective tracking of the lth feature point in the second frame is zero; when d l When ≤g*ε2, the lth feature point p contained in the second frame can be determined 2,l Relative to the tracked feature point p in the first frame 1,l For effective tracking, at this time, s 2,l =s 1,l+1 = 0 + 1 = 1, that is, the effective tracking cumulative count of the l-th feature point in the second frame is 1 at this time.

[0096] And so on, the Euclidean distance d between the l-th feature point and the feature point it is tracked with l can be compared with the first preset threshold g*ε3 of the third frame, and then the effective tracking cumulative count s 3,l of the l-th feature point in the third frame can be obtained; the Euclidean distance d between the l-th feature point and the feature point it is tracked with l can be compared with the first preset threshold g*ε4 of the fourth frame, and then the effective tracking cumulative count s 4,l of the l-th feature point in the fourth frame can be obtained; the Euclidean distance d between the l-th feature point and the feature point it is tracked with l can be compared with the first preset threshold g*ε5 of the fifth frame, and then the effective tracking cumulative count s 5,l of the l-th feature point in the fifth frame can be obtained.

[0097] In this way, for the l-th feature point, the effective tracking cumulative count s N / K,l of the l-th feature point in the N / K-th frame can be obtained. Among them, 1 ≤ l ≤ L, and L is the total number of feature points included in each frame of the first group of frames.

[0098] The product of the preset amplitude coefficient q and the number of frames N / K included in the first group of frames can be calculated: q*N / K as the second preset threshold corresponding to the first group of frames, where 0 < q < 1. Next, each of the L feature points included in each frame can be traversed. When s N / K,l < q*N / K, it indicates that the effective tracking cumulative count of the l-th feature point is small, and this l-th feature point will be regarded as an invalid feature point. Then, all frames within the first group of frames can be traversed, and the feature point p il can be removed from all the feature points of the i-th frame to obtain the set of effective feature points

[0099] In step 204, the homography matrix corresponding to each frame in the first group of frames is obtained.

[0100] For example, for the i-th frame in the first group of frames, according to the effective feature points of the i-th frame and the previous frame of the i-th frame, that is, the effective feature points Determine the homography matrix O corresponding to the (i-1)th frame i-1 . Furthermore, we can use the effective feature points of the i-th frame and the previous frame of the i-th frame, i.e., the valid feature points of the (i-1)th frame Use the minimum median value method (LMEDS) to determine the homography matrix O corresponding to the (i-1)th frame i-1 For example, based on the valid feature points of the second frame and the previous frame of the second frame, that is, the valid feature points of the first frame Determine the homography matrix O1 corresponding to the first frame; according to the valid feature points of the fourth frame and the previous frame of the 4th frame, that is, the valid feature points of the 3rd frame Determine the homography matrix O3 corresponding to the third frame, and so on.

[0101] It should be noted that, for the last frame in the first set of frames, its corresponding homography matrix can always be a unit matrix, that is, the present disclosure may not perform jitter repair on the last frame in the first set of frames, but perform jitter repair on the first frame to the second to last frame in the first set of frames. Alternatively, the above-mentioned unit matrix may be used to perform jitter repair on the last frame of the first set of frames.

[0102] In step 205, for other frame groups, the calculation process of the first frame group is repeated to obtain the homography matrix corresponding to each frame in the corresponding frame group.

[0103] In step 206, a homography matrix corresponding to each frame from the first frame to the (N-1)th frame contained in the video to be processed is smoothed to obtain a homography matrix corresponding to the frame after smoothing.

[0104] For example, the Gaussian smoothing method can be used to smooth the homography matrix, that is, a low-pass filter can be used to low-pass filter the homography matrix. i , 1≤i≤N, the homography matrix corresponding to each frame of the video to be processed can be multiplied to obtain the motion trajectory. It should be noted that, as mentioned above, since the homography matrix corresponding to the last frame of the video to be processed is always a unit matrix, all homography matrices O i The result of the multiplication is actually the homography matrix O1 corresponding to the first frame ~ the homography matrix O corresponding to the (N-1)th frame N-1 The result of multiplication of .

[0105] Then, the degree of picture shaking can be determined based on the motion trajectory. Next, the smoothing strength can be determined according to the degree of picture shaking, wherein the greater the degree of picture shaking, the greater the smoothing strength.

[0106] Next, for the first frame of the video to be processed, the homography matrix O1 corresponding to the first frame can be smoothed based on the determined smoothing strength; for the second frame of the video to be processed, the homography matrix O2 corresponding to the second frame can be smoothed based on the determined smoothing strength; ...; for the (N-1)th frame of the video to be processed, the homography matrix O2 corresponding to the (N-1)th frame can be smoothed based on the determined smoothing strength. N-1 Smoothing and more.

[0107] In step 207, coordinate transformation is performed on each frame using the homography matrix after smoothing corresponding to the frame, so as to obtain a frame after the jitter is repaired.

[0108] For example, the smoothed homography matrix corresponding to the first frame can be used to perform coordinate transformation on the first frame, so as to obtain the first frame after the jitter is repaired;...; the smoothed homography matrix corresponding to the (N-1) frame can be used to perform coordinate transformation on the (N-1) frame, so as to obtain the (N-1) frame after the jitter is repaired, and so on.

[0109] Figure 3 is a block diagram illustrating a video processing apparatus according to an exemplary embodiment of the present disclosure.

[0110] Reference Figure 3 The device 300 may include an acquisition module 301, a feature point motion difference determination module 302, a valid feature point determination module 303 and a repair module 304.

[0111] The acquisition module 301 may acquire at least one feature point of each frame of the video to be processed, wherein the video to be processed may include multiple frames.

[0112] According to an exemplary embodiment of the present disclosure, at least one feature point of each frame is a corner point of each of the multiple regions divided into each frame. In one example, a video frame can be divided into multiple regions, and the corner point of each of the multiple regions can be determined as at least one feature point of the video frame. For example, each frame can be divided into multiple grid regions of the same size, and the Harris corner points H of the grid regions can be determined as the at least one feature point of the video frame. j As the feature points of the grid area, 1≤j≤M, M is the total number of grid areas divided in each frame, and all the feature points of each frame can be recorded as All feature points By M H j composition.

[0113] In one example, a "Harris corner point" may be an extreme point, i.e., a point with particularly outstanding attributes in some aspect. For example, it may be an isolated point with the maximum or minimum intensity in some attribute, the end point of a line segment, or a point with the maximum local curvature on a curve, etc. Of course, the feature points disclosed in the present invention are not limited to regional corner points, and feature points may also be determined in other ways.

[0114] According to an exemplary embodiment of the present disclosure, for the first frame of the video to be processed, at least one feature point of the first frame may be a corner point of each of the multiple regions divided by the first frame; for each subsequent frame after the first frame of the video to be processed, the feature point of each subsequent frame is a tracking feature point obtained based on the feature point of the previous frame and corresponding to the feature point of the previous frame. Further, for each subsequent frame, an optical flow method may be used to track at least one feature point of the previous frame to obtain at least one feature point of the subsequent frame.

[0115] The feature point motion difference determination module 302 may determine the feature point motion difference between each feature point of each frame of the video to be processed and the corresponding feature point of the previous frame of each frame.

[0116] According to an exemplary embodiment of the present disclosure, the feature point motion difference may be the Euclidean distance between the feature points.

[0117] The effective feature point determination module 303 can determine effective feature points from at least one feature point based on the feature point motion difference between each feature point of each frame of the video to be processed and the corresponding feature point of the previous frame of each frame. The feature point motion difference of the feature point is inversely correlated with the effectiveness of the feature point, that is, the greater the feature point motion difference of the feature point, the lower the effectiveness of the feature point. The effectiveness of the feature point can indicate the quality of the feature point tracking, that is, it can indicate the stability of the feature point. The higher the effectiveness of the feature point, the more accurate the tracking of the feature point is, and the better the stability of the feature point is.

[0118] According to an exemplary embodiment of the present disclosure, for a first feature point among at least one feature point, the effective feature point determination module 303 may determine the number of effective tracking frames. The effective tracking frame is a frame after the first frame of the video to be processed and includes a second feature point, the feature point motion difference between the second feature point and the first feature point is less than a first preset threshold, and the first feature point is any feature point among the at least one feature point. When the number of effective tracking frames is greater than the second preset threshold, the effective feature point determination module 303 may determine the first feature point as an effective feature point, wherein the effective feature point is a feature point with a higher degree of effectiveness, that is, the effective feature point is a feature point with relatively good stability.

[0119] In one embodiment, for each of at least one feature point, the following operations may be performed: Determine the number of frames after the first frame of the video to be processed and including the current feature point with a feature point motion difference less than a first preset threshold. When this number is greater than a second preset threshold, the current feature point may be determined as a valid feature point.

[0120] Of course, the method for determining valid feature points is not limited to this. According to the principle that the feature point motion difference of a feature point is inversely correlated with its validity, valid feature points may also be determined by other possible means. For example, for each feature point, the sum of the feature point motion differences between any two adjacent frames may also be statistically calculated. This sum is inversely correlated with the validity of the feature point, that is, the larger the sum, the worse the validity of the feature point. When this sum does not exceed a set threshold, the feature point may be determined as a valid feature point, and so on.

[0121] According to an exemplary embodiment of the present disclosure, there is a correspondence between valid tracking frames and a first preset threshold. The first preset threshold corresponding to any valid tracking frame may be determined according to a preset distance coefficient g and the average value ε of the feature point motion differences of any valid tracking frame. The average value ε of the feature point motion differences of any valid tracking frame is the average value of the feature point motion differences between the feature points of any valid tracking frame and the corresponding feature points of the previous frame of any valid tracking frame.

[0122] In one example, the first preset threshold may be set for each frame after the first frame of the video to be processed, that is, there is a correspondence between video frames and the first preset threshold. The first preset thresholds of different video frames may be the same or different.

[0123] According to an exemplary embodiment of the present disclosure, the second preset threshold may be determined according to a preset amplitude coefficient and the number of frames included in the video to be processed. In one example, the second preset threshold may be the product of the preset amplitude coefficient q and the number of frames included in the video to be processed. Where 0 < q < 1. As described above, assuming that the video to be processed contains a total of 5 frames, the second preset threshold may be 5q. For the nth feature point, if the number of valid tracking frames after frame 1 of the video to be processed is greater than the second preset threshold 5q, the nth feature point may be determined as a valid feature point.

[0124] If the number of valid tracking frames after frame 1 of the video to be processed is less than or equal to the second preset threshold 5q, the nth feature point may be determined as an invalid feature point. Next, the invalid feature point, the nth feature point p i,n may be removed from all the feature points of the ith frame to obtain the set of valid feature points of the ith frame That is, the nth feature point among all the feature points of each frame can be eliminated, and at this time, the corresponding valid feature point set can be obtained for each frame.

[0125] The repair module 304 can repair the previous frame of each frame of the video to be processed according to the correspondence between the valid feature points of each frame of the video to be processed and the valid feature points of the previous frame of each frame.

[0126] According to an exemplary embodiment of the present disclosure, the effective feature points of each frame can be And the valid feature points of the previous frame of each frame Determine the corresponding relationship of each frame to the previous frame. The corresponding relationship is used to describe the relationship between three-dimensional points on the same plane in space in two video frames. In an example, the corresponding relationship can be a homography matrix O i-1 Furthermore, we can use the effective feature points of each frame And the valid feature points of the previous frame of each frame The minimum median value method (LMEDS) is used to determine the homography matrix O corresponding to the previous frame of each frame. i-1 .

[0127] Next, for each frame of the video to be processed, the previous frame can be repaired based on the homography matrix corresponding to the current frame. For example, as mentioned above, assuming that the video to be processed contains a total of 5 frames, then for frame 1, frame 2, frame 3, and frame 4, they can all be repaired based on their corresponding homography matrices, that is:

[0128] The homography matrix O1 corresponding to frame 1 can be used to repair frame 1;

[0129] The homography matrix O2 corresponding to frame 2 can be used to repair frame 2;

[0130] The homography matrix O3 corresponding to frame 3 can be used to repair frame 3;

[0131] Frame 4 can be repaired using the homography matrix O4 corresponding to frame 4.

[0132] It should be noted that, for the last frame of the video to be processed, its corresponding homography matrix can be a unit matrix, that is, the present disclosure may not perform jitter repair on the last frame of the video to be processed, but only perform jitter repair on the first frame to the second to last frame of the video to be processed. Alternatively, the above unit matrix may be used to perform jitter repair on the last frame of the video to be processed.

[0133] According to an exemplary embodiment of the present disclosure, the repair module 304 can also perform smoothing on the correspondence between the valid feature points of each frame of the video to be processed and the valid feature points of the previous frame of each frame to obtain a smoothed correspondence. Then, based on the smoothed correspondence, motion compensation can be performed on the previous frame of each frame, thereby repairing the previous frame of each frame.

[0134] For example, based on the homography matrix corresponding to the previous frame of each frame of the video to be processed, the homography matrix corresponding to the current frame can be smoothed to obtain the homography matrix after smoothing corresponding to the current frame.

[0135] Based on the four homography matrices O1, O2, O3 and O4 corresponding to frames 1, 2, 3 and 4 of the video to be processed, the homography matrix O1 corresponding to frame 1 can be smoothed to obtain the smoothed homography matrix corresponding to frame 1; the homography matrix O2 corresponding to frame 2 can be smoothed to obtain the smoothed homography matrix corresponding to frame 2; the homography matrix O3 corresponding to frame 3 can be smoothed to obtain the smoothed homography matrix corresponding to frame 3; the homography matrix O4 corresponding to frame 4 can be smoothed to obtain the smoothed homography matrix corresponding to frame 4. Next, the previous frame of each frame can be repaired based on the smoothed homography matrix corresponding to the previous frame of each frame, that is:

[0136] Frame 1 can be repaired based on the homography matrix O1 after smoothing corresponding to frame 1;

[0137] Frame 2 can be repaired based on the homography matrix O2 after smoothing corresponding to frame 2;

[0138] Frame 3 can be repaired based on the homography matrix O3 after smoothing corresponding to frame 3;

[0139] Frame 4 may be repaired based on the homography matrix O4 corresponding to frame 4 after smoothing.

[0140] According to an exemplary embodiment of the present disclosure, the repair module 304 can also obtain the motion trajectory based on the correspondence between the effective feature points of each frame of the video to be processed and the effective feature points of the previous frame of each frame. It should be noted that the Gaussian smoothing method can be used to smooth the homography matrix, that is, a low-pass filter can be used to low-pass filter the homography matrix. Further, the homography matrix corresponding to the previous frame of each frame of the video to be processed can be multiplied to obtain the motion trajectory. For example, the four homography matrices O1, O2, O3 and O4 corresponding to the frames 1, 2, 3 and 4 of the video to be processed can be multiplied to obtain the motion trajectory.

[0141] Based on the above embodiment, the repair module 304 can determine the degree of picture jitter based on the motion trajectory, wherein the greater the difference in coordinates of the corresponding feature points in two adjacent frames in the to-be-processed video indicated by the motion trajectory, the greater the degree of picture jitter, that is, the difference in coordinates of the corresponding feature points is positively correlated with the degree of picture jitter. Next, a smoothing parameter can be determined according to the degree of picture jitter, wherein the greater the degree of picture jitter, the greater the smoothing strength indicated by the smoothing parameter, that is, the degree of picture jitter is positively correlated with the smoothing strength.

[0142] In one embodiment, the repair module 304 can use a smoothing parameter to smooth the correspondence to obtain a smoothed correspondence. In a possible embodiment, for the previous frame of each frame of the video to be processed, the homography matrix corresponding to the current frame can be smoothed based on a determined smoothing strength. For example, for the homography matrix O1 corresponding to frame 1 of the video to be processed, the homography matrix O1 can be smoothed using a determined smoothing strength; for the homography matrix O2 corresponding to frame 2 of the video to be processed, the homography matrix O2 can be smoothed using a determined smoothing strength; for the homography matrix O3 corresponding to frame 3 of the video to be processed, the homography matrix O3 can be smoothed using a determined smoothing strength; for the homography matrix O4 corresponding to frame 4 of the video to be processed, the homography matrix O4 can be smoothed using a determined smoothing strength.

[0143] Based on the above embodiments, the homography matrix after smoothing corresponding to each frame can be used to perform coordinate transformation on the frame, so as to obtain the frame after the jitter is repaired. For example, the homography matrix after smoothing corresponding to frame 1 can be used to perform coordinate transformation on frame 1, so as to obtain the frame after the jitter is repaired in frame 1; the homography matrix after smoothing corresponding to frame 2 can be used to perform coordinate transformation on frame 2, so as to obtain the frame after the jitter is repaired in frame 2, and so on. In this way, the degree of picture jitter can be analyzed first, and then the homography matrix can be adaptively smoothed according to the degree of picture jitter, so as to achieve a better effect of jitter repair.

[0144] According to an exemplary embodiment of the present disclosure, the video processing device 300 may further include a division module, which may divide all frames of the video to be processed into at least one frame group. Next, the repair module 304 may repair the previous frame of each frame contained in each frame group based on the correspondence between the valid feature points of each frame contained in each frame group in the at least one frame group and the valid feature points of the previous frame of each frame.

[0145] For example, assuming that the video to be processed contains N frames, the N frames can be divided into K groups, and the number of frames contained in each group is N / K. The following operations can be performed for each frame group: determine at least one feature point of each frame in the current frame group, and the feature point motion difference between each feature point of each frame of the current frame group and the corresponding feature point of the previous frame of each frame. Next, based on the feature point motion difference between each feature point of each frame of the current frame group and the corresponding feature point of the previous frame of each frame, a valid feature point can be determined from at least one feature point. Among them, the feature point motion difference of each feature point is negatively correlated with its effectiveness. Finally, the previous frame of each frame of the current frame group can be repaired according to the corresponding relationship between the valid feature points of each frame of the current frame group and the valid feature points of the previous frame of each frame. In this way, by grouping the video frames contained in the video to be processed, the picture motion between the frame groups can be made relatively small, which is conducive to the tracking of feature points.

[0146] Figure 4 is a block diagram illustrating an electronic device 400 according to an exemplary embodiment of the present disclosure.

[0147] Reference Figure 4 The electronic device 400 includes at least one memory 401 and at least one processor 402. The at least one memory 401 stores instructions. When the instructions are executed by the at least one processor 402, the video processing method according to the exemplary embodiment of the present disclosure is executed.

[0148] As an example, the electronic device 400 may be a PC, a tablet device, a personal digital assistant, a smart phone, or other device capable of executing the above instructions. Here, the electronic device 400 is not necessarily a single electronic device, but may also be any device or circuit capable of executing the above instructions (or instruction sets) individually or in combination. The electronic device 400 may also be part of an integrated control system or a system manager, or may be configured as a portable electronic device interconnected with a local or remote (e.g., via wireless transmission) interface.

[0149] In the electronic device 400, the processor 402 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller or a microprocessor. As an example and not limitation, the processor may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0150] The processor 402 may execute instructions or codes stored in the memory 401, which may also store data. Instructions and data may also be sent and received over a network via a network interface device, which may employ any known transmission protocol.

[0151] The memory 401 may be integrated with the processor 402, for example, by placing RAM or flash memory within an integrated circuit microprocessor or the like. In addition, the memory 401 may include a separate device, such as an external disk drive, a storage array, or any other storage device that can be used by a database system. The memory 401 and the processor 402 may be operatively coupled, or may communicate with each other, such as through an I / O port, a network connection, etc., so that the processor 402 can read files stored in the memory.

[0152] In addition, the electronic device 400 may also include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.) All components of the electronic device 400 may be connected to each other via a bus and / or a network.

[0153] According to an exemplary embodiment of the present disclosure, a computer-readable storage medium may also be provided, and when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the above-mentioned video processing method. Examples of computer-readable storage media here include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as, multimedia card, secure digital (SD) card or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, any other device is configured to store computer programs and any associated data, data files and data structures in a non-transitory manner and provide the computer programs and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the above-mentioned computer-readable storage medium can be run in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. In addition, in one example, the computer program and any associated data, data files and data structures are distributed on a networked computer system, so that the computer program and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers.

[0154] According to an exemplary embodiment of the present disclosure, a computer program product may also be provided, including a computer program, and when the computer program is executed by a processor, the video processing method according to the present disclosure is implemented.

[0155] According to the video processing method, device, electronic device and storage medium disclosed in the present invention, effective feature points of each frame can be extracted to repair each frame, that is, invalid feature points in at least one feature point of each frame can be eliminated, and the picture motion can be analyzed based on the stable feature points of each frame, and then each frame can be repaired, which can avoid the distortion of the frame and make the naturalness and coordination of the frame repaired by jitter better. Furthermore, since the attributes of corner points are particularly prominent in certain aspects, that is, corner points are points with relatively distinct features, it is more convenient to identify and track corner points as feature points of video frames. Furthermore, when the number of valid tracking frames is greater than the second preset threshold, it means that the number of valid tracking times of the corresponding first feature point is sufficient. At this time, there is sufficient reason to believe that the first feature point is a valid feature point, and the accuracy of determining the valid feature point is better. Further, since the average motion difference of the feature points of the video frame can reflect the average level of the corresponding feature points of the previous frame tracked by all the feature points contained in the video frame, the first preset threshold of the video frame is determined based on the preset distance coefficient and the average motion difference of the feature points of the video frame, and the method of determining the effective feature points in the video frame based on the first preset threshold is relatively objective, which can ensure the accuracy of determining the effective feature points. Further, since the number of effective tracking frames cannot exceed the total number of video frames contained in the video to be processed, the second preset threshold can be determined according to the preset amplitude coefficient and the number of frames contained in the video to be processed, which can avoid the second preset threshold being set too small or too large to cause errors in the confirmation of effective feature points, and ensure the accuracy of determining the effective feature points. Further, the previous frame of each frame can be repaired based on the smoothed corresponding relationship, which can make the effect of jitter repair better; further, the degree of picture jitter can be analyzed first, and then the homography matrix can be adaptively smoothed according to the degree of picture jitter, which can make the effect of jitter repair better; further, by grouping the video frames contained in the video to be processed, the picture motion between frame groups can be relatively small, which is conducive to the tracking of feature points.

[0156] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0157] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A video processing method, characterized in that: include: Obtain at least one feature point of each frame of the video to be processed; Determine a feature point motion difference between each feature point of each frame of the video to be processed and a corresponding feature point of a previous frame of each frame; Determine a valid feature point from the at least one feature point based on the feature point motion difference between each feature point of each frame of the video to be processed and a corresponding feature point of a previous frame of each frame, wherein the feature point motion difference of the feature point is inversely correlated with the validity of the feature point; Repairing the previous frame of each frame of the video to be processed according to the correspondence between the valid feature points of each frame of the video to be processed and the valid feature points of the previous frame of each frame; Wherein, determining a valid feature point from the at least one feature point based on the feature point motion difference between each feature point of each frame of the video to be processed and a corresponding feature point of a previous frame of each frame includes: For a first feature point among the at least one feature point, determining the number of valid tracking frames, wherein the valid tracking frame is a frame after the first frame of the video to be processed and including a second feature point, a feature point motion difference between the second feature point and the first feature point is less than a first preset threshold, and the first feature point is any feature point among the at least one feature point; When the number is greater than a second preset threshold, the first feature point is determined as a valid feature point.

2. The video processing method according to claim 1, characterized in that: The at least one feature point of each frame is a corner point of each area among the multiple areas divided into each frame.

3. The video processing method according to claim 2, characterized in that: For a first frame of the video to be processed, at least one feature point of the first frame is a corner point of each of the multiple regions divided by the first frame; For each subsequent frame after the first frame of the to-be-processed video, the feature points of each subsequent frame are tracking feature points obtained according to the feature points of the previous frame and corresponding to the feature points of the previous frame.

4. The video processing method according to claim 1, characterized in that: The valid tracking frame has a corresponding relationship with the first preset threshold value. The first preset threshold value corresponding to any valid tracking frame is determined based on a preset distance coefficient and an average value of feature point motion differences of any valid tracking frame. The average value of feature point motion differences of any valid tracking frame is an average value of feature point motion differences between feature points of any valid tracking frame and corresponding feature points of a previous frame of any valid tracking frame.

5. The video processing method according to claim 1, characterized in that: The second preset threshold is determined according to a preset amplitude coefficient and the number of frames included in the video to be processed.

6. The video processing method according to claim 1, characterized in that: The repairing of the previous frame of each frame of the video to be processed according to the correspondence between the valid feature points of each frame of the video to be processed and the valid feature points of the previous frame of each frame comprises: Based on the correspondence between the effective feature points of each frame of the video to be processed and the effective feature points of the previous frame of each frame, smoothing the correspondence to obtain a smoothed correspondence; Based on the smoothed corresponding relationship, motion compensation is performed on a previous frame of each frame.

7. The video processing method according to claim 6, characterized in that: The step of smoothing the correspondence between the valid feature points of each frame of the video to be processed and the valid feature points of the previous frame of each frame to obtain the smoothed correspondence includes: Acquire a motion trajectory according to a correspondence between valid feature points of each frame of the video to be processed and valid feature points of a previous frame of each frame; Based on the motion trajectory, determining the degree of image jitter; Determine the smoothing parameters according to the degree of image jitter; The corresponding relationship is smoothed using the smoothing parameter to obtain a smoothed corresponding relationship.

8. The video processing method according to claim 1, characterized in that: Also includes: Dividing all frames of the video to be processed into at least one frame group; The repairing of the previous frame of each frame of the video to be processed according to the correspondence between the valid feature points of each frame of the video to be processed and the valid feature points of the previous frame of each frame comprises: According to the correspondence between the valid feature points of each frame included in each frame group in the at least one frame group and the valid feature points of the previous frame of each frame, the previous frame of each frame included in each frame group is repaired.

9. The video processing method according to any one of claims 1 to 8, characterized in that: The feature point motion difference is the Euclidean distance between the feature points.

10. A video processing device, characterized in that: include: An acquisition module is configured to acquire at least one feature point of each frame of the video to be processed; A feature point motion difference determination module is configured to determine a feature point motion difference between each feature point of each frame of the video to be processed and a corresponding feature point of a previous frame of each frame; An effective feature point determination module is configured to determine an effective feature point from the at least one feature point based on the feature point motion difference between each feature point of each frame of the video to be processed and a corresponding feature point of a previous frame of each frame, wherein the feature point motion difference of the feature point is inversely correlated with the effectiveness of the feature point; A repair module is configured to repair the previous frame of each frame of the video to be processed according to the correspondence between the valid feature points of each frame of the video to be processed and the valid feature points of the previous frame of each frame; Wherein, the effective feature point determination module is configured as follows: For a first feature point among the at least one feature point, determining the number of valid tracking frames, wherein the valid tracking frame is a frame after the first frame of the video to be processed and including a second feature point, a feature point motion difference between the second feature point and the first feature point is less than a first preset threshold, and the first feature point is any feature point among the at least one feature point; When the number is greater than a second preset threshold, the first feature point is determined as a valid feature point.

11. The video processing device according to claim 10, characterized in that: The at least one feature point of each frame is a corner point of each area among the multiple areas divided into each frame.

12. The video processing device according to claim 11, characterized in that: For a first frame of the video to be processed, at least one feature point of the first frame is a corner point of each of the multiple regions divided by the first frame; For each subsequent frame after the first frame of the to-be-processed video, the feature points of each subsequent frame are tracking feature points obtained according to the feature points of the previous frame and corresponding to the feature points of the previous frame.

13. The video processing device according to claim 10, characterized in that: The valid tracking frame has a corresponding relationship with the first preset threshold value. The first preset threshold value corresponding to any valid tracking frame is determined based on a preset distance coefficient and an average value of feature point motion differences of any valid tracking frame. The average value of feature point motion differences of any valid tracking frame is an average value of feature point motion differences between feature points of any valid tracking frame and corresponding feature points of a previous frame of any valid tracking frame.

14. The video processing device according to claim 10, characterized in that: The second preset threshold is determined according to a preset amplitude coefficient and the number of frames included in the video to be processed.

15. The video processing device according to claim 10, characterized in that: The repair module is configured to: Based on the correspondence between the effective feature points of each frame of the video to be processed and the effective feature points of the previous frame of each frame, smoothing the correspondence to obtain a smoothed correspondence; Based on the smoothed corresponding relationship, motion compensation is performed on a previous frame of each frame.

16. The video processing device according to claim 15, characterized in that: The repair module is configured to: Acquire a motion trajectory according to a correspondence between valid feature points of each frame of the video to be processed and valid feature points of a previous frame of each frame; Based on the motion trajectory, determining the degree of image jitter; Determine the smoothing parameters according to the degree of image jitter; The corresponding relationship is smoothed using the smoothing parameter to obtain a smoothed corresponding relationship.

17. The video processing device according to claim 10, characterized in that: The video processing device also includes: A division module, configured to divide all frames of the video to be processed into at least one frame group; The repair module is configured to repair the previous frame of each frame included in each frame group according to the correspondence between the valid feature points of each frame included in each frame group in the at least one frame group and the valid feature points of the previous frame of each frame.

18. The video processing device according to any one of claims 10 to 17, characterized in that: The feature point motion difference is the Euclidean distance between the feature points.

19. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the video processing method according to any one of claims 1 to 9.

20. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the video processing method as claimed in any one of claims 1 to 9.

21. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the video processing method according to any one of claims 1 to 9 is implemented.

Images