Video processing method, device and video stabilization method

Abstract

The present disclosure relates to a video processing method, device, and video stabilization method, the video processing method comprising: obtaining two video frames in a video to be processed; for each of the two video frames, executing: obtaining a first feature point corresponding to the current video frame, performing a position transformation on the first feature point according to a predetermined position transformation method to obtain a second feature point corresponding to the current video frame, combining the first feature point and the second feature point to obtain a combined feature point of the current video frame; and aligning the two video frames based on the combined feature point of each of the two video frames. The video processing method, device, and video stabilization method according to the present disclosure can solve the problem of poor processing effect of video stabilization due to insufficient feature point estimation, improve the video frame alignment effect, and enhance the processing effect of video stabilization.

Description

Technical Field

[0001] The present disclosure relates to the field of video processing technology, and in particular to a video processing method, device and video stabilization method. Background Art

[0002] With the continuous development of video shooting technology, video shooting scenes and shooting equipment are diverse, which puts higher requirements on the stability of video images.

[0003] In the existing video processing technology, video stabilization processing can be performed through a video stabilization algorithm, which mainly uses homography transformation to align the front and back video frames, and the homography transformation needs to be implemented through the homography matrix between the front and back video frames. Here, since the estimation of the homography matrix depends on the extraction result of the feature extraction operator used to extract the feature points of the front and back video frames, when there is no sufficient feature point estimation for the video frame during the feature extraction process, the homography matrix calculated based on such feature points is inaccurate, which may produce an erroneous alignment result and a poor processing effect on video stabilization. Summary of the invention

[0004] The present disclosure provides a video processing method, a device and a video stabilization method, so as to at least solve the problem in the related art that insufficient feature point estimation leads to poor processing effect of video stabilization.

[0005] According to a first aspect of an embodiment of the present disclosure, a video processing method is provided, the video processing method comprising: obtaining two video frames in a video to be processed; for each of the two video frames, executing: obtaining a first feature point corresponding to a current video frame, performing a position transformation on the first feature point according to a predetermined position transformation method to obtain a second feature point corresponding to the current video frame, combining the first feature point and the second feature point to obtain a combined feature point of the current video frame; and aligning the two video frames based on the combined feature point of each of the two video frames.

[0006] Optionally, the step of obtaining the first feature point corresponding to the current video frame includes: determining the first feature point by a feature point interpolation processing method based on the third feature point extracted from the current video frame, wherein the feature point interpolation processing method is determined according to the ratio of reducing the resolution of the two video frames, and the number of the first feature points is less than the number of the third feature points. Alternatively, reducing the resolution of the two video frames at the same ratio to obtain two reduced video frames; extracting feature points from the two reduced video frames respectively to obtain the first feature point corresponding to each of the two reduced video frames. Alternatively, for each of the two video frames, extracting a third feature point from the video frame, and determining the third feature point as the first feature point.

[0007] Optionally, the step of obtaining a second feature point corresponding to the current video frame includes: taking the center line in the width direction and the center line in the height direction of the current video frame as the axis of symmetry, performing an axisymmetric transformation on the first feature point to obtain a first symmetry point and a second symmetry point; and using the first symmetry point and the second symmetry point as the second feature point.

[0008] Optionally, the step of obtaining a second feature point corresponding to the current video frame also includes: taking the center of the current video frame as the center of symmetry, performing a center-symmetric transformation on the first feature point to obtain a third symmetric point; and using the third symmetric point as the second feature point.

[0009] Optionally, the step of combining the first feature point and the second feature point includes: combining the first feature point and the second feature point by superimposing a feature point matrix.

[0010] Optionally, the step of aligning the two video frames based on the combined feature points of each of the two video frames includes: determining a homography matrix based on the combined feature points; and aligning the two video frames based on the homography matrix.

[0011] Optionally, the step of obtaining two video frames in the video to be processed includes: for each pair of adjacent video frames in the video to be processed, obtaining the feature point distance between adjacent video frames, wherein the feature point distance is the distance between a third feature point extracted from a previous video frame in the adjacent video frames and a third feature point extracted from a subsequent video frame in the adjacent video frames; and determining the adjacent video frames whose feature point distance satisfies a predetermined condition as the two video frames.

[0012] Optionally, the feature point distance is the Euclidean distance between the third feature points.

[0013] Optionally, the step of determining adjacent video frames whose feature point distance satisfies a predetermined condition as the two video frames includes: determining adjacent video frames whose feature point distance is greater than a first threshold and less than or equal to a second threshold as the two video frames, wherein the first threshold and the second threshold are determined based on parameters related to picture motion during video shooting.

[0014] According to a second aspect of an embodiment of the present disclosure, a video stabilization method is provided, which includes obtaining a video to be processed; performing video frame alignment on video frames in the video to be processed according to the video processing method described in the present disclosure; and obtaining a stabilized video corresponding to the video to be processed based on the aligned video frames.

[0015] According to a third aspect of an embodiment of the present disclosure, a video processing device is provided, comprising: an acquisition unit, configured to acquire two video frames in a video to be processed; a combination unit, configured to perform the following operations for each of the two video frames: acquire a first feature point corresponding to a current video frame, transform the first feature point according to a predetermined position transformation method to obtain a second feature point corresponding to the current video frame, and combine the first feature point and the second feature point to obtain a combined feature point of the current video frame; and an alignment unit, configured to align the two video frames based on the combined feature point of each of the two video frames.

[0016] Optionally, the combination unit is further configured to obtain the first feature point in the following manner: based on the third feature point extracted from the current video frame, the first feature point is determined by a feature point interpolation processing method, wherein the feature point interpolation processing method is determined according to the ratio of reducing the resolution of the two video frames, and the number of the first feature points is less than the number of the third feature points. Alternatively, the resolution of the two video frames is reduced at the same ratio to obtain two reduced video frames; feature points are extracted from the two reduced video frames respectively to obtain the first feature point corresponding to each of the two reduced video frames. Alternatively, for each of the two video frames, a third feature point is extracted from the video frame, and the third feature point is determined as the first feature point.

[0017] Optionally, the combination unit is also configured to: take the center line in the width direction and the center line in the height direction of the current video frame as the axis of symmetry, perform an axisymmetric transformation on the first feature point to obtain a first symmetric point and a second symmetric point; and use the first symmetric point and the second symmetric point as the second feature point.

[0018] Optionally, the combining unit is further configured to: take the center of the current video frame as the symmetry center, perform a center-symmetric transformation on the first feature point to obtain a third symmetric point; and use the third symmetric point as the second feature point.

[0019] Optionally, the combining unit is further configured to: combine the first feature point and the second feature point by superimposing a feature point matrix.

[0020] Optionally, the alignment unit is further configured to: determine a homography matrix based on the combined feature points; and align the two video frames based on the homography matrix.

[0021] Optionally, the acquisition unit is further configured to: for each pair of adjacent video frames in the video to be processed, acquire the feature point distance between adjacent video frames, wherein the feature point distance is the distance between a third feature point extracted from a previous video frame in the adjacent video frames and a third feature point extracted from a subsequent video frame in the adjacent video frames; and determine the adjacent video frames whose feature point distance satisfies a predetermined condition as the two video frames.

[0022] Optionally, the feature point distance is the Euclidean distance between the third feature points.

[0023] Optionally, the acquisition unit is further configured to: determine adjacent video frames whose feature point distance is greater than a first threshold and less than or equal to a second threshold as the two video frames, wherein the first threshold and the second threshold are determined based on parameters related to picture motion during video shooting.

[0024] According to a fourth aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing instructions executable by the processor, wherein the processor executable instructions, when executed by the processor, cause the processor to execute the video processing method according to the present disclosure or the video stabilization method according to the present disclosure.

[0025] According to a fifth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the video processing method according to the present disclosure or the video stabilization method according to the present disclosure.

[0026] According to a sixth aspect of an embodiment of the present disclosure, a computer program product is provided, which includes computer instructions, and when the computer instructions are executed by a processor, the video processing method according to the present disclosure or the video stabilization method according to the present disclosure is implemented.

[0027] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0028] The second feature point can be obtained by positionally transforming the first feature point corresponding to the two video frames, thereby obtaining a combined feature point, and the video frames are aligned based on the combined feature point, thereby improving the video frame alignment effect and improving the video stabilization processing effect.

[0029] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0031] Figure 1 The figure is a flowchart of a video processing method according to an exemplary embodiment.

[0032] Figure 2 The present invention is a flowchart showing a step of obtaining two video frames in a video to be processed in a video processing method according to an exemplary embodiment.

[0033] Figure 3 The present invention is a flowchart showing a step of determining combined feature points in a video processing method according to an exemplary embodiment.

[0034] Figure 4 The figure is a flowchart of a step of obtaining a second feature point in a video processing method according to an exemplary embodiment.

[0035] Figure 5 The flowchart of an example of a video processing method is shown according to an exemplary embodiment.

[0036] Fig. 6A , Figure 6B , Figure 6C and Fig.6D is a schematic diagram showing a comparison of example results of video frames processed according to an existing video processing method and a video processing method according to an exemplary embodiment.

[0037] Figure 7 The figure is a flow chart of a video stabilization method according to an exemplary embodiment.

[0038] Figure 8 The invention is a block diagram of a video processing device according to an exemplary embodiment.

[0039] Fig. 9It is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0040] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0041] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0042] It should be noted that the phrase "at least one of the items" in the present disclosure includes three types of parallel situations: "any one of the items", "a combination of any number of the items", and "all of the items". For example, "including at least one of A and B" includes the following three parallel situations: (1) including A; (2) including B; (3) including A and B. Another example is "executing at least one of step 1 and step 2" which means the following three parallel situations: (1) executing step 1; (2) executing step 2; (3) executing step 1 and step 2.

[0043] As the memory of electronic devices such as mobile devices increases and sensors become more advanced, the quality of the videos captured has made a qualitative leap. However, electronic devices such as mobile phones are facing the demand for being thinner and lighter, so the size of the sensor is also limited. Therefore, how to obtain higher video quality under the environment of device hardware constraints is a challenging problem.

[0044] Higher video quality requires not only image clarity but also video stability. To this end, video stabilization algorithms are used to make up for hardware deficiencies. Video stabilization is a very complex problem, mainly due to the complexity of motion. Generally speaking, video motion can be divided into three categories, namely, the shaking of the camera, the motion of the subject, and the motion caused by automatic zoom during shooting. In existing video stabilization methods, the motion of the camera and the motion of the subject between frames can be aligned through the homography matrix. However, the processing effect of the motion caused by automatic zoom is poor.

[0045] Specifically, existing video stabilization methods mainly use homography transformation to align the previous and next frames. Assuming that a video sequence of N frames is given, every 50 frames are key frames, feature point detection is performed on the key frames, and optical flow tracking is performed on the feature points on the subsequent non-key frames. During the optical flow tracking process, if it is found that the number of feature points in a certain area is less than 1, the feature points are re-detected and then tracked again. When the previous and next video frames meet the set number of feature points, the feature points of the previous and next video frames are first estimated using a feature extraction operator (for example, the Oriented Fast and Rotated Brief (ORB)) to obtain two two-dimensional matrices, and then the homography matrix is ​​estimated based on these two-dimensional matrices, and the homography matrix is ​​applied to the previous frame to complete the alignment.

[0046] In the above process, since the estimation of the homography matrix depends on the result of the feature extraction operator, if there is no sufficient feature point estimation for the video frame during the feature extraction process, the homography matrix calculated based on such feature points is inaccurate. For example, in the existing video stabilization method, more effective feature points can be obtained for frames with clear pictures, so most of the camera movement and subject movement can be solved. However, when the video is automatically zoomed during shooting, the current frame will be blurred compared to the previous frame, so there is no sufficient feature point estimation for the current frame. At this time, the calculated homography matrix is ​​inaccurate and may produce wrong alignment results.

[0047] In view of the above problems, a video processing method, a video processing device, a video stabilization method, an electronic device, a computer-readable storage medium, and a computer program product according to exemplary embodiments of the present disclosure are provided below with reference to the accompanying drawings. It should be noted that the exemplary embodiments of the present disclosure can be applied to devices used for computing services, including but not limited to personal computers, tablet computers, smart phones, etc.

[0048] Figure 1 is a flowchart of a video processing method according to an exemplary embodiment. Figure 1 As shown, the video processing method may include the following steps:

[0049] In step S10, two video frames in the video to be processed may be obtained.

[0050] The video processing method according to the exemplary embodiment of the present disclosure can be used to process the alignment of video frames in video images that are in motion due to any reason, for example, the alignment of video frames caused by the shaking of the shooting device, the motion of the subject, and the motion caused by automatic zooming during shooting can be processed to improve the alignment effect. Therefore, in one example, in this step, any two adjacent video frames in the video to be processed can be obtained for processing.

[0051] In addition, as described above, in the existing video processing method, the feature points cannot be fully estimated in the video frame alignment process caused by the motion caused by the auto zoom, resulting in poor alignment results. In this case, it can be firstly clarified what kind of motion has occurred in the previous and next video frames, and when there is auto zoom motion in the previous and next video frames, the video processing method according to the exemplary embodiment of the present disclosure can be applied.

[0052] Thus, in another example, Figure 2 As shown, in this step, the step of obtaining two video frames in the video to be processed may include:

[0053] In step S11, for each pair of adjacent video frames in the video to be processed, the feature point distance between the adjacent video frames may be obtained.

[0054] In this step, the feature point distance may be the distance between the feature point extracted from the previous video frame in the adjacent video frames and the feature point extracted from the next video frame in the adjacent video frames. Here, the feature points extracted from the adjacent video frames may be implemented by any feature point extraction method such as the existing Scale-invariant feature transform (SIFT) algorithm. For ease of description, the feature points extracted from the adjacent video frames described in this step are referred to as "third feature points" below.

[0055] The feature point distance may refer to the Euclidean distance between the feature points. For example, assuming that the feature point matrix of the previous video frame is Fn and the feature point matrix of the next video frame is Fr, the Euclidean distance D between the two feature points can be expressed by the following formula (1):

[0056]

[0057] Although the feature point distance is described above as the Euclidean distance, it is not limited thereto, and the feature point distance may also be determined by other distance calculation methods to characterize the position difference between the feature points of the previous and next video frames.

[0058] In step S12, adjacent video frames whose feature point distances satisfy a predetermined condition may be determined as two video frames to be processed.

[0059] In this step, the video frames that need to be processed according to the method of the exemplary embodiment of the present disclosure may be determined by constraining the feature point distances between adjacent video frames.

[0060] As an example, adjacent video frames with the feature point distance greater than the first threshold and less than or equal to the second threshold can be determined as two video frames to be processed. Here, the first threshold and the second threshold can be determined by parameters related to the movement of the picture during video shooting. However, the present disclosure is not limited thereto, and the first threshold and the second threshold can also be set according to actual video processing needs.

[0061] Specifically, when it is less than the first threshold, it can be considered that the blur degree between the front and rear video frames is small, and video stabilization processing may not be required; when it is greater than the second threshold, it can be considered that there is movement of the shooting device and movement of the object being photographed. In this case, existing video stabilization algorithms such as homography transformation can be used for processing; when it is greater than the first threshold and less than the second threshold, for example, it can be considered that a zoom movement has occurred.

[0062] For example, adjacent video frames with the feature point distance a satisfying 0.00015 < a ≤ 0.001 can be determined as two video frames to be processed. When the feature point distance a satisfies the above conditions, it can be considered that the device for shooting the video undergoes automatic zoom during shooting, resulting in video frame blur. In this regard, the video processing method of the exemplary embodiments of the present disclosure can be applied for processing.

[0063] Of course, the predetermined conditions are not limited thereto, and two video frames to be processed can also be determined by setting a threshold for the feature point distance. For example, adjacent video frames with the feature point distance greater than the third threshold can be determined as the video frames to be processed. Here, the third threshold can be determined according to parameters related to the movement of the picture during video shooting or set according to actual processing needs.

[0064] In step S20, for each of the two video frames, combined feature points can be determined.

[0065] Specifically, as Figure 3 shown, step S20 may include the following operations:

[0066] In step S21, a first feature point corresponding to the current video frame can be obtained.

[0067] Here, the current video frame can be each of the two video frames, and the first feature point can be a feature point obtained based on the current video frame, which can be obtained by extraction or based on existing feature points.

[0068] Specifically, in the step of obtaining the first feature point corresponding to the current video frame, the first feature point can be obtained by one of the following methods:

[0069] In one example, based on the feature points extracted from the current video frame, the first feature point can be determined through a feature point interpolation processing method.

[0070] In this example, the resolution of the two video frames can be reduced for fast video processing. For example, the resolution of the two video frames can be scaled to 1 / 2 times of the original, but the present disclosure is not limited to this. The resolution of the two video frames can also be scaled to 1 / 4 times, 1 / 8 times, etc. of the original.

[0071] When the resolution of the video frame is reduced, the number of feature points can be reduced accordingly. For example, the feature point extracted from the current video frame can be, for example, the third feature point extracted in the above step, and the third feature point can be subjected to feature point interpolation processing such as bicubic linear interpolation to obtain the first feature point. In this example, the resolution of the reduced video frame is smaller than the resolution of the original video frame, and the number of the first feature points is smaller than the number of the third feature points.

[0072] Here, the feature point interpolation processing method can be determined according to the ratio of reducing the resolution of the two video frames. According to different ratios, an appropriate feature point interpolation processing method can be selected. For example, when the reduction ratio is relatively large, the number of the first feature points can be reduced relatively more than the third feature points; when the reduction ratio is relatively small, the number of the first feature points can be reduced relatively less than the third feature points. In this example, the determination of the first feature point depends on the determination of the third feature point described above.

[0073] In another example, the resolutions of two video frames may be reduced at the same ratio to obtain two reduced video frames; feature points may be extracted from the two reduced video frames respectively to obtain first feature points corresponding to each of the two reduced video frames.

[0074] Specifically, similar to the above example, in this example, the resolution of the two video frames to be processed can be reduced, and here, the first feature point can be extracted from the reduced video frame by any feature point extraction method such as the existing SIFT algorithm. In this example, the determination of the first feature point may not depend on the determination of the third feature point described above, and the two may be extracted in the same or different manners.

[0075] In yet another example, the third feature point in the two video frames may be determined as the first feature point.

[0076] In this example, the two video frames to be processed do not need to be subjected to resolution reduction processing. Thus, the third feature point extracted from each of the two video frames as described above can be used as the first feature point in this step to perform the following processing.

[0077] In step S22, the first feature point may be transformed in a predetermined position transformation manner to obtain a second feature point corresponding to the current video frame.

[0078] In this step, the predetermined position transformation method may include axisymmetric transformation, center symmetric transformation, translation transformation, etc. One or more of these transformations may be performed on the known first feature point to obtain a new second feature point, and the second feature point may be used as a supplementary feature point to participate in the subsequent video frame alignment operation. Of course, the predetermined position transformation method is not limited to this, and may also include diagonal symmetry, geometric magnification, and other transformation methods.

[0079] Specifically, if Figure 4 As shown, the step of obtaining the second feature point corresponding to the current video frame may include: step S221, taking the center line in the width direction and the center line in the height direction of the current video frame as the axis of symmetry, performing an axisymmetric transformation on the first feature point to obtain a first symmetric point and a second symmetric point; step S222, taking the first symmetric point and the second symmetric point as the second feature point.

[0080] Here, when the video frame to be processed has not been reduced, the current video frame refers to the video frame to be processed; when the video frame to be processed has been reduced, the current video frame refers to the reduced video frame.

[0081] For example, the first feature point matrix corresponding to the two video frames to be processed may be cp1 and lp1, and the coordinates of cp1 and lp1 may be reassigned to reconstruct the second symmetric point matrix cp2 and cp3 based on the first feature point matrix cp1 and to reconstruct the third symmetric point matrix lp2 and lp3 based on the first feature point matrix lp1. In the operation, the second symmetric point matrix and the third symmetric point matrix may be assigned as follows:

[0082] cp2=cp1, lp2=lp1, cp2.x=width-cp1.x, lp2.x=width-lp1.x;

[0083] cp3=cp1, lp3=lp1, cp3.y=height-cp1.y, lp3.y=height-lp1.y,

[0084] Wherein, width and height are the width and height of the video frame respectively. When the video frame is reduced, width and height are the width and height of the reduced video frame respectively, for example, it can be 1 / 2 of the video frame size. Here, cp2 and lp2 are equivalent to the first symmetry point obtained by performing an axisymmetric transformation on the first feature point with the center line in the width direction of the current video frame as the axis of symmetry; cp3 and lp3 are equivalent to the second symmetry point obtained by performing an axisymmetric transformation on the first feature point with the center line in the height direction of the current video frame as the axis of symmetry. cp2, cp3, lp2 and lp3 can be collectively used as the second feature point.

[0085] In addition, the step of obtaining the second feature point corresponding to the current video frame may also include: taking the center of the current video frame as the symmetry center, performing a center-symmetric transformation on the first feature point to obtain a third symmetric point; and using the third symmetric point as the second feature point.

[0086] Still taking the above example, the first feature point matrix corresponding to the two video frames to be processed may be cp1 and lp1, and the coordinates of cp1 and lp1 may be reassigned to reconstruct the third symmetric point matrix cp4 based on the first feature point matrix cp1 and reconstruct the third symmetric point matrix lp4 based on the first feature point matrix lp1. In the operation, the assignment method of the third symmetric point matrix may be as follows:

[0087] cp4=cp2, lp4=lp2, cp4.y=height-cp2.y, lp4.y=height-lp2.y.

[0088] Here, cp4 and lp4 are equivalent to the third symmetric points obtained by performing a central symmetric transformation on the first feature point with the center of the current video frame as the symmetry center. The third symmetric points cp4 and lp4 can be used as the second feature points together with the first symmetric points cp2, lp2 and the second symmetric points cp3 and lp3.

[0089] Although an exemplary embodiment of obtaining the second feature point based on the first feature point based on axial symmetry and central symmetry is described above, it is not limited to this. For example, the second feature point can also be obtained by translating the first feature point, and according to the calculation speed requirement, part or all of the first feature points can be selectively transformed to obtain the second feature point.

[0090] In the video processing method of the exemplary embodiment of the present disclosure, by reconstructing / generating the second feature point based on the determined first feature point, the feature point can be quickly supplemented in the case of insufficient feature point extraction. For example, if the area where the first feature point is located is the upper left area of ​​the frame and there is no first feature point in the upper right area of ​​the frame, after the axisymmetric transformation in the vertical direction, the second feature point can be obtained in the upper right area of ​​the frame, thus supplementing the problem of blank feature points in the upper right area.

[0091] In addition, video processing often involves a large amount of video frame processing, so it is necessary to pay attention to the processing speed of the algorithm. According to the video processing method of the exemplary embodiment of the present invention, the feature points are supplemented by transforming the predetermined position of the first feature point. While achieving the feature point supplementation, the processing speed can also be ensured to be faster.

[0092] In step S23, the first feature point and the second feature point may be combined to obtain a combined feature point of the current video frame.

[0093] In this step, the first feature point and the second feature point may be combined to obtain a combined feature point by superimposing a feature point matrix.

[0094] Specifically, still taking the above example, the newly constructed feature point matrices cp1, cp2, cp3, cp4 and lp1, lp2, lp3, lp4 can be superimposed together to obtain vector matrices cur_sc and last_sc, so as to use the matrices cur_sc and last_sc to calculate the homography matrix in which the previous and next video frames are stationary or zoomed. Of course, the method of combining the first feature point and the second feature point is not limited to this. For example, the first feature point image and the second feature point image can also be superimposed by a method such as image superposition to obtain a combined feature point image.

[0095] In step S30, the two video frames are aligned based on the combined feature points of each of the two video frames.

[0096] In this step, after the combined feature points are obtained, a homography matrix can be determined based on the combined feature points; based on the homography matrix, the two video frames are aligned. Here, for example, the homography matrix can be calculated using a random sampling consensus algorithm (RANSAC), however, it is not limited thereto, and other homography matrix calculation methods such as a least squares method, an LM iterative algorithm, etc. can also be used.

[0097] Although SIFT is used as the feature extraction operator in the above description, in other exemplary embodiments, the ORB operator may also be used for feature extraction, and the speed will be further improved.

[0098] An exemplary embodiment of a video processing method according to an exemplary embodiment of the present disclosure is described above, which obtains a second feature point by performing a position transformation on the first feature point corresponding to two video frames, thereby obtaining a combined feature point, and performs video frame alignment based on the combined feature point. This can solve the problem in the related art that there is insufficient feature point estimation, resulting in poor processing effect of video stabilization, improve the video frame alignment effect, and enhance the processing effect of video stabilization.

[0099] In addition, according to this method, it is also possible to define preset conditions for judging different movements, and then adopt a strategy of reconstructing the matrix for the static zoom scene, and use the newly constructed matrix to estimate the homography matrix of the zoom point. Thus, motion-adaptive video stabilization processing can be performed according to the movement between video frames, and three common motion scenes can be automatically processed without manual switching and selection.

[0100] The video processing method according to the exemplary embodiment of the present disclosure takes into account three inter-frame motion states, namely, the motion of the camera, the motion of the object, and the motion during auto zoom, wherein the first two motions are dynamic changes, while the latter is a static change. The method can be applied to these three motions at the same time, and can adaptively match the scenes of the three motions without manual switching. Figure 5 An example of applying the video processing method is given.

[0101] Figure 5 A flowchart of an example of a video processing method according to an exemplary embodiment is shown. Figure 5 As shown, in step S1, feature extraction can be performed on image 1 and image 2 as two video frames to be processed; in step S2, the feature point distance can be calculated based on the result of feature extraction to determine which type of motion the previous and next video frames belong to; when the motion belongs to a focal point motion, in step S3, the second feature point matrix can be reconstructed, so that in step S4, the zoom homography matrix can be calculated using the focal point, and the homography transformation can be performed in step S5 to align the video frames.

[0102] like Figure 5 As shown, the video processing method according to the exemplary embodiment of the present disclosure is a video stabilization algorithm that is adaptive to both static and dynamic scenes, which is not only applicable to scenes where inter-frame motion is generated by camera motion and object motion, but can also stabilize inter-frame changes during automatic zoom.

[0103] Refer to the following Fig. 6A , Figure 6B , Figure 6C and Fig.6DThe comparison of the example results of video frames processed by the existing video processing method and the video processing method according to an exemplary embodiment is shown. Here, the subjective effect of human vision is used for evaluation, mainly observing the degree of jitter after the video passes through the algorithm, whether there is a jelly feeling and unnatural distortion.

[0104] like Fig. 6A and Figure 6B As shown, Fig. 6A For traditional video processing methods, Figure 6B This is the video processing method disclosed in the present invention. Since most of the video is a forest and the leaves are blurred, it can be roughly regarded as a cut-focus scene. Fig. 6A and Figure 6B The comparison results show that Fig. 6A The trees in the picture appear unnaturally tilted and have a strong sense of distortion. This is because the estimated homography matrix has errors in the blurred scene, resulting in unnatural stretching. In contrast, the video processing method disclosed in the present invention has better control over the distortion of the picture. Figure 6B There is basically no strong pulling feeling.

[0105] like Figure 6C and Fig.6D As shown, Figure 6C For traditional video processing methods, Fig.6D The video processing method disclosed in the present invention. Figure 6C and Fig.6D The comparison results show that Figure 6C The font of "Yellow Braised Chicken" in the picture is obviously stretched forward and backward. Fig.6D The font of "Yellow Braised Chicken" in the video is more natural. For this example, from the continuous playback of the video, Figure 6C The results showed an obvious jelly effect. Fig.6D The results are more stable.

[0106] According to an exemplary embodiment of the present disclosure, a video stabilization method is also provided. Figure 7 As shown, the video stabilization method may include the following steps:

[0107] In step S100, a video to be processed may be obtained. Here, the video to be processed may be a video in any form, for example, a video frame including blur or a video frame with picture motion.

[0108] In step S200, according to the video processing method described in the exemplary embodiments of the present disclosure, video frames in the video to be processed may be aligned.

[0109] In this step, the video frames in the video to be processed can be aligned by the video processing method according to the present disclosure described above, where the aligned video frames can be a part of all the video frames in the video to be processed or all the video frames. The specific video frame alignment method has been described in detail in the video processing method according to the exemplary embodiment of the present disclosure above, and will not be repeated here.

[0110] In step S300, a stable video corresponding to the video to be processed may be obtained based on the aligned video frames.

[0111] In this step, a new video can be formed based on the aligned video frames. Since the video frames have been aligned, the newly formed video has a relatively stable picture compared to the initially acquired video to be processed, thereby achieving stable processing of the video.

[0112] According to the video stabilization processing method of the exemplary embodiment of the present disclosure, since the video frame alignment method according to the present disclosure is adopted in the process of video frame alignment, the video frames in the video can be better aligned, so that the video stabilization processing effect is better and a video with smoother picture quality is obtained.

[0113] Figure 8 is a block diagram of a video encoding device according to an exemplary embodiment. Figure 8 The video encoding device includes an acquisition unit 100, a combination unit 200 and an alignment unit 300.

[0114] The acquisition unit 100 is configured to acquire two video frames in the video to be processed.

[0115] The combination unit 200 is configured to perform the following operations for each of the two video frames: obtain a first feature point corresponding to the current video frame, transform the first feature point according to a predetermined position transformation method to obtain a second feature point corresponding to the current video frame, and combine the first feature point and the second feature point to obtain a combined feature point of the current video frame.

[0116] The alignment unit 300 is configured to align the two video frames based on the combined feature points of each of the two video frames.

[0117] As an example, the acquisition unit 100 is also configured to: for each pair of adjacent video frames in the video to be processed, obtain the feature point distance between the adjacent video frames, wherein the feature point distance is the distance between the third feature point extracted from the previous video frame in the adjacent video frames and the third feature point extracted from the next video frame in the adjacent video frames; and determine the adjacent video frames whose feature point distance satisfies a predetermined condition as two video frames.

[0118] As an example, the feature point distance is the Euclidean distance between the third feature points.

[0119] As an example, adjacent video frames whose feature point distance is greater than a first threshold and less than or equal to a second threshold are determined as two video frames, wherein the first threshold and the second threshold are determined based on parameters related to picture motion during video shooting.

[0120] As an example, the combination unit 200 is further configured to obtain the first feature point in the following manner: based on the third feature point extracted from the current video frame, the first feature point is determined by a feature point interpolation processing method, wherein the feature point interpolation processing method is determined according to the ratio of reducing the resolution of the two video frames, and the number of the first feature points is less than the number of the third feature points. Alternatively, the resolution of the two video frames is reduced at the same ratio to obtain two reduced video frames; the first feature point is extracted based on the two reduced video frames respectively. Alternatively, the third feature point in the two video frames is determined as the first feature point.

[0121] As an example, the combination unit 200 is also configured to: use the center line in the width direction and the center line in the height direction of the current video frame as the axis of symmetry, perform an axisymmetric transformation on the first feature point to obtain a first symmetric point and a second symmetric point; and use the first symmetric point and the second symmetric point as the second feature point.

[0122] As an example, the combining unit 200 is further configured to: take the center of the current video frame as the symmetry center, perform a center-symmetric transformation on the first feature point to obtain a third symmetric point; and use the third symmetric point as the second feature point.

[0123] As an example, the combining unit 200 is further configured to combine the first feature point and the second feature point by superimposing a feature point matrix.

[0124] As an example, the alignment unit 300 is further configured to: determine a homography matrix based on the combined feature points; and align two video frames based on the homography matrix.

[0125] Regarding the device in the above embodiment, the specific manner in which each unit performs the operation has been described in detail in the embodiment of the method, and will not be elaborated here.

[0126] Fig. 9 FIG. 1 is a block diagram of an electronic device according to an exemplary embodiment. Fig. 9 As shown, the electronic device 10 includes a processor 101 and a memory 102 for storing processor executable instructions. Here, when the processor executable instructions are executed by the processor, the processor is prompted to execute the video processing method as described in the above exemplary embodiment.

[0127] As an example, the electronic device 10 is not necessarily a single device, but may be any collection of devices or circuits that can execute the above instructions (or instruction sets) individually or in combination. The electronic device 10 may also be part of an integrated control system or system manager, or may be configured as an electronic device interconnected with a local or remote (e.g., via wireless transmission) interface.

[0128] In the electronic device 10, the processor 101 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller or a microprocessor. As an example and not a limitation, the processor 101 may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0129] The processor 101 may execute instructions or codes stored in the memory 102, wherein the memory 102 may also store data. Instructions and data may also be sent and received over a network via a network interface device, wherein the network interface device may employ any known transmission protocol.

[0130] The memory 102 may be integrated with the processor 101, for example, by placing RAM or flash memory within an integrated circuit microprocessor or the like. In addition, the memory 102 may include a separate device, such as an external disk drive, a storage array, or any other storage device that can be used by a database system. The memory 102 and the processor 101 may be operatively coupled, or may communicate with each other, such as through an I / O port, a network connection, etc., so that the processor 101 can read files stored in the memory 102.

[0131] In addition, the electronic device 10 may also include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.) All components of the electronic device 10 may be connected to each other via a bus and / or a network.

[0132] In an exemplary embodiment, a computer-readable storage medium may also be provided. When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device can perform the video processing method as described in the above exemplary embodiment. The computer-readable storage medium may be, for example, a memory including instructions. Optionally, the computer-readable storage medium may be: a read-only memory (ROM), a random access memory (RAM), a random access programmable read-only memory (PROM), an electrically erasable programmable read-only memory (EEPROM), a dynamic random access memory (DRAM), a static random access memory (SRAM), a flash memory, a non-volatile memory, a CD-ROM, a CD-R, a CD+R, a CD-RW, a CD+RW, a DVD-ROM, a DVD-R, a DVD+R, a DVD-RW, a DVD+RW, a DVD-RAM, a BD-ROM, a BD-R, a BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as, multimedia card, secure digital (SD) card or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, any other device is configured to store computer programs and any associated data, data files and data structures in a non-transitory manner and provide the computer programs and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the above-mentioned computer-readable storage medium can be run in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. In addition, in one example, the computer program and any associated data, data files and data structures are distributed on a networked computer system, so that the computer program and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers.

[0133] In an exemplary embodiment, a computer program product may also be provided. The computer program product includes computer instructions. When the computer instructions are executed by a processor, the video processing method as described in the exemplary embodiment above is implemented.

[0134] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0135] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A video processing method, characterized in that: The video processing method comprises: Acquire two video frames in the video to be processed, wherein the two video frames are adjacent video frames; For each of the two video frames, perform the following operations: Get the first feature point corresponding to the current video frame, According to a predetermined position transformation method, the first feature point is transformed to obtain a second feature point corresponding to the current video frame, wherein the predetermined position transformation method can transform the position of the first feature point to a blank area of ​​the feature point in the current video frame, Combining the first feature point and the second feature point to obtain a combined feature point of the current video frame; The two video frames are aligned based on the combined feature points of each of the two video frames.

2. The video processing method according to claim 1, characterized in that: The step of obtaining a first feature point corresponding to the current video frame includes: Based on the third feature points extracted from the current video frame, the first feature points are determined by a feature point interpolation processing method, wherein the feature point interpolation processing method is determined according to a ratio of reducing the resolution of the two video frames, and the number of the first feature points is less than the number of the third feature points, or, Reducing the resolutions of the two video frames at the same ratio to obtain two reduced video frames; Extracting feature points from the two reduced video frames respectively to obtain the first feature points corresponding to each of the two reduced video frames, or, For each of the two video frames, a third feature point is extracted from the video frame, and the third feature point is determined as the first feature point.

3. The video processing method according to claim 1, characterized in that: The step of obtaining a second feature point corresponding to the current video frame includes: Taking the center line in the width direction and the center line in the height direction of the current video frame as symmetry axes, performing an axisymmetric transformation on the first feature point to obtain a first symmetric point and a second symmetric point; The first symmetrical point and the second symmetrical point are used as the second feature points.

4. The video processing method according to claim 3, characterized in that: The step of obtaining a second feature point corresponding to the current video frame further includes: Taking the center of the current video frame as the symmetry center, performing a center-symmetric transformation on the first feature point to obtain a third symmetric point; The third symmetrical point is used as the second feature point.

5. The video processing method according to claim 1, characterized in that: The step of combining the first feature point and the second feature point comprises: The first feature point and the second feature point are combined by superimposing a feature point matrix.

6. The video processing method according to claim 1, characterized in that: Based on the combined feature points of each of the two video frames, the step of aligning the two video frames comprises: Based on the combined feature points, determining a homography matrix; Based on the homography matrix, the two video frames are aligned.

7. The video processing method according to claim 1, characterized in that: The steps of obtaining two video frames in the video to be processed include: For each pair of adjacent video frames in the video to be processed, obtaining a feature point distance between the adjacent video frames, wherein the feature point distance is a distance between a third feature point extracted from a previous video frame in the adjacent video frames and a third feature point extracted from a subsequent video frame in the adjacent video frames; Adjacent video frames whose feature point distances satisfy a predetermined condition are determined as the two video frames.

8. The video processing method according to claim 7, characterized in that: The feature point distance is the Euclidean distance between the third feature points.

9. The video processing method according to claim 8, characterized in that: The step of determining adjacent video frames whose feature point distances satisfy a predetermined condition as the two video frames comprises: Adjacent video frames whose feature point distance is greater than a first threshold and less than or equal to a second threshold are determined as the two video frames, wherein the first threshold and the second threshold are determined based on parameters related to picture motion during video shooting.

10. A video stabilization method, characterized in that: The video stabilization method comprises: Get the video to be processed; According to the video processing method according to any one of claims 1 to 9, video frame alignment is performed on the video frames in the video to be processed; Based on the aligned video frames, a stable video corresponding to the video to be processed is obtained.

11. A video processing device, characterized in that: The video processing device comprises: An acquisition unit is configured to acquire two video frames in the video to be processed, wherein the two video frames are adjacent video frames; The combining unit is configured to perform the following operations for each of the two video frames: obtaining a first feature point corresponding to the current video frame, performing a position transformation on the first feature point according to a predetermined position transformation method to obtain a second feature point corresponding to the current video frame, and combining the first feature point and the second feature point to obtain a combined feature point of the current video frame, wherein the predetermined position transformation method can transform the position of the first feature point to a blank area of ​​feature points in the current video frame; The alignment unit is configured to align the two video frames based on the combined feature points of each of the two video frames.

12. The video processing device according to claim 11, characterized in that: The combining unit is further configured to obtain the first feature point in the following manner: Based on the third feature points extracted from the current video frame, the first feature points are determined by a feature point interpolation processing method, wherein the feature point interpolation processing method is determined according to a ratio of reducing the resolution of the two video frames, and the number of the first feature points is less than the number of the third feature points, or, Reducing the resolutions of the two video frames at the same ratio to obtain two reduced video frames; Extracting feature points from the two reduced video frames respectively to obtain the first feature points corresponding to each of the two reduced video frames, or, For each of the two video frames, a third feature point is extracted from the video frame, and the third feature point is determined as the first feature point.

13. The video processing device according to claim 11, characterized in that: The combination unit is also configured to: use the center line in the width direction and the center line in the height direction of the current video frame as the axis of symmetry, perform an axisymmetric transformation on the first feature point to obtain a first symmetric point and a second symmetric point; and use the first symmetric point and the second symmetric point as the second feature point.

14. The video processing device according to claim 13, characterized in that: The combination unit is further configured to: take the center of the current video frame as the symmetry center, perform a center-symmetric transformation on the first feature point to obtain a third symmetric point; and use the third symmetric point as the second feature point.

15. The video processing device according to claim 11, characterized in that: The combining unit is further configured to combine the first feature point and the second feature point by superimposing a feature point matrix.

16. The video processing device according to claim 11, characterized in that: The alignment unit is further configured to: determine a homography matrix based on the combined feature points; and align the two video frames based on the homography matrix.

17. The video processing device according to claim 11, characterized in that: The acquisition unit is also configured to: for each pair of adjacent video frames in the video to be processed, acquire the feature point distance between the adjacent video frames, wherein the feature point distance is the distance between the third feature point extracted from the previous video frame of the adjacent video frames and the third feature point extracted from the next video frame of the adjacent video frames; and determine the adjacent video frames whose feature point distance satisfies a predetermined condition as the two video frames.

18. The video processing device according to claim 17, characterized in that: The feature point distance is the Euclidean distance between the third feature points.

19. The video processing device according to claim 18, characterized in that: The acquisition unit is further configured to determine adjacent video frames whose feature point distance is greater than a first threshold and less than or equal to a second threshold as the two video frames, wherein the first threshold and the second threshold are determined based on parameters related to picture motion during video shooting.

20. An electronic device, characterized in that: The electronic device comprises: processor; a memory for storing instructions executable by the processor, When the processor executable instructions are executed by the processor, the processor is prompted to execute the video processing method according to any one of claims 1 to 9 or the video stabilization method according to claim 10.

21. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the video processing method according to any one of claims 1 to 9 or the video stabilization method according to claim 10.

22. A computer program product comprising computer instructions, characterized in that When the computer instructions are executed by a processor, the video processing method according to any one of claims 1 to 9 or the video stabilization method according to claim 10 is implemented.

Images