Image stabilization method and device for single-frame image of video

Abstract

The embodiment of the invention provides an image stabilization method and device for a single-frame image of a video, and relates to the technical field of data processing, and the method specifically comprises the steps: decoding a to-be-processed video, and obtaining a plurality of continuous single-frame images in real time; generating a target image sequence based on a preset threshold number of single-frame images; for each two adjacent frames of single-frame images in the target image sequence, respectively extracting corresponding feature points, so as to obtain a first homography matrix corresponding to the two adjacent frames of single-frame images based on the feature points; generating a first homography matrix sequence based on the plurality of first homography matrixes, and performing Gaussian filtering processing on the first homography matrix sequence to obtain a target homography matrix; and based on the target homography matrix, performing transmission transformation on a middle single-frame image at the middle frame position of the target image sequence, and generating a target image stabilization image corresponding to the middle single-frame image. According to the method and the device, the quality of the generated stabilized image can be improved while the efficiency of obtaining the stabilized image is improved.

Description

Technical Field

[0001] The present disclosure relates to the technical field of data processing, and in particular to a method and device for stabilizing a single-frame image of a video. Background Art

[0002] With the increasing popularity of social media and short videos in modern society, videos are playing an increasingly important role in our daily lives. However, because daily videos are usually shot with amateur devices such as mobile phones and drones, which do not have image stabilization functions, the shooting effect is not good. If professional hardware equipment is used for image stabilization, it has disadvantages such as inconvenience in carrying and high cost. Video stabilization algorithms have become a must-have solution to improve video jitter effects.

[0003] Existing video stabilization algorithms are mainly divided into two categories: one is the stabilization algorithm based on 2D image transformation, and the other is the stabilization algorithm based on 3D scene reconstruction.

[0004] The image stabilization algorithm based on 2D image transformation is a technology that obtains a stable image by performing motion estimation between consecutive video images and then performing motion filtering and motion compensation on each frame of the image.

[0005] The image stabilization algorithm based on 3D scene reconstruction uses multiple frames of images to construct a 3D point cloud, calculates depth information through data analysis, and uses different compensation methods according to the differences in movement patterns of different areas in the image to solve the problem of parallax.

[0006] The above-mentioned stabilization algorithm based on 2D image transformation uses motion estimation to predict the future state to smooth multiple frames of images. However, it is difficult to accurately predict the future state for some scenes with fast turns or random motion states, so the effect is not good. The stabilization algorithm based on 3D scene reconstruction has a large amount of calculation, which is difficult to achieve the requirements of real-time processing and has slightly poor robustness.

[0007] Therefore, how to improve the efficiency of acquiring stabilized images while improving the quality of generated stabilized images. Summary of the invention

[0008] In view of this, an embodiment of the present application provides a method and device for stabilizing a single frame of a video, which can improve the efficiency of video stabilization.

[0009] In a first aspect, an embodiment of the present application provides a method for stabilizing a single-frame image of a video, comprising:

[0010] Decode the video to be processed and obtain multiple continuous single-frame images in real time;

[0011] When the number of the single-frame images reaches a preset threshold, generating a target image sequence based on the preset threshold number of single-frame images;

[0012] For each of the single-frame images of two adjacent frames in the target image sequence, corresponding feature points are extracted respectively, so as to obtain a first homography matrix corresponding to the single-frame images of the two adjacent frames based on the feature points;

[0013] Generate a first homography matrix sequence based on the multiple first homography matrices, and perform Gaussian filtering on the first homography matrix sequence to obtain a target homography matrix;

[0014] Based on the target homography matrix, a transmission transformation is performed on the intermediate single-frame image at the intermediate frame position of the target image sequence to generate a target stabilized image corresponding to the intermediate single-frame image.

[0015] As an optional implementation of the embodiment of the present application, when the number of the single-frame images reaches a preset threshold, generating a target image sequence based on the preset threshold single-frame images includes:

[0016] When the number of the single-frame images reaches a preset threshold, converting the image formats of the single-frame images of the preset threshold into grayscale image formats to generate an initial image sequence;

[0017] Each of the single-frame images in the initial image sequence is reduced based on a preset reduction coefficient to generate a target image sequence.

[0018] As an optional implementation of the embodiment of the present application, for each of the single-frame images of two adjacent frames in the target image sequence, corresponding feature points are extracted respectively to obtain a first homography matrix corresponding to the single-frame images of the two adjacent frames based on the feature points, including:

[0019] For each of the single-frame images of two adjacent frames in the target image sequence, corresponding feature points are extracted based on an ORB feature point extraction algorithm;

[0020] Feature point matching is performed on the single-frame images of each two adjacent frames to calculate a first homography matrix corresponding to the single-frame images of the two adjacent frames.

[0021] As an optional implementation of the embodiment of the present application, the performing feature point matching on the single-frame images of each two adjacent frames to calculate a first homography matrix corresponding to the single-frame images of the two adjacent frames includes:

[0022] Performing feature point matching on the single-frame images of each two adjacent frames, and performing iterative calculations using a RANSAC algorithm to obtain an initial first homography matrix corresponding to the single-frame images of the two adjacent frames;

[0023] Based on a preset magnification factor, the initial first homography matrix is ​​magnified to obtain the first homography matrix.

[0024] As an optional implementation of the embodiment of the present application, generating a first homography matrix sequence based on multiple first homography matrices, and performing Gaussian filtering on the first homography matrix sequence to obtain a target homography matrix includes:

[0025] generating a first homography matrix sequence based on a plurality of the first homography matrices;

[0026] Based on preset Gaussian coefficients, Gaussian filtering is performed on the first homography matrix sequence to obtain a target homography matrix.

[0027] As an optional implementation of the embodiment of the present application, the step of performing a transmission transformation on an intermediate single-frame image at an intermediate frame position of the target image sequence based on the target homography matrix to generate a target stabilized image corresponding to the intermediate single-frame image includes:

[0028] Converting the image format of the intermediate single-frame image at the intermediate frame position of the target image sequence into a YUV image format to generate a target intermediate single-frame image;

[0029] A transmission transformation is performed on the target intermediate single-frame image based on the target homography matrix to generate a target stabilized image corresponding to the target intermediate single-frame image.

[0030] As an optional implementation of the embodiment of the present application, the method further includes:

[0031] Repeat the steps of decoding the video to be processed, acquiring multiple continuous single-frame images in real time, generating a first homography matrix sequence based on multiple first homography matrices, and performing Gaussian filtering on the first homography matrix sequence to acquire a target homography matrix, until the target image stabilization frame corresponding to each frame in the video to be processed is acquired;

[0032] Encode the stabilized frame corresponding to each frame in the video to be processed to obtain a target stabilized video corresponding to the video to be processed.

[0033] In a second aspect, an embodiment of the present application provides an image stabilization device for a single frame image of a video, comprising:

[0034] A decoding unit, used for decoding the video to be processed and obtaining multiple continuous single-frame images in real time;

[0035] A generating unit, configured to generate a target image sequence based on the preset threshold number of single-frame images when the number of the single-frame images reaches a preset threshold;

[0036] An extraction unit, configured to extract corresponding feature points respectively for each of the single-frame images of two adjacent frames in the target image sequence, so as to obtain a first homography matrix corresponding to the single-frame images of the two adjacent frames based on the feature points;

[0037] A processing unit, configured to generate a first homography matrix sequence based on the plurality of first homography matrices, and perform Gaussian filtering on the first homography matrix sequence to obtain a target homography matrix;

[0038] A transformation unit is used to perform a transmission transformation on an intermediate single-frame image at an intermediate frame position of the target image sequence based on the target homography matrix to generate a target stabilized image corresponding to the intermediate single-frame image.

[0039] As an optional implementation of the embodiment of the present application, the generation unit is specifically used to convert the image format of the single-frame images of the preset threshold value into a grayscale image format to generate an initial image sequence when the number of the single-frame images reaches a preset threshold value; and to reduce the size of each of the single-frame images in the initial image sequence based on a preset reduction coefficient to generate a target image sequence.

[0040] As an optional implementation of the embodiment of the present application, the extraction unit is specifically used to extract corresponding feature points for the single-frame images of every two adjacent frames in the target image sequence based on the ORB feature point extraction algorithm; and perform feature point matching on the single-frame images of every two adjacent frames to calculate the first homography matrix corresponding to the single-frame images of the two adjacent frames.

[0041] As an optional implementation of the embodiment of the present application, the extraction unit is specifically used to perform feature point matching on the single-frame images of each two adjacent frames, and to perform iterative calculations through the RANSAC algorithm to obtain an initial first homography matrix corresponding to the single-frame images of the two adjacent frames; based on a preset magnification factor, the initial first homography matrix is ​​amplified to obtain the first homography matrix.

[0042] As an optional implementation of the embodiment of the present application, the extraction unit is specifically used to generate a first homography matrix sequence based on multiple first homography matrices; based on preset Gaussian coefficients, perform Gaussian filtering on the first homography matrix sequence to obtain a target homography matrix.

[0043] As an optional implementation of the embodiment of the present application, the extraction unit is specifically used to convert the image format of the intermediate single frame image at the intermediate frame position of the target image sequence into a YUV image format to generate a target intermediate single frame image; perform a transmission transformation on the target intermediate single frame image based on the target homography matrix to generate a target stabilized image corresponding to the target intermediate single frame image.

[0044] As an optional implementation of an embodiment of the present application, the stabilized image acquisition for images in the video also includes an encoding unit, which is specifically used to repeat the decoding of the video to be processed, acquire multiple continuous single-frame images in real time to generate a first homography matrix sequence based on multiple first homography matrices, and perform Gaussian filtering on the first homography matrix sequence to obtain the target homography matrix step until the target stabilized frame corresponding to each frame in the video to be processed is obtained; encode the stabilized frame corresponding to each frame in the video to be processed to obtain the target stabilized video corresponding to the video to be processed.

[0045] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a memory and a processor, wherein the memory is used to store a computer program; and the processor is used to enable the electronic device to implement the image stabilization method for a single frame image of a video as described in any of the above embodiments when executing the computer program.

[0046] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a computing device, the computing device implements the image stabilization method for a single frame image of a video as described in any of the above embodiments.

[0047] The embodiment of the present application provides a method for stabilizing a single-frame image of a video: by acquiring a preset threshold number of continuous single-frame images in real time to generate a target image sequence, and then generating a first homography matrix sequence based on every two adjacent frames in the target image sequence, so as to describe the geometric transformation relationship between multiple adjacent frames through the first homography matrix sequence, and then comprehensively acquire the relationship between other multiple frames in the target image sequence and the intermediate single-frame image, and then perform weighted averaging on the first homography matrix sequence based on Gaussian filtering to obtain a smoothed target homography matrix, and then perform a projection transformation on the current intermediate image frame through the target homography matrix, that is, A target stabilized image frame can be obtained; therefore, the present application can fully consider the geometric transformation relationship between multiple adjacent frames by obtaining the first homography matrix between multiple adjacent frames related to the intermediate single-frame image, and then perform Gaussian filtering on the first homography matrix between multiple adjacent frames to obtain a smoothed target homography row matrix, so that the corresponding target stabilized image obtained by the transmission transformation of the intermediate single-frame image through the target homography row matrix is ​​more accurate and of higher quality. At the same time, in the present application, as long as the number of the single-frame images reaches a preset threshold, the subsequent processing can be started to improve the efficiency of obtaining the stabilized image. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings required in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0050] Figure 1 One of the step flow charts of the image stabilization method for a single frame image of a video provided in an embodiment of the present application;

[0051] Figure 2 A second flowchart of the method for stabilizing a single-frame image of a video provided in an embodiment of the present application;

[0052] Figure 3 A schematic diagram of a structure for acquiring image stabilization images in a video provided by an embodiment of the present application;

[0053] Figure 4 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0054] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0055] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0056] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a concrete way. In addition, in the description of the embodiments of the present application, unless otherwise specified, the meaning of "multiple" refers to two or more.

[0057] It should be noted that, in this article, the term "comprises" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0058] The present application embodiment provides a method for stabilizing a single frame of a video. Figure 1 As shown, the image stabilization method for a single frame of a video includes the following steps S101-S105:

[0059] S101 . Decode the video to be processed and obtain multiple continuous single-frame images in real time.

[0060] With the development of science and technology, social media and short videos are becoming more and more popular in modern society, and videos are playing an increasingly important role in our daily lives. However, because daily videos are usually shot with amateur equipment such as mobile phones and drones, which do not have image stabilization functions, the shooting effect is not good. If professional hardware equipment is used for image stabilization, it has disadvantages such as inconvenience in carrying and high cost. Therefore, after obtaining videos shot by various devices, it is necessary to stabilize the video to improve video jitter.

[0061] In an embodiment of the present application, each frame of the video to be processed is taken as an intermediate frame, and the intermediate frame is taken as the frame to be stabilized. A preset threshold number of single-frame images are obtained each time to generate the target image sequence. By including the target image sequence of the frame to be stabilized, the dynamic trajectory of multiple frames before is considered, and the target stabilized frame corresponding to the frame to be stabilized is obtained comprehensively and accurately, and then the single-frame image is stabilized and repaired.

[0062] Therefore, when stabilizing the video to be processed, it is necessary to decode it to obtain multiple continuous single-frame images corresponding to the video to be processed, so that the multiple single-frame images can be used for subsequent analysis and processing to obtain the stabilized frames corresponding to each single-frame image. Specifically, when decoding the video to be processed, the specific implementation method of obtaining multiple continuous single-frame images in real time can be to use a video decoder to decode and obtain multiple single-frame images.

[0063] S102: When the number of the single-frame images reaches a preset threshold, generate a target image sequence based on the preset threshold number of single-frame images.

[0064] In an embodiment of the present application, when stabilizing the video to be processed, it is necessary to obtain single-frame images of a preset number of frames in a row, and then obtain the stabilized frame of the intermediate frame through the single-frame images of the preset number of frames. Therefore, it is necessary to set the preset threshold, and then in the process of real-time video decoding, each time the number of single-frame images obtained by decoding reaches the preset threshold, a frame of target stabilized frame can be generated; in order to facilitate the determination of the intermediate frame each time, the value of the preset threshold in the present application is an odd number.

[0065] Exemplarily, when the preset threshold is set to 5, during the video decoding process, single-frame image frames from the 1st frame to the 5th frame are obtained, then the current intermediate frame is the third frame, and the target image sequence is generated based on the single-frame image frames from the 1st frame to the 5th frame, and then the following step S103 is executed.

[0066] S103 . For each of the single-frame images of two adjacent frames in the target image sequence, extract corresponding feature points respectively, so as to obtain a first homography matrix corresponding to the single-frame images of the two adjacent frames based on the feature points.

[0067] Among them, the homography matrix (Homography matrix) in computer vision, the homography of a plane is defined as the projection mapping from one plane to another plane. If a point P on the calibration plate is mapped to a point m on the imager using homogeneous coordinates, this mapping relationship matrix is ​​the homography matrix between the two planes. In this application, by obtaining the first homography matrix of each two adjacent frames in the target image sequence, the coordinate transformation relationship between multiple frames related to the intermediate single-frame image is comprehensively obtained, and then the first homography matrix sequence is weighted averaged based on Gaussian filtering to achieve a smoothing effect, and the target homography matrix for transforming the current intermediate image frame is obtained, and then the current intermediate image frame is projected and transformed by the target homography matrix, and the target image stabilization frame can be obtained, thereby achieving the anti-shake effect of the video to be processed.

[0068] Furthermore, in order to accurately analyze the relationship between two adjacent frames of images, it is necessary to extract representative features from each frame that can describe the local or overall characteristics of the image. These features should be unaffected by common transformations such as translation, rotation, and scaling of the image to a certain extent, so that they can be accurately matched and analyzed later.

[0069] Because there are often some geometric transformation relationships and feature changes between two adjacent frames due to factors such as object movement, camera movement or subtle changes in the scene, this information can be captured by analyzing two adjacent frames to facilitate subsequent image stabilization.

[0070] In some embodiments, when the target image sequence includes 5 single-frame images, the single-frame images of each adjacent two frames in the target image sequence include: the 1st frame and the 2nd frame, the 2nd frame and the 3rd frame, the 3rd frame and the 4th frame, and the 4th frame and the 5th frame, which are 4 pairs of single-frame images of adjacent two frames. Subsequently, the feature points corresponding to each pair of adjacent two single-frame images are extracted respectively, and then the feature points between the two adjacent frames are used for matching, and then the correspondence between the feature points of the two frames is determined, for example, which feature point a certain feature point in the first frame image corresponds to in the second frame image. Afterwards, the corresponding first homography matrix can be obtained based on the definition formula of the homography matrix through the feature point coordinates in the first frame image and the corresponding feature point coordinates in the second frame image. Based on the above method, the first homography matrix between each pair of adjacent frames is calculated to obtain 4 first homography matrices.

[0071] It should be noted that, in the above step S102, when the target image sequence is generated by a preset threshold, there may be a situation where the boundary frame cannot be used as an intermediate frame. Furthermore, when subsequently generating the target stabilized frame corresponding to the boundary frame, due to the lack of the first homography matrix, a first homography matrix sequence that meets the requirements cannot be generated. To solve this problem, in the present application, when obtaining the first homography matrix sequence corresponding to the boundary frame, the missing first homography matrix is ​​replaced by a unit matrix for subsequent calculations.

[0072] For example, when the preset threshold value is 5, when generating the target stabilized frame corresponding to the first frame of the video to be processed, the first frame needs to be used as an intermediate frame. Since there is no single frame of video before the first frame, the target image sequence is: [first frame, second frame, third frame]. At this time, the first homography matrix obtained according to the target image sequence has only two first homography matrices. In order to facilitate the uniformity of subsequent calculations, the unit matrix is ​​used instead of the missing two first homography matrices.

[0073] S104: Generate a first homography matrix sequence based on the multiple first homography matrices, and perform Gaussian filtering on the first homography matrix sequence to obtain a target homography matrix.

[0074] In some embodiments, after obtaining multiple first homography matrices for single-frame images adjacent to a preset threshold frame, these homography matrices may contain noise, which may be caused by errors in feature point detection and matching, etc. Therefore, it is necessary to perform Gaussian filtering on the first homography matrix sequence to smooth the noise, so that the target homography matrix transformation obtained by smoothing is more stable and accurate, and then the target stabilized image obtained by performing a subsequent perspective transformation on the intermediate single-frame image of the image sequence through the target homography matrix is ​​more accurate and of higher quality.

[0075] S105 . Performing a transmission transformation on an intermediate single-frame image at an intermediate frame position of the target image sequence based on the target homography matrix to generate a target stabilized image corresponding to the intermediate single-frame image.

[0076] Among them, perspective transformation (Warping) is a transformation method that projects a plane image onto another plane. It can simulate the perspective effect of human eyes or cameras when observing objects. In a two-dimensional image, perspective transformation can be described by a 3x3 homography matrix. This matrix defines the mapping relationship between points on the source image plane and points on the target image plane.

[0077] After the target homography matrix is ​​obtained, the target homography matrix can be used to perform the above-mentioned transmission transformation on the intermediate frame in the current target image sequence to obtain the corresponding target stabilized image frame.

[0078] It should be noted that when the target homography matrix is ​​used to perform transmission transformation on the image frame to be stabilized, the image frame to be stabilized is an image in YUV format. When the image in YUV format is used for transmission transformation, the image size will be reduced to 60% of the original. Therefore, using the image in YUV format for transmission transformation can save memory space and improve processing speed compared to using RGBA images for transmission transformation.

[0079] After obtaining the target image stabilization frame in step S105, it is necessary to obtain the target image stabilization frame corresponding to each frame in the video to be processed according to the method provided in steps S101 to S104 above, and then encode the target image stabilization frame corresponding to each frame to obtain the image stabilization video corresponding to the video to be processed. The specific steps include:

[0080] Repeat the steps of decoding the video to be processed, acquiring multiple continuous single-frame images in real time to generate a first homography matrix sequence based on multiple first homography matrices, and performing Gaussian filtering on the first homography matrix sequence to obtain a target homography matrix, until the target stabilized frame corresponding to each frame in the video to be processed is obtained.

[0081] Encode the stabilized frame corresponding to each frame in the video to be processed to obtain a target stabilized video corresponding to the video to be processed.

[0082] The embodiment of the present application provides a method for stabilizing a single-frame image of a video: by acquiring a preset threshold number of continuous single-frame images in real time to generate a target image sequence, and then generating a first homography matrix sequence based on every two adjacent frames in the target image sequence, so as to describe the geometric transformation relationship between multiple adjacent frames through the first homography matrix sequence, and then comprehensively acquire the relationship between other multiple frames in the target image sequence and the intermediate single-frame image, and then perform weighted averaging on the first homography matrix sequence based on Gaussian filtering to obtain a smoothed target homography matrix, and then perform a projection transformation on the current intermediate image frame through the target homography matrix, that is, A target stabilized image frame can be obtained; therefore, the present application can fully consider the geometric transformation relationship between multiple adjacent frames by obtaining the first homography matrix between multiple adjacent frames related to the intermediate single-frame image, and then perform Gaussian filtering on the first homography matrix between multiple adjacent frames to obtain a smoothed target homography row matrix, so that the corresponding target stabilized image obtained by the transmission transformation of the intermediate single-frame image through the target homography row matrix is ​​more accurate and of higher quality. At the same time, in the present application, as long as the number of the single-frame images reaches a preset threshold, the subsequent processing can be started to improve the efficiency of obtaining the stabilized image.

[0083] As an extension and refinement of the above embodiment, refer to Figure 2As shown, the present application provides a method for stabilizing a single frame of a video, further comprising the following steps S201 to S208:

[0084] S201 . Decode the video to be processed and obtain multiple continuous single-frame images in real time.

[0085] S202: When the number of the single-frame images reaches a preset threshold, convert the image formats of the single-frame images of the preset threshold into a grayscale image format to generate an initial image sequence.

[0086] In some embodiments, the image frames obtained by decoding the video to be processed are usually images in RGBA format or other formats other than grayscale format. In order to facilitate subsequent feature extraction and transmission transformation, it is necessary to convert the image format of a preset threshold number of single-frame images obtained by decoding the video to be processed into a grayscale image format.

[0087] Among them, RGBA format is a 32-bit color encoding format, where R (Red) represents the red channel, G (Green) represents the green channel, B (Blue) represents the blue channel, and A (Alpha) represents the transparency channel. The value of each channel is usually represented by 8 bits, ranging from 0 to 255, so that a total of 10000 color and transparency combinations can be represented.

[0088] A grayscale image is an image that contains only brightness information but no color information. Each pixel is represented by a single value, which is usually between 0 (black) and 255 (white). For example, 0 represents pure black, 255 represents pure white, and the values ​​in between represent different shades of gray. This representation method makes the data structure of grayscale images relatively simple, and it is more efficient in storage and processing than color images.

[0089] Specifically, the specific method of converting the image in RGBA format into grayscale format can be implemented using existing methods. Then, the M image frames in the initial image sequence are converted into grayscale images.

[0090] S203: Downscaling each of the single-frame images in the initial image sequence based on a preset downscaling coefficient to generate a target image sequence.

[0091] In the embodiment of the present application, it is also necessary to perform a reduction process on each of the single-frame images in the initial image sequence, in order to greatly improve the single-frame processing speed when extracting feature points of the image later. Specifically, the reduction coefficient can be expressed in the form of a matrix, referring to the following expression:

[0092]

[0093] Among them, scaleH represents the reduction coefficient, and s can be understood as the reduction multiple.

[0094] It should be noted that when reducing an image, a reasonable scaling factor should be set according to the actual situation. When reducing an image, the number of pixels in the image will decrease. For example, if the side length of an image is reduced to half of the original, the number of pixels will become one quarter of the original. When extracting feature points, it is usually necessary to calculate each pixel or a local area of ​​a pixel. Reducing the number of pixels will reduce the amount of calculation.

[0095] S204 . For each of the single-frame images of two adjacent frames in the target image sequence, extract corresponding feature points based on an ORB feature point extraction algorithm.

[0096] Specifically, the ORB (Oriented FAST and Rotated BRIEF) feature point extraction algorithm is a fast and efficient feature point extraction and description algorithm. It combines the speed advantage of the FAST (Features from Accelerated Segment Test) corner detection algorithm and the efficiency of the BRIEF (Binary Robust Independent Elementary Features) descriptor, and adds the calculation of the feature point direction on this basis, making the feature points rotationally invariant.

[0097] In the embodiment of the present application, corresponding feature points are extracted respectively by using an ORB feature point extraction algorithm to efficiently and quickly obtain feature points, thereby improving the efficiency of obtaining a target stabilized image frame.

[0098] S205 , performing feature point matching on the single-frame images of each two adjacent frames to calculate a first homography matrix corresponding to the single-frame images of the two adjacent frames.

[0099] In the embodiment of the present application, when obtaining the first homography matrix corresponding to the single-frame images of two adjacent frames, it is necessary to first extract the feature points corresponding to the single-frame images of each of the two adjacent frames, and then use the feature points between the two adjacent frames for matching, and then determine the correspondence between the feature points of the two frames, for example, which feature point a certain feature point in the first frame image corresponds to in the second frame image. Then, the corresponding first homography matrix is ​​obtained by iterative calculation through the RANSAC algorithm.

[0100] The specific implementation method of performing feature point matching on the single-frame images of each two adjacent frames to calculate the first homography matrix corresponding to the single-frame images of the two adjacent frames includes the following steps 1 and 2:

[0101] Step 1: perform feature point matching on the single-frame images of each two adjacent frames, and perform iterative calculations using a RANSAC algorithm to obtain an initial first homography matrix corresponding to the single-frame images of the two adjacent frames.

[0102] Among them, the RANSAC (Random Sample Consensus) algorithm is a model fitting method for processing data containing noise and errors. In this case, a homography matrix that can accurately describe the relationship between the two frames of images is fitted by extracting the feature points of the two adjacent frames of images, which is used to describe the perspective transformation relationship between the two frames of images.

[0103] In each iteration, a subset is first randomly selected from the extracted feature points. For calculating the homography matrix between two frames of images, four pairs of feature points are usually randomly selected (because the homography matrix has 8 degrees of freedom, four pairs of feature points can uniquely determine a homography matrix).

[0104] Then, using the extracted feature point subset, a hypothetical homography matrix is ​​calculated according to the corresponding mathematical model, and then all the extracted feature points are verified using this hypothetical homography matrix. The specific verification method is to transform the feature points in the first frame image to the corresponding positions in the second frame image according to the model, and then calculate the distance between the transformed feature points and the corresponding feature points in the actual second frame image (i.e., the reprojection error).

[0105] Repeat the above iterative process several times (for example, set the number of iterations to N), and record the reprojection error of the current assumed homography matrix after each iteration. Finally, select the assumed homography matrix with the smallest reprojection error as the final result, that is, as the first homography matrix. In this way, through multiple iterations, a relatively accurate model can be found to describe the relationship between two adjacent frames of images in the presence of noise (for example, there may be errors in feature point extraction, there may be errors in matching, etc.).

[0106] Through the above steps, for every two adjacent frames in the target image sequence, features are first extracted and then iteratively calculated through the RANSAC algorithm. This can accurately obtain the homography matrix that can describe the relationship between the two frames of images, providing an important foundation for subsequent image processing tasks.

[0107] Step 2: Based on a preset magnification factor, the initial first homography matrix is ​​magnified to obtain the first homography matrix.

[0108] In the embodiment of the present application, since each of the single-frame images in the initial image sequence is reduced in S203, it is necessary to enlarge the initial first homography matrix based on the corresponding magnification factor after obtaining the initial first homography matrix, so as to transform the initial first homography matrix back to the size of the single-frame image in the video to be processed, so as to generate the first homography matrix, and then make the target homography matrix obtained by the first homography matrix sequence applicable to the transmission transformation of the image frame to be stabilized.

[0109] It should be noted that when performing the enlargement processing, it is necessary to use the enlargement coefficient corresponding to the reduction coefficient in step S203, that is, when each of the single-frame images in the initial image sequence is reduced to 1 / 4 of the original single-frame image through the reduction coefficient, then at this time it is necessary to enlarge the initial first homography matrix to 4 times the original initial first homography matrix through the enlargement coefficient.

[0110] Specifically, similarly, the magnification coefficient can be expressed in matrix form and can be obtained by calculating the inverse matrix of the reduction matrix. The obtained magnification coefficient refers to the following expression:

[0111]

[0112] Among them, scaleH_inv represents the inverse matrix of the reduction matrix, that is, the magnification coefficient corresponding to the reduction coefficient, and s can be understood as the magnification factor.

[0113] S206: Generate a first homography matrix sequence based on the multiple first homography matrices, and perform Gaussian filtering on the first homography matrix sequence to obtain a target homography matrix.

[0114] Specifically, the step of generating a first homography matrix sequence based on the plurality of first homography matrices, and performing Gaussian filtering on the first homography matrix sequence to obtain a refinement step of a target homography matrix includes the following steps consistent with step 2:

[0115] Step 1: Generate a first homography matrix sequence based on multiple first homography matrices.

[0116] Step 2: Based on preset Gaussian coefficients, perform Gaussian filtering on the first homography matrix sequence to obtain a target homography matrix.

[0117] Specifically, when using the Gaussian rate wave to perform Gaussian filtering on the first homography matrix sequence, it is necessary to first obtain the preset Gaussian coefficients. Based on the Gaussian filtering principle, for each first homography matrix in a filtering window, a weight will be calculated by a Gaussian function according to its distance from the center of the window. The weight of the homography matrix at the center is the largest, and the weights on both sides will decrease accordingly. Regarding the Gaussian coefficient of each homography matrix, it is determined based on the offset from the center of the window. This calculation process is actually to perform weighted summation of multiple homography matrices in the window according to the Gaussian coefficients, so that the homography matrix close to the center of the window has a larger weight, thereby achieving a filtering effect.

[0118] Exemplarily, when the target image sequence A includes 5 frames of images, 4 first homography matrices will be obtained. In this way, it is impossible to determine which first homography matrix is ​​at the center of the window. In this case, the centroid position of the first homography matrix sequence is inserted into a unit matrix, and then the unit matrix is ​​used to represent the geometric transformation of the third frame in the target image sequence relative to itself during Gaussian filtering.

[0119] When generating the target homography matrix of the middle single frame image in the target image sequence A, that is, the third frame, the window size M is set to 5. For one-dimensional Gaussian filtering, according to the following Gaussian function formula:

[0120]

[0121] Among them, σ is the standard deviation, which can be set according to the window size and data characteristics, and adjusted according to the actual situation. A larger standard deviation will make the distribution of the Gaussian function wider and the filtering effect smoother, but may over-blur the data; a smaller standard deviation will make the filtering more "local", retaining more details, but the filtering effect may be weaker.

[0122] Assume that the center of the window is i=0, then the range of the offset x relative to the center of the window is from arrive Based on M=5, the value range of x is: -2, -1, 0, 1, 2; substituting the standard deviation and the x value into the above formula, we can get G(-2), G(-1), G(0), G(1) and G(2). Then, the target homography matrix of the third frame can be calculated by the following formula:

[0123]

[0124] S207 , converting the image format of the intermediate single-frame image at the intermediate frame position of the target image sequence into a YUV image format to generate a target intermediate single-frame image.

[0125] Specifically, the reason for converting the format of the intermediate frame in the target image sequence into the YUV image format and then performing the transmission transformation is that the RGBA image obtained from the video to be processed is converted into a YUV image, and the image size will be reduced to 60% of the original single-frame image. Therefore, using the YUV image for transmission transformation can save memory space and improve processing speed compared to using the RGBA original image for transmission transformation.

[0126] It should be noted that the YUV image format is mainly used for the transmission and storage of video signals. It divides color information into a luminance component Y and two chrominance components U and V. The Y component represents the brightness information of the image, which determines the degree of black and white of the image. For example, in a black and white image, only the Y component is needed to fully represent the image. The U and V components represent the chrominance information of the color. The U component is usually associated with blue, and the V component is usually associated with red. They describe the hue and saturation of the color.

[0127] S208 . Perform a transmission transformation on the target intermediate single-frame image based on the target homography matrix to generate a target stabilized image corresponding to the target intermediate single-frame image.

[0128] Furthermore, by performing a transmission transformation on the target intermediate frame obtained after the format conversion in combination with the target homography matrix, the target stabilized image frame corresponding to the target intermediate frame can be obtained.

[0129] The present application can fully consider the geometric transformation relationship between multiple adjacent frames by acquiring the first homography matrix between multiple adjacent frames related to the intermediate single-frame image, and then perform Gaussian filtering on the first homography matrix between multiple adjacent frames to obtain a smoothed target homography row matrix, so that the corresponding target stabilized image obtained by the transmission transformation of the intermediate single-frame image through the target homography row matrix is ​​more accurate and of higher quality. At the same time, in the present application, as long as the number of the single-frame images reaches a preset threshold, subsequent processing can be started to improve the efficiency of acquiring stabilized images.

[0130] Based on the same inventive concept, as an implementation of the above method, an embodiment of the present application also provides a stabilization device for a single frame image of a video. This embodiment corresponds to the aforementioned method embodiment. For ease of reading, this embodiment will no longer repeat the details of the aforementioned method embodiment one by one, but it should be clear that an image stabilization device for a single frame image of a video in this embodiment can correspond to all the contents of the aforementioned method embodiment.

[0131] The embodiment of the present application provides an image stabilization device for a single frame image of a video, Figure 3 This is a schematic diagram of the structure of obtaining the image stabilization image for the image in the video, such as Figure 3 As shown, the image stabilization image acquisition 300 for the image in the video includes:

[0132] The decoding unit 301 is used to decode the video to be processed and obtain multiple continuous single-frame images in real time;

[0133] A generating unit 302 is configured to generate a target image sequence based on the preset threshold number of single-frame images when the number of the single-frame images reaches a preset threshold;

[0134] The extraction unit 303 is used to extract corresponding feature points for each of the single-frame images of two adjacent frames in the target image sequence, so as to obtain a first homography matrix corresponding to the single-frame images of the two adjacent frames based on the feature points;

[0135] A processing unit 304 is configured to generate a first homography matrix sequence based on the plurality of first homography matrices, and perform Gaussian filtering on the first homography matrix sequence to obtain a target homography matrix;

[0136] The transformation unit 305 is used to perform a transmission transformation on the intermediate single-frame image at the intermediate frame position of the target image sequence based on the target homography matrix to generate a target stabilized image corresponding to the intermediate single-frame image.

[0137] As an optional implementation of an embodiment of the present application, the stabilized image acquisition for images in the video also includes an encoding unit, which is specifically used to repeat the decoding of the video to be processed, acquire multiple continuous single-frame images in real time to generate a first homography matrix sequence based on multiple first homography matrices, and perform Gaussian filtering on the first homography matrix sequence to obtain the target homography matrix step until the target stabilized frame corresponding to each frame in the video to be processed is obtained; encode the stabilized frame corresponding to each frame in the video to be processed to obtain the target stabilized video corresponding to the video to be processed.

[0138] As an optional implementation of the embodiment of the present application, the generation unit 302 is specifically used to convert the image format of the single-frame images of the preset threshold value into a grayscale image format to generate an initial image sequence when the number of the single-frame images reaches a preset threshold value; and to reduce the size of each of the single-frame images in the initial image sequence based on a preset reduction coefficient to generate a target image sequence.

[0139] As an optional implementation of the embodiment of the present application, the extraction unit 303 is specifically used to extract corresponding feature points for the single-frame images of every two adjacent frames in the target image sequence based on the ORB feature point extraction algorithm; and perform feature point matching on the single-frame images of every two adjacent frames to calculate the first homography matrix corresponding to the single-frame images of the two adjacent frames.

[0140] As an optional implementation of the embodiment of the present application, the extraction unit 303 is specifically used to perform feature point matching on the single-frame images of each two adjacent frames, and to perform iterative calculations through the RANSAC algorithm to obtain an initial first homography matrix corresponding to the single-frame images of the two adjacent frames; based on a preset magnification factor, the initial first homography matrix is ​​magnified to obtain the first homography matrix.

[0141] As an optional implementation of the embodiment of the present application, the extraction unit 303 is specifically used to generate a first homography matrix sequence based on multiple first homography matrices; based on a preset Gaussian coefficient, perform Gaussian filtering on the first homography matrix sequence to obtain a target homography matrix. Combined with the Gaussian coefficient, perform Gaussian filtering on the first homography matrix sequence to obtain a target homography matrix.

[0142] As an optional implementation of the embodiment of the present application, the extraction unit 303 is specifically used to convert the image format of the intermediate single frame image at the intermediate frame position of the target image sequence into a YUV image format to generate a target intermediate single frame image; perform a transmission transformation on the target intermediate single frame image based on the target homography matrix to generate a target stabilized image corresponding to the target intermediate single frame image.

[0143] Based on the same inventive concept, an embodiment of the present disclosure also provides an electronic device. Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure, such as Figure 4 As shown, the electronic device provided in this embodiment includes: a memory 401 and a processor 402, wherein the memory 401 is used to store a computer program; and the processor 402 is used to execute the audio data processing method provided in the above embodiment when executing the computer program.

[0144] Based on the same inventive concept, an embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the computing device implements the image stabilization method for a single frame image of a video provided in the above embodiment.

[0145] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media that include computer-usable program code.

[0146] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0147] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0148] Computer readable media include permanent and non-permanent, removable and non-removable storage media. Storage media can be implemented by any method or technology to store information, and the information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0149] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for stabilizing a single-frame image of a video, characterized in that: include: Decode the video to be processed and obtain multiple continuous single-frame images in real time; When the number of the single-frame images reaches a preset threshold, generating a target image sequence based on the preset threshold number of single-frame images; For each of the single-frame images of two adjacent frames in the target image sequence, corresponding feature points are extracted respectively, so as to obtain a first homography matrix corresponding to the single-frame images of the two adjacent frames based on the feature points; Generate a first homography matrix sequence based on the multiple first homography matrices, and perform Gaussian filtering on the first homography matrix sequence to obtain a target homography matrix; Based on the target homography matrix, a transmission transformation is performed on the intermediate single-frame image at the intermediate frame position of the target image sequence to generate a target stabilized image corresponding to the intermediate single-frame image.

2. The method according to claim 1, characterized in that The method further comprises: Repeat the steps of decoding the video to be processed, acquiring multiple continuous single-frame images in real time, generating a first homography matrix sequence based on multiple first homography matrices, and performing Gaussian filtering on the first homography matrix sequence to acquire a target homography matrix, until the target image stabilization frame corresponding to each frame in the video to be processed is acquired; Encode the stabilized frame corresponding to each frame in the video to be processed to obtain a target stabilized video corresponding to the video to be processed.

3. The method according to claim 1, characterized in that When the number of the single-frame images reaches a preset threshold, generating a target image sequence based on the preset threshold number of single-frame images includes: When the number of the single-frame images reaches a preset threshold, converting the image formats of the single-frame images of the preset threshold into grayscale image formats to generate an initial image sequence; Each of the single-frame images in the initial image sequence is reduced based on a preset reduction coefficient to generate a target image sequence.

4. The method according to claim 1, characterized in that: The extracting corresponding feature points from each of the single-frame images of two adjacent frames in the target image sequence to obtain a first homography matrix corresponding to the single-frame images of the two adjacent frames based on the feature points includes: For each of the single-frame images of two adjacent frames in the target image sequence, corresponding feature points are extracted based on an ORB feature point extraction algorithm; Feature point matching is performed on the single-frame images of each two adjacent frames to calculate a first homography matrix corresponding to the single-frame images of the two adjacent frames.

5. The method according to claim 4, characterized in that The performing feature point matching on the single-frame images of each two adjacent frames to calculate a first homography matrix corresponding to the single-frame images of the two adjacent frames includes: Performing feature point matching on the single-frame images of each two adjacent frames, and performing iterative calculations using a RANSAC algorithm to obtain an initial first homography matrix corresponding to the single-frame images of the two adjacent frames; Based on a preset magnification factor, the initial first homography matrix is ​​magnified to obtain the first homography matrix.

6. The method according to claim 1, characterized in that The step of generating a first homography matrix sequence based on the plurality of first homography matrices and performing Gaussian filtering on the first homography matrix sequence to obtain a target homography matrix includes: generating a first homography matrix sequence based on a plurality of the first homography matrices; Based on preset Gaussian coefficients, Gaussian filtering is performed on the first homography matrix sequence to obtain a target homography matrix.

7. The method according to claim 1, characterized in that The step of performing a transmission transformation on an intermediate single-frame image at an intermediate frame position of the target image sequence based on the target homography matrix to generate a target stabilized image corresponding to the intermediate single-frame image includes: Converting the image format of the intermediate single-frame image at the intermediate frame position of the target image sequence into a YUV image format to generate a target intermediate single-frame image; A transmission transformation is performed on the target intermediate single-frame image based on the target homography matrix to generate a target stabilized image corresponding to the target intermediate single-frame image.

8. An image stabilization device for a single frame of a video, characterized in that: include: A decoding unit, used for decoding the video to be processed and obtaining multiple continuous single-frame images in real time; A generating unit, configured to generate a target image sequence based on the preset threshold number of single-frame images when the number of the single-frame images reaches a preset threshold; An extraction unit, configured to extract corresponding feature points respectively for each of the single-frame images of two adjacent frames in the target image sequence, so as to obtain a first homography matrix corresponding to the single-frame images of the two adjacent frames based on the feature points; A processing unit, configured to generate a first homography matrix sequence based on the plurality of first homography matrices, and perform Gaussian filtering on the first homography matrix sequence to obtain a target homography matrix; A transformation unit is used to perform a transmission transformation on an intermediate single-frame image at an intermediate frame position of the target image sequence based on the target homography matrix to generate a target stabilized image corresponding to the intermediate single-frame image.

9. An electronic device, characterized in that: include: A memory and a processor, wherein the memory is used to store a computer program; and the processor is used to enable the electronic device to implement the image stabilization method for a single frame image of a video as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a computing device, the computing device implements the image stabilization method for a single frame image of a video as described in any one of claims 1-7.

Images