Video image stability control method and system, electronic equipment and storage medium

Abstract

The invention discloses a video image stability control method and system, electronic equipment and a storage medium, and the method comprises the steps: S1, obtaining an image sequence in a video image, and estimating a motion parameter of a camera according to the difference between two adjacent frames of the obtained image sequence; and S2, carrying out real-time filtering processing on the estimated motion parameters so as to eliminate local motion, and carrying out smoothing processing according to a detected motion track so as to offset camera jitter and correct image jitter so as to generate a stable video image. According to the video image stabilization control method and system, the electronic equipment and the storage medium provided by the invention, each frame of image in the video sequence can be automatically stabilized. According to the invention, the electronic image can be processed in real time, and the problem of jittering during video shooting is solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of video data processing, and relates to an image stabilization control system, and particularly to a video image stabilization control method, system, electronic device, and storage medium. Background Art

[0002] Generally, digital video stabilization processing usually uses post - processing software to repair shaky videos. However, this method is affected by the length of video clips and cannot obtain stable videos in real - time, and professional knowledge is required to know how to repair them smoothly.

[0003] With the booming development of the live - streaming industry, more and more users regard video live - streaming as the main information carrier, which has deeply penetrated into people's daily lives in the community era and is widely used in smart city applications, the Internet of Things industry, etc. However, most people who upload videos may lack professional equipment or professional shooting skills. Therefore, the quality and effect of videos produced in a moving environment are poor and prone to jitter.

[0004] In order to reduce video jitter, traditional image stabilization techniques can be divided into three categories: mechanical video stabilization, optical video stabilization, and electronic image stabilization. The principle of mechanical video stabilization is to combine a servo system and mechanical sensors such as gyroscopes to ensure the stability of camera shooting. Mechanical devices can counteract part of the carrier jitter in the opposite direction, so video stabilization can be achieved. Common methods include installing professional photography auxiliary equipment, such as tripods, or using camera gimbals, handheld stabilizers, etc. The principle of optical video stabilization is to actively adjust the optical path through optical instruments to compensate for the light deviation and image movement caused by external lens jitter, so as to obtain a stable image. Because optical image stabilization is hardly interfered by the external environment, it is widely used in the lenses of high - end camera devices, such as pocket cameras, action cameras, or mobile phones with physical lens anti - shake, to reduce or eliminate slight shaking during shooting.

[0005] Electronic image stabilization processes the camera movement to compensate for jitter by performing image processing on each jittery frame in the video image, using motion filtering and motion compensation to reduce irregular jitter between image sequences, thereby achieving motion balance and making the video look smoother. Electronic image stabilization can stabilize videos without the need for specific equipment assistance, and has the advantages of low cost and more flexible operation, and can be widely used in various fields. Generally, digital video stabilization uses post - processing software to repair shaky videos, but this method cannot obtain stable videos in real - time, and professional knowledge is required to know how to repair them smoothly.

[0006] In view of this, there is an urgent need to design a new video image processing method to overcome at least some of the above - mentioned defects of existing video image processing methods. Summary of the Invention

[0007] The present invention provides a method, a system, an electronic device and a storage medium for video image stabilization control, which can automatically perform stabilization processing on each frame of image in a video sequence.

[0008] To solve the above technical problems, according to one aspect of the present invention, the following technical solutions are adopted:

[0009] A method for video image stabilization control, the video image stabilization control method includes:

[0010] Step S1, obtain an image sequence in the video image, and estimate the motion parameters of the camera according to the difference between two adjacent frames of the obtained image sequence;

[0011] Step S2, perform real-time filtering processing on the estimated motion parameters to eliminate local motion; perform motion compensation according to the detected motion trajectory to offset the camera shake, correct the image shake, and thus generate a stable video image.

[0012] As an implementation manner of the present invention, in step S1, each frame of the video sequence is sequentially read from the video. Except for the first frame, the previous frame and the current frame are read as grayscale images each time; the contrast of the input video image is enhanced during image preprocessing to make the image clear;

[0013] Traverse all frames in the video. To find the movement between the current frame and the previous frame, use an AI model to detect the correspondence between adjacent frame images; the input of the AI model is the current frame and the previous frame, and the AI model finds the correspondence existing between two adjacent frames before and after, so as to establish the motion relationship between adjacent frames. The AI model will predict the offset of the line coordinate points of the two images.

[0014] As an implementation manner of the present invention, in step S2, use an AI model to confirm the mutual relationship between corresponding points in the two grayscale images, so as to correct the distortion caused by shake between two adjacent video frames;

[0015] Considering the correspondence between two planes between two frames, the AI model outputs several groups of offset corresponding points between vertices, which are used as the input required for subsequent solving of the unit matrix. Then, use the direct linear transformation algorithm DLT (Direct Linear Transform) to obtain the homography matrix coefficients between adjacent frames;

[0016] Represent the homography matrix coefficients between adjacent frames as the current camera motion information, and multiply the homography matrix coefficients by the previous frame motion trajectory to obtain the motion trajectory of the jittery video.

[0017] Smoothing the motion trajectory using a digital filter, and performing spatial image warping adjustment on the original image according to the new motion trajectory to obtain a stable frame sequence.

[0018] As an implementation manner of the present invention, in step S1, the scenes captured by adjacent images are on the same plane, or two consecutive frames of images are captured when the camera rotates simply. Calculate the homography matrix through four pairs of matching feature points extracted from the two images. According to the predefined corner points and offsets, obtain the position corresponding to the features of the current frame in the previous frame, and obtain the transformation parameters of the homography matrix H through the direct linear transformation algorithm; the definition of the homography matrix H is as shown in Formula 1:

[0019]

[0020] Among them, h11, h12, h21, and h22 are rotation coefficients, and h13 and h23 are translation coefficients;

[0021] Apply the homography matrix H to the perspective transformation formula 2, where (x, y) represents the coordinates of the feature points of the current frame, and (x’, y’) represents the coordinates of the transformed feature points;

[0022]

[0023] The homography matrix coefficients between adjacent frames can represent the current camera motion information. Analyze the inter-frame motion situation to find the motion trajectory of the camera; multiply the homography matrix coefficients by the previous frame motion trajectory to obtain the motion trajectory of the jittery video;

[0024] In step S2, in order to perform video stabilization, a filtering method is used to smooth the original motion trajectory. The Savitzky-Golay filtering algorithm (SG Filter) used is a digital signal processing algorithm for smoothing signals;

[0025] Use the polynomial fitting method of partial least squares (PLS) to fit the local features of the curve, thereby achieving smoothing; SG Filter simultaneously realizes smoothing and denoising, reduces the noise of the data, and processes non-linear signals;

[0026] This algorithm has a fast convergence speed and is suitable for real-time signal processing. Therefore, the present invention selects SG Filter to smooth the motion trajectory, and finally performs spatial image warping adjustment (Warping) on the original image according to the new motion trajectory to obtain a stable frame sequence.

[0027] According to another aspect of the present invention, the following technical solution is adopted: A video image stabilization control system, the video image stabilization control system includes:

[0028] A camera motion parameter acquisition module is used to acquire an image sequence in a video image, and estimate the camera's motion parameters according to the difference between two adjacent frames of the acquired image sequence;

[0029] An image jitter correction module is used to perform real-time filtering on the motion parameters estimated by the camera motion parameter acquisition module, so as to eliminate local motion, perform motion compensation according to the detected motion trajectory to offset the camera jitter, correct the image jitter, and thus generate a stable video image.

[0030] As an implementation mode of the present invention, the camera motion parameter acquisition module sequentially reads each frame of the video sequence from the video. Except for the first frame, the previous frame and the current frame are read as grayscale images each time; the contrast of the input video image is enhanced during image preprocessing to make the image clear;

[0031] Traverse all frames in the video. To find the movement between the current frame and the previous frame, use an AI model to detect the correspondence between adjacent frame images; the input of the AI model is the current frame and the previous frame, and the AI model searches for the correspondence existing between two adjacent frames before and after, so as to establish the motion relationship between adjacent frames. The AI model will predict the offset of the line coordinate points of the two images.

[0032] As an implementation mode of the present invention, the image jitter correction module uses an AI model to confirm the mutual relationship between corresponding points in two grayscale images, so as to correct the distortion caused by jitter between two adjacent video frames;

[0033] Considering the correspondence between two planes between two frames, the AI model outputs several groups of offset corresponding points between vertices, which are used as the input required for subsequent solving of the unit matrix. Then, the direct linear transformation algorithm DLT (Direct Linear Transform) is used to obtain the homography matrix coefficients between adjacent frames;

[0034] The homography matrix coefficients between adjacent frames represent the current camera motion information. Multiply the homography matrix coefficients by the previous frame's motion trajectory to obtain the motion trajectory of the jittery video.

[0035] Use a digital filter to smooth the motion trajectory, and perform spatial image warping adjustment (Warping) on the original image according to the new motion trajectory to obtain a stable frame sequence.

[0036] As an implementation manner of the present invention, during the process of the camera motion parameter acquisition module estimating the motion parameters of the camera, the scenes captured by adjacent images are on the same plane or two consecutive frames of images are captured when the camera rotates simply. The homography matrix is calculated through four pairs of matching feature points extracted from the two images. According to the predefined corner points and offsets, the position corresponding to the features of the current frame in the previous frame is obtained, and the transformation parameters of the homography matrix H are obtained through the direct linear transformation algorithm; the definition of the homography matrix H is as shown in Formula 1:

[0037]

[0038] Where h11, h12, h21, and h22 are rotation coefficients, and h13 and h23 are translation coefficients;

[0039] The homography matrix H is applied to the perspective transformation formula 2, where (x, y) represents the coordinates of the feature points of the current frame, and (x’, y’) represents the coordinates of the transformed feature points;

[0040]

[0041] The homography matrix coefficients between adjacent frames can represent the current camera motion information. Analyze the inter-frame motion situation to find the motion trajectory of the camera; multiply the homography matrix coefficients with the previous frame's motion trajectory to obtain the motion trajectory of the jittery video;

[0042] During the process of the image jitter correction module performing image stabilization processing, in order to perform image stabilization, a filtering method is used to smooth the original motion trajectory. The Savitzky-Golay filtering algorithm (SG Filter) used is a digital signal processing algorithm for smoothing signals;

[0043] The polynomial fitting method of partial least squares is used to fit the local features of the curve, thereby achieving smoothing processing; SG Filter simultaneously realizes smoothing and denoising, reduces the noise of the data, and processes non-linear signals;

[0044] Adjust the spatial image distortion of the original image according to the new motion trajectory to obtain a stable frame sequence.

[0045] According to another aspect of the present invention, the following technical solution is adopted: An electronic device includes a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the steps of the above method are implemented.

[0046] According to another aspect of the present invention, the following technical solution is adopted: A storage medium stores computer program instructions thereon, and when the computer program instructions are executed by a processor, the steps of the above method are implemented.

[0047] The beneficial effects of the present invention are as follows: The video image stabilization control method, system, electronic device, and storage medium proposed by the present invention can automatically perform stabilization processing on each frame of the video sequence. The present invention can perform real-time processing on electronic images and solve the problem of jitter during video shooting.

[0048] The present invention proposes an electronic image stabilization algorithm for immediate video processing. This algorithm can be applied in an online real-time mode to automatically perform stabilization processing on each frame of the video sequence to generate a smoother stabilized video and improve the viewing quality.

[0049] The electronic image stabilization method of the present invention can perform video stabilization without the assistance of specific equipment, has the advantages of low cost and more flexible operation, and can be widely applied in various fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 It is a flowchart of the video image stabilization control method in an embodiment of the present invention.

[0051] Figure 2 It is a schematic diagram of the composition of the video image stabilization control system in an embodiment of the present invention.

[0052] Figure 3 It is a schematic diagram of the AI model calculating the offset between four vertices in an embodiment of the present invention.

[0053] Figure 4 It is a schematic diagram of the direct linear transformation algorithm in an embodiment of the present invention.

[0054] Figure 5 It is a schematic diagram of the composition of the electronic device in an embodiment of the present invention.

[0055] Figure 6 It is a schematic diagram comparing the offset of coordinate points with the prior art in an embodiment of the present invention.

[0056] Figure 7 It is a schematic diagram of the AI model predicting the offset of coordinate points of two graph lines in an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0057] The preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0058] To further understand the present invention, the preferred implementation of the present invention will be described below with reference to the embodiments. However, it should be understood that these descriptions are only for further explaining the features and advantages of the present invention, rather than limiting the claims of the present invention.

[0059] The description of this part is only for several typical embodiments, and the present invention is not limited to the scope described in the embodiments. The mutual replacement of the same or similar prior art means and some technical features in the embodiments is also within the scope of description and protection of the present invention.

[0060] The expressions of the steps in the various embodiments in the specification are only for convenience of description, and the implementation manner of the present application is not limited by the order of step implementation.

[0061] "Coupling" or "connection" in the specification includes both direct connection and indirect connection.

[0062] The present invention discloses a method for video image stabilization control. Figure 1 is a flowchart of the video image stabilization control method in an embodiment of the present invention; please refer to Figure 1 The video image stabilization control method includes:

[0063]

Step S1

[0064] In an embodiment of the present invention, in step S1, each frame of the video sequence is sequentially read from the video. Except for the first frame, the previous frame and the current frame are read as grayscale images each time; because the stabilization algorithm can operate based on grayscale values only without the need for RGB, and at the same time, it can improve the operation speed. At the same time, the input video image can be preprocessed to enhance the contrast to make the image clear.

[0065] Traverse all frames in the video. To find the movement between the current frame and the previous frame, an AI model is used to detect the corresponding relationship between adjacent frame images; the input of the AI model is the current frame and the previous frame, and the AI model searches for the corresponding relationship existing between two adjacent frames before and after, so as to establish the motion relationship between adjacent frames, and the AI model will predict the offset of the coordinate points of the two images.

[0066]

Step S2

[0067] In an embodiment of the present invention, in step S2, the mutual relationship between corresponding points in the two grayscale images confirmed by the AI model is used to correct the distortion caused by shake between two adjacent video frames.

[0068] Considering the correspondence between two planes between two frames, the AI model outputs several sets of offset corresponding points between vertices as the input required for subsequent solving of the unit matrix. Then, using the Direct Linear Transform (DLT) algorithm, the homography matrix coefficients between adjacent frames can be obtained (as shown in Figure 4 );

[0069] The homography matrix coefficients between adjacent frames represent the current camera motion information. Multiply the homography matrix coefficients by the previous frame's motion trajectory to obtain the motion trajectory of the jittery video.

[0070] Use a digital filter to smooth the motion trajectory, and perform spatial image warping adjustment on the original image according to the new motion trajectory to obtain a stable frame sequence.

[0071] Please continue to refer to Figure 1 , in a usage scenario of the present invention, in step S1, the scenes captured by adjacent images are on the same plane, or two adjacent frames of images are captured when the camera rotates simply. Calculate the homography matrix through four pairs of matching feature points extracted from the two images (which can be combined with Figure 3 ); According to the predefined corner points and offsets, obtain the position corresponding to the features of the current frame in the previous frame, and obtain the transformation parameters of the homography matrix H through the Direct Linear Transform algorithm; The definition of the homography matrix H is as shown in Formula 1:

[0072]

[0073] Among them, h11, h12, h21, and h22 are rotation coefficients, and h13 and h23 are translation coefficients; h31, h32, and h33 can usually be 1.

[0074] Calculate the motion trajectory according to the homography transformation matrix and smooth the motion trajectory. Because the scenes captured by adjacent images are on the same plane, or two adjacent frames of images are captured when the camera rotates simply, the homography matrix can be calculated through four pairs of matching feature points extracted from the two images. According to the predefined corner points and offsets, as shown in Figure 7 , the AI model will predict the offsets of the two graph coordinate points. Obtain the homography transformation matrix representing the motion based on the feature changes between the front and rear two images. It is possible to know the position corresponding to the features of the current frame in the previous frame. For example, through the Direct Linear Transform (DLT) algorithm, establish the relationship between the image and the projection of the corresponding object, and the transformation parameters of the homography matrix H can be obtained.

[0075] The formula for applying the homography matrix H to perspective transformation is as follows. The homography matrix is a 3x3 matrix that describes the mapping relationship from plane to plane in projective geometry. It has 8 degrees of freedom and consists of nine elements. h11, h12, h21, and h22 are rotation coefficients, and h13 and h23 are translation coefficients. Usually, the last element (h31, h32, h33) is set to 1. Among them, (x, y) represents the coordinates of the feature points in the current frame, and (x’, y’) represents the coordinates of the transformed feature points.

[0076] In an embodiment of the present invention, the homography process between two frames can be referred to Figure 7 ; For the inter-frame motion estimation of the front and rear two frames of images, for all pixels of the two pictures, find the corresponding relationship (X->X’, which can be transformed using Homographies).

[0077] Apply the homography matrix H to the perspective transformation formula 2, where (x, y) represents the coordinates of the feature points in the current frame, and (x’, y’) represents the coordinates of the transformed feature points;

[0078]

[0079] The homography matrix coefficients between adjacent frames can represent the current camera motion information. Analyze the inter-frame motion situation to find the motion trajectory of the camera; multiply the homography matrix coefficients with the previous frame's motion trajectory to obtain the motion trajectory of the jittery video.

[0080] In step S2, to perform image stabilization, a filtering method is used to smooth the original motion trajectory. The Savitzky-Golay filtering algorithm (SG Filter) used is a digital signal processing algorithm for smoothing signals and differentiating them. The SG Filter algorithm uses the polynomial fitting method of partial least squares (PLS) to fit the local features of the curve, thereby achieving smoothing; the SG Filter simultaneously realizes smoothing and denoising, reduces the noise of the data, and processes non-linear signals. This algorithm has a fast convergence speed and is suitable for real-time signal processing; the SG Filter can be selected to smooth the motion trajectory, and finally, according to the new motion trajectory, perform spatial image warping adjustment (Warping) on the original image to obtain a stable frame sequence.

[0081] The purpose of the electronic image stabilization method is to given a jittery video, output a video with relatively strong stability and less distortion. In an embodiment of the present invention, the anti-shake algorithm includes: 1. Motion estimation; 2. Jitter identification; 3. Motion compensation.

[0082] Electronic image stabilization method is to detect the motion parameters of the camera according to the difference between two adjacent frames of the image sequence through image processing algorithms, and to offset the camera shake through the smoothing of the motion trajectory, so as to correct the image shake and obtain a stable video image. The process includes analyzing the motion pattern in the video, estimating the motion parameters of the current frame relative to the previous frame, such as translation, rotation, and scaling, by finding the point correspondence between two frames. According to the motion estimation result, the estimated motion parameters are filtered in real time to remove local motion, and finally smoothed according to the detected motion trajectory to generate a stable video image.

[0083] In the motion estimation step, the inter-frame global motion between two consecutive frames except the first frame is estimated. Figure 1 For the image stabilization process, the present invention performs image preprocessing on the input video to enhance the contrast to make the image clear and convert the RGB into a grayscale image. Then, all frames in the video are traversed. In order to find the movement between the current frame and the previous frame, the present invention uses an AI image registration model, aiming to find the spatial mapping relationship between the pixels of the adjacent previous image and the pixels of the subsequent image. Traditional image registration uses artificial features to find the correspondence, while deep learning uses neural networks to directly learn the geometric transformation between two images. Convolutional neural networks learn complex image features, so the method of using deep learning can learn the homography in an end-to-end manner. In the present invention, the input of the AI model is the current frame and the previous frame, and the model will find the correspondence existing between two adjacent frames before and after, so as to establish the motion relationship between adjacent frames.

[0084] The present invention also discloses a video image stabilization control system. Figure 2 It is a schematic diagram of the composition of the video image stabilization control system in an embodiment of the present invention; please refer to Figure 2 , the video image stabilization control system includes: a camera motion parameter acquisition module 1 and an image shake correction module 2.

[0085] The camera motion parameter acquisition module 1 is used to acquire the image sequence in the video image, and estimate the motion parameters of the camera according to the difference between two adjacent frames of the acquired image sequence.

[0086] The image shake correction module 2 is used to perform real-time filtering processing on the motion parameters estimated by the camera motion parameter acquisition module, so as to remove local motion, perform motion compensation according to the detected motion trajectory to offset the camera shake, correct the image shake, and generate a stable video image.

[0087] In an embodiment of the present invention, the camera motion parameter acquisition module sequentially reads each frame of the video sequence from the video. Except for the first frame, each time the previous frame and the current frame are read as grayscale images; because the stabilization algorithm can perform operations based only on grayscale values without the need for RGB, and at the same time can improve the operation speed. In the image preprocessing of the input video image, the contrast is enhanced to make the image clear;

[0088] Traverse all the frames in the video. To find the movement between the current frame and the previous frame, an AI model is used to detect the corresponding relationship between adjacent frame images; the input of the AI model is the current frame and the previous frame. The AI model finds the corresponding relationship existing between two adjacent frames before and after, thereby establishing the motion relationship between adjacent frames. The AI model will predict the offset of the line coordinate points of the two images.

[0089] In an embodiment of the present invention, the image jitter correction module uses an AI model to confirm the mutual relationship between corresponding points in two grayscale images, thereby correcting the distortion caused by jitter between two adjacent video frames;

[0090] Considering the corresponding relationship between two planes between two frames, the AI model outputs several groups of offset corresponding points between vertices as the input required for subsequent solving of the unit matrix. Then, the direct linear transformation algorithm DLT (Direct Linear Transform) is used to obtain the homography matrix coefficients between adjacent frames;

[0091] Represent the homography matrix coefficients between adjacent frames as the current camera motion information, and multiply the homography matrix coefficients by the previous frame's motion trajectory to obtain the motion trajectory of the jittery video.

[0092] Use a digital filter to smooth the motion trajectory, and perform spatial image warping adjustment (Warping) on the original image according to the new motion trajectory to obtain a stable frame sequence.

[0093] In a usage scenario of the present invention, during the process of the camera motion parameter acquisition module estimating the camera's motion parameters, the scenes captured by adjacent images are on the same plane or two adjacent frames before and after are captured when the camera rotates simply. Calculate the homography matrix through four pairs of matching feature points extracted from the two images. According to the predefined corner points and offsets, obtain the position corresponding to the features of the current frame in the previous frame, and obtain the transformation parameters of the homography matrix H through the direct linear transformation algorithm; the definition of the homography matrix H is as shown in Formula 1:

[0094]

[0095] Among them, h11, h12, h21, h22 are rotation coefficients, and h13, h23 are translation coefficients; h31, h32, h33 can usually be 1.

[0096] Apply the homography matrix H to Equation 2 of the perspective transformation, where (x, y) represents the coordinates of the feature points in the current frame, and (x’, y’) represents the coordinates of the transformed feature points;

[0097]

[0098] The homography matrix coefficients between adjacent frames can represent the current camera motion information. Analyze the inter-frame motion to find the motion trajectory of the camera; Multiply the homography matrix coefficients with the previous frame's motion trajectory to obtain the motion trajectory of the jittery video;

[0099] During the process of the image jitter correction module performing image stabilization processing, in order to perform image stabilization, a filtering method is used to smooth the original motion trajectory. The Savitzky-Golay filtering algorithm (SG Filter) used is a digital signal processing algorithm for smoothing signals;

[0100] This algorithm uses the polynomial fitting method of partial least squares (PLS) to fit the local features of the curve, thereby achieving smoothing; SG Filter simultaneously achieves smoothing and denoising, reduces the noise of the data, and processes non-linear signals. This algorithm has a fast convergence speed and is suitable for real-time signal processing. Therefore, the present invention selects SG Filter to smooth the motion trajectory, and finally performs spatial image warping adjustment on the original image according to the new motion trajectory to obtain a stable frame sequence.

[0101] The present invention also discloses an electronic device, Figure 5 which is a schematic diagram of the composition of the electronic device in an embodiment of the present invention; Please refer to Figure 5 , at the hardware level, the electronic device includes a memory, a processor, and at least one network interface; the processor can be a microprocessor, and the memory can include internal memory, such as it can include random access memory (RAM), and can also include non-volatile memory, etc. Of course, the electronic device can also be provided with other hardware according to needs.

[0102] The processor, network interface, and memory can be interconnected through an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, an EISA (Extended Industry Standard Architecture) bus, etc.; the bus can include an address bus, a data bus, a control bus, etc. The memory is used to store programs (which can include an operating system program and application programs); the programs can include program codes, and the program codes can include computer operation instructions. The memory can include a memory and a non-volatile memory, and provide instructions and data to the processor.

[0103] In one embodiment, the processor can read the corresponding program from the non-volatile memory into the memory and then run it; the processor can execute the program stored in the memory and is specifically used to perform the following operations (as Figure 1 shown):

[0104]

Step S1

[0105]

Step S2

[0106] The present invention further discloses a storage medium, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the following steps of the method of the present invention are implemented (as Figure 1 shown):

[0107]

Step S1

[0108]

Step S2

[0109] The purpose of the electronic image stabilization method is to take a jittery video as input and output a video with relatively strong stability and less distortion. This invention uses deep learning methods to design a video stabilization algorithm and combines the Savitzky-Golay filtering method for motion trajectory smoothing. First, each frame of the video sequence is sequentially read from the video file. Except for the first frame, the previous frame and the current frame are read as grayscale images each time because the stabilization algorithm can perform operations based only on the grayscale values without the need for RGB, which can also improve the operation speed. In the second step, an AI model is used to confirm the relationship between corresponding points in the two grayscale images, so as to correct the distortion caused by jitter between the two frames. In this step, we consider the corresponding relationship between two planes between the two frames. The AI model will output the offset corresponding points between four sets of vertices as the input required for subsequent solving of the unit matrix. Then, the direct linear transformation algorithm DLT (Direct Linear Transform) can be used to obtain the homography matrix coefficients between adjacent frames. The homography matrix coefficients between adjacent frames represent the current camera motion information. Multiplying the homography matrix coefficients with the previous frame's motion trajectory can obtain the motion trajectory of the jittery video. The SG Filter is a digital filter that can smooth the motion trajectory. Finally, spatial image warping adjustment (Warping) is performed on the original image according to the new motion trajectory to obtain a stable frame sequence.

[0110] In summary, the video image stabilization control method, system, electronic device, and storage medium proposed by this invention can automatically perform stabilization processing on each frame image in the video sequence. This invention can perform real-time processing on electronic images and solve the problem of jitter during video shooting.

[0111] This invention proposes an electronic image stabilization algorithm for immediate video processing. This algorithm can be applied in an online real-time mode to automatically perform stabilization processing on each frame image in the video sequence to generate a smoother stabilized video and improve the viewing quality.

[0112] The electronic image stabilization method of this invention can stabilize videos without the assistance of specific equipment, with the advantages of low cost and more flexible operation, and can be widely applied in various fields.

[0113] The technical features of the above-described embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as within the scope described in this specification.

[0114] The description and application of the present invention herein are illustrative and are not intended to limit the scope of the present invention to the above embodiments. The effects or advantages involved in the embodiments may not be reflected in the embodiments due to various factors, and the description of the effects or advantages is not used to limit the embodiments. Modifications and changes to the disclosed embodiments are possible, and various components of substitution and equivalence of the embodiments are known to those of ordinary skill in the art. Those skilled in the art should clearly understand that the present invention can be implemented in other forms, structures, arrangements, proportions, and with other components, materials, and parts without departing from the spirit or essential characteristics of the present invention. Other modifications and changes can be made to the disclosed embodiments without departing from the scope and spirit of the present invention.

Claims

1. A video image stabilization control method, characterized in that, The video image stabilization control method includes: Step S1: Obtain the image sequence in the video image, and estimate the motion parameters of the camera according to the difference between two adjacent frames of the obtained image sequence; Step S2: Perform real-time filtering processing on the estimated motion parameters to eliminate local motion; perform motion compensation according to the detected motion trajectory to offset the camera shake, correct the image shake, and thus generate a stable video image.

2. The video image stabilization control method according to claim 1, wherein: In step S1, each frame of the video sequence is sequentially read from the video. Except for the first frame, the previous frame and the current frame are read as grayscale images each time; the contrast of the input video image is enhanced during image preprocessing to make the image clear; Traverse all frames in the video. To find the movement between the current frame and the previous frame, use an AI model to detect the correspondence between adjacent frame images; the input of the AI model is the current frame and the previous frame. The AI model searches for the correspondence existing between two adjacent frames before and after, thereby establishing the motion relationship between adjacent frames. The AI model will predict the offset of the two graph line coordinate points.

3. The video image stabilization control method according to claim 1, wherein: In step S2, use an AI model to confirm the mutual relationship between corresponding points in the two grayscale images, so as to correct the distortion caused by shake between two adjacent video frames; Considering the correspondence between two planes between two frames, the AI model outputs several groups of offset corresponding points between vertices as the input required for subsequent solving of the unit matrix. Then, use the direct linear transformation algorithm DLT to obtain the homography matrix coefficients between adjacent frames; Represent the homography matrix coefficients between adjacent frames as the current camera motion information, multiply the homography matrix coefficients by the motion trajectory of the previous frame, and obtain the motion trajectory of the jittery video. Use a digital filter to smooth the motion trajectory, and perform spatial image warping adjustment on the original image according to the new motion trajectory to obtain a stable frame sequence.

4. The video image stabilization control method according to claim 1, wherein: In step S1, the scenes captured by adjacent images are on the same plane or the two front and back frames of images are collected when the camera rotates simply. Calculate the homography matrix through four pairs of matching feature points extracted from the two images. According to the predefined corner points and offsets, obtain the position corresponding to the features of the current frame in the previous frame, and obtain the transformation parameters of the homography matrix H through the direct linear transformation algorithm; the definition of the homography matrix H is as shown in formula 1: where h11, h12, h21, h22 are rotation coefficients, and h13, h23 are translation coefficients; Apply the homography matrix H to formula 2 for perspective transformation, where (x, y) represents the coordinate of the feature point of the current frame, and (x’, y’) represents the coordinate of the transformed feature point; The homography matrix coefficients between adjacent frames can represent the current camera motion information. Analyze the inter-frame motion situation to find the motion trajectory of the camera; multiply the homography matrix coefficients by the motion trajectory of the previous frame to obtain the motion trajectory of the jittery video; In step S2, in order to perform image stabilization, a filtering method is used to smooth the original motion trajectory. The filtering algorithm used is a digital signal processing algorithm for smoothing signals. This algorithm uses the polynomial fitting method of partial least squares to fit the local features of the curve, thereby achieving smoothing; the SG Filter simultaneously achieves smoothing and denoising, reduces the noise of the data, and processes non-linear signals; the original image is adjusted for spatial image distortion according to the new motion trajectory to obtain a stable frame sequence.

5. A video image stabilization control system, characterized in that, The video image stabilization control system includes: A camera motion parameter acquisition module for acquiring an image sequence in a video image and estimating the motion parameters of the camera according to the difference between two adjacent frames of the acquired image sequence. An image jitter correction module for performing real-time filtering on the motion parameters estimated by the camera motion parameter acquisition module to eliminate local motion; performing motion compensation according to the detected motion trajectory to offset the jitter of the camera and correct the image jitter, thereby generating a stable video image.

6. The video image stabilization control system according to claim 5, wherein: The camera motion parameter acquisition module sequentially reads each frame of the video sequence from the video. Except for the first frame, the previous frame and the current frame are read as grayscale images each time; the contrast of the input video image is enhanced during image preprocessing to make the image clear. Traverse all frames in the video. To find the movement between the current frame and the previous frame, an AI model is used to detect the correspondence between adjacent frame images; the input of the AI model is the current frame and the previous frame. The AI model finds the correspondence existing between two adjacent frames before and after, thereby establishing the motion relationship between adjacent frames. The AI model will predict the offset of the line coordinate points of the two images.

7. The video image stabilization control system according to claim 5, wherein: The image jitter correction module uses an AI model to confirm the mutual relationship between corresponding points in two grayscale images, thereby correcting the distortion caused by jitter between two adjacent video frames. Considering the correspondence between two planes between two frames, the AI model outputs several groups of offset corresponding points between vertices as the input required for subsequent solving of the unit matrix. Then, the direct linear transformation algorithm DLT is used to obtain the homography matrix coefficients between adjacent frames. The homography matrix coefficients between adjacent frames represent the current camera motion information. The homography matrix coefficients are multiplied by the motion trajectory of the previous frame to obtain the motion trajectory of the jittery video. A digital filter is used to smooth the motion trajectory, and the original image is adjusted for spatial image distortion according to the new motion trajectory to obtain a stable frame sequence.

8. The video image stabilization control system according to claim 5, wherein: During the process of the camera motion parameter acquisition module estimating the motion parameters of the camera, the scenes captured by adjacent images are on the same plane, or the front and back two frames of images are captured when the camera rotates simply. The homography matrix is calculated through four pairs of matching feature points extracted from the two images. According to the predefined corner points and offsets, the position corresponding to the features of the current frame in the previous frame is obtained, and the transformation parameters of the homography matrix H are obtained through the direct linear transformation algorithm; the definition of the homography matrix H is as shown in Formula 1: Where h11, h12, h21, h22 are rotation coefficients, and h13, h23 are translation coefficients; Apply the homography matrix H to Formula 2 of perspective transformation, where (x, y) represents the coordinates of the feature points of the current frame, and (x’, y’) represents the coordinates of the transformed feature points; The homography matrix coefficients between adjacent frames can represent the current camera motion information. Analyze the inter-frame motion situation to find the motion trajectory of the camera; multiply the homography matrix coefficients by the previous frame's motion trajectory to obtain the motion trajectory of the jittery video; During the process of the image jitter correction module performing image stabilization processing, in order to perform image stabilization, a filtering algorithm is used to smooth the original motion trajectory. The filtering algorithm used is a digital signal processing algorithm for smoothing signals; This algorithm uses the polynomial fitting method of partial least squares to fit the local features of the curve, thereby achieving smoothing processing; the filtering algorithm simultaneously achieves smoothing and denoising, reduces the noise of the data, and processes non-linear signals; Adjust the spatial image distortion of the original image according to the new motion trajectory to obtain a stable frame sequence.

9. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 4.

10. A storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, it implements the steps of the method according to any one of claims 1 to 4.

Images