Video processing method and related device

Abstract

The invention discloses a video processing method and a related device, and relates to the technical field of video processing, and the method comprises the steps: sequentially obtaining original video frames in an original video stream collected by a camera device; and after the first original video frame is obtained, obtaining a motion track of at least one effective sampling point in the first original video frame. And based on the motion trail of the at least one effective sampling point, determining the equipment motion trail of the camera equipment. And obtaining a reverse compensation parameter for the first original video frame based on the equipment motion trail. And compensating each pixel in the first original video frame based on the reverse compensation parameter to obtain a first target video frame.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a video processing method and related apparatus. Background Technology

[0002] To ensure stable and clear video footage, optical image stabilization is usually used to actively counteract camera shake and correct the image. Optical image stabilization is a hardware function that requires the camera equipment to have the corresponding physical components to achieve, and its effect is poor. Summary of the Invention

[0003] In view of the above problems, this application provides a video processing method and related apparatus, the specific solution of which is as follows:

[0004] The first aspect of this application provides a video processing method, including:

[0005] Obtain the original video frames from the original video stream captured by the camera device in sequence;

[0006] After obtaining the first original video frame, the motion trajectory of at least one valid sampling point in the first original video frame is obtained;

[0007] Based on the motion trajectory of the at least one valid sampling point, the device motion trajectory of the camera is determined;

[0008] Based on the device's motion trajectory, inverse compensation parameters for the first original video frame are obtained;

[0009] Based on the inverse compensation parameters, each pixel in the first original video frame is compensated to obtain the first target video frame.

[0010] In one possible implementation, obtaining the motion trajectory of at least one valid sampling point in the first original video frame includes:

[0011] If the first original video frame is not the first video frame in the first original video frame sequence, the pyramid optical flow algorithm is used to trace the effective sampling points of the preceding original video frames in the first original video frame to obtain the inter-frame offset vector of at least one candidate sampling point; wherein, the first original video frame sequence is any original video frame sequence obtained by dividing the original video stream according to the first duration.

[0012] Based on the inter-frame offset vector of each of the candidate sampling points, at least one valid sampling point is determined from the at least one candidate sampling point;

[0013] Based on the inter-frame offset vector of the at least one valid sampling point, the motion trajectory of the at least one valid sampling point is obtained.

[0014] In one possible implementation, the motion trajectory of the at least one valid sampling point is obtained based on the inter-frame offset vector of the at least one valid sampling point, including:

[0015] If the number of valid sampling points is determined to be not less than a first quantity threshold, the motion trajectory of the at least one valid sampling point is obtained based on the inter-frame offset vector of the at least one valid sampling point.

[0016] One possible implementation also includes:

[0017] If the first original video frame is the first video frame, or the number of valid sampling points is less than the first number threshold, corner detection is performed on the first original video frame to determine multiple key points and obtain the valid sampling points of the first original video frame.

[0018] In one possible implementation, determining the at least one valid sampling point from the at least one candidate sampling point based on the inter-frame offset vector of each of the candidate sampling points includes:

[0019] Based on the inter-frame offset vector of multiple edge reference points, determine whether camera device movement has occurred;

[0020] If so, for each candidate sampling point, determine whether the length of the motion trajectory of the candidate sampling point is consistent with the length of the motion trajectory of the edge reference point;

[0021] If so, the candidate sampling point is determined as a valid sampling point.

[0022] In one possible implementation, determining the device motion trajectory of the camera equipment based on the motion trajectory of the at least one valid sampling point includes:

[0023] Based on the motion trajectory of the at least one valid sampling point, the inter-frame motion parameters of the camera device between the first original video frame and the preceding original video frame are obtained.

[0024] The device motion trajectory of the camera is updated based on the inter-frame motion parameters.

[0025] In one possible implementation, based on the device motion trajectory, inverse compensation parameters for the first original video frame are obtained, including:

[0026] Based on the motion trajectory of the device, the shaking characteristics of the camera device within a first time window are obtained, and the shaking characteristics include shaking frequency and shaking amplitude;

[0027] If the jitter feature triggers any one of the compensation conditions, based on the device motion trajectory, the inter-frame motion parameters of the camera device between the first original video frame and the preceding original video frame are obtained; wherein, each of the compensation conditions is used to limit the numerical range of the jitter frequency and / or the jitter amplitude;

[0028] The inverse compensation parameters are obtained by inverting the inter-frame motion parameters.

[0029] In one possible implementation, based on the inverse compensation parameters, each pixel in the first original video frame is compensated to obtain the first target video frame, including:

[0030] Each pixel in the first original video frame is moved according to the offset and offset direction indicated by the inverse compensation parameters to obtain the first target video frame.

[0031] One possible implementation also includes:

[0032] Identify the pixel missing regions in the first target video frame and the effective pixel boundaries adjacent to the pixel missing regions;

[0033] Using the pixels on the effective pixel boundary as a reference, interpolation is performed to obtain the filling pixel sequence;

[0034] The missing pixel region is filled with the filling pixel sequence along a direction perpendicular to the effective pixel boundary and pointing towards the missing pixel region to obtain the target output video frame.

[0035] A second aspect of this application provides a video processing apparatus, comprising:

[0036] The video frame acquisition unit is used to sequentially acquire the original video frames in the original video stream captured by the camera device;

[0037] The sampling point trajectory tracking unit is used to obtain the motion trajectory of at least one valid sampling point in the first original video frame after obtaining the first original video frame.

[0038] The device motion detection unit is used to determine the device motion trajectory of the camera device based on the motion trajectory of the at least one valid sampling point;

[0039] The compensation parameter determination unit is used to obtain inverse compensation parameters for the first original video frame based on the device motion trajectory.

[0040] The jitter compensation unit is used to compensate each pixel in the first original video frame based on the inverse compensation parameters to obtain the first target video frame.

[0041] Using the above technical solutions, the video processing method and related apparatus provided in this application embodiment sequentially obtain the original video frames in the original video stream captured by the camera device; after obtaining the first original video frame, obtain the motion trajectory of at least one valid sampling point in the first original video frame; based on the motion trajectory of at least one valid sampling point, determine the device motion trajectory of the camera device; based on the device motion trajectory, obtain the inverse compensation parameters for the first original video frame; based on the inverse compensation parameters, compensate each pixel in the first original video frame to obtain the first target video frame. Attached Figure Description

[0042] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0043] Figure 1 A flowchart illustrating a video processing method provided in this application;

[0044] Figure 2 A flowchart illustrating yet another video processing method provided in this application;

[0045] Figure 3 A flowchart illustrating yet another video processing method provided in this application;

[0046] Figure 4 A flowchart illustrating yet another video processing method provided in this application;

[0047] Figure 5 A flowchart illustrating yet another video processing method provided in this application;

[0048] Figure 6 A flowchart illustrating yet another video processing method provided in this application;

[0049] Figure 7 A schematic diagram illustrating a specific implementation of a video processing method provided in this application;

[0050] Figure 8 A schematic diagram of the structure of a video processing device provided in this application;

[0051] Figure 9 This is a schematic diagram of the structure of an electronic device provided in this application. Detailed Implementation

[0052] The embodiments of this application are described below with reference to the accompanying drawings. The terminology used in the implementation section of this application is for explaining specific embodiments only and is not intended to limit the scope of this application.

[0053] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0054] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar elements and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing elements with the same properties in the description of embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of units is not necessarily limited to those units, but may include other units not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0055] Reference Figure 1 , Figure 1 This is a flowchart illustrating a video processing method provided in an embodiment of this application, as shown below. Figure 1 As shown in the embodiment of this application, a video processing method may include steps 101 to 105, which are described in detail below.

[0056] Step 101: Sequentially obtain the original video frames from the original video stream captured by the camera device.

[0057] In this embodiment, the original video stream is a collection of original video frames acquired by the camera device and arranged in chronological order. This step sequentially acquires each original video frame according to the time sequence of the original video frame in the original video stream, and then performs image stabilization processing on each original video frame to ensure the continuity of the time sequence.

[0058] In one alternative embodiment, the original video stream is a video stream captured by a camera device such as a mobile phone, action camera, or drone aerial photography, which may contain shaky footage.

[0059] Step 102: After obtaining the first original video frame, obtain the motion trajectory of at least one valid sampling point in the first original video frame.

[0060] In this embodiment, effective sampling points are feature points in consecutive video frames of the original video stream that are easy to track and can reflect scene motion, such as corner points, high-contrast edge points, or feature points in specific texture regions. The motion trajectory of each effective sampling point refers to the ordered set of positions of the effective sampling points over a continuous time period, reflecting the positional changes of the effective sampling point in consecutive video frames.

[0061] In this embodiment, the apparent motion of the scene is quantified by obtaining the motion trajectory of at least one valid sampling point in the first original video frame. For example, optical flow and feature matching algorithms can be combined to track the movement of multiple valid sampling points in consecutive video frames, thereby forming the motion trajectory of each valid sampling point.

[0062] Step 103: Determine the device motion trajectory of the camera equipment based on the motion trajectory of at least one valid sampling point.

[0063] In this embodiment, the motion trajectory of the camera device itself, i.e., the device motion trajectory, is determined by combining the motion trajectory of at least one valid sampling point. It should be noted that the device motion trajectory refers to the frame motion trajectory of the video stream caused by the movement of the camera device (e.g., rotation and translation) after excluding the independent movement of objects in the scene.

[0064] In one alternative embodiment, by analyzing the consistency of the motion trajectories of multiple valid sampling points, the motion parameters of the camera device itself can be deduced, thereby fitting the device motion trajectory in consecutive video frames.

[0065] Step 104: Based on the device motion trajectory, obtain the inverse compensation parameters for the first original video frame.

[0066] In this embodiment, the reverse compensation parameters are a set of geometric transformation parameters used to counteract camera shake. Specifically, the reverse compensation parameters cancel out the device shake parameters indicated by the device motion trajectory. For example, the compensation direction in the reverse compensation parameters is opposite to the device offset direction, and the compensation value is equal to the device offset.

[0067] For example, based on the device motion trajectory corresponding to the first original video frame, it is determined that the camera device shifted 8 pixels to the right when capturing the first original video frame. Then, the inverse compensation parameter for the first original video frame is the geometric transformation parameter for shifting 8 pixels to the left.

[0068] Step 105: Based on the inverse compensation parameters, compensate each pixel in the first original video frame to obtain the first target video frame.

[0069] In this embodiment, the target position of each pixel in the first original video frame is calculated based on the inverse compensation parameters, thereby transforming the entire first original video frame to generate the first target video frame.

[0070] As can be seen from the above technical solution, the video processing method provided in this application sequentially obtains the original video frames in the original video stream captured by the camera device. After obtaining the first original video frame, the motion trajectory of at least one valid sampling point in the first original video frame is obtained. Based on the motion trajectory of at least one valid sampling point, the device motion trajectory of the camera device is determined. Based on the device motion trajectory, inverse compensation parameters for the first original video frame are obtained. Based on the inverse compensation parameters, each pixel in the first original video frame is compensated to obtain the first target video frame. By tracking the valid sampling points in the original video frame, the motion trajectory of the camera device is obtained, and inverse compensation is performed on each video frame according to the device's running trajectory. This can eliminate various types of shaking caused by the movement of the shooting device, thereby improving the image stabilization effect.

[0071] Furthermore, compared to optical image stabilization, this application does not rely on specific physical components and can be widely used in mainstream camera devices that do not have optical image stabilization hardware.

[0072] Based on the above embodiments, see Figure 2 , Figure 2 This application provides yet another video processing method. Figure 2 The following illustrates the specific implementation process of step 102, namely obtaining the motion trajectory of at least one valid sampling point in the first original video frame, as follows: Figure 2 As shown in the embodiment of this application, a video processing method may include steps 201 to 203, which are described in detail below.

[0073] Step 201: If the first original video frame is not the first video frame in the sequence of first original video frames, the pyramid optical flow algorithm is used to trace the effective sampling points of the preceding original video frames in the first original video frame to obtain the inter-frame offset vector of at least one candidate sampling point.

[0074] In this embodiment, the preceding original video frame refers to the video frame preceding the first original video frame, and the first original video frame sequence is any original video frame sequence obtained by dividing the original video stream according to the first duration. Optionally, each time, starting from the first frame, original video frames within the original video stream of the target duration are received frame by frame to obtain an original video frame sequence. The first frame can be the first frame after the end of the previous original video frame sequence, or it can be an original video frame in the original video frame sequence where the number of valid sampling points is less than a threshold.

[0075] In this embodiment, the pyramid optical flow algorithm, a pixel tracking algorithm combining pyramid and optical flow algorithms, is used to estimate the motion of each pixel in the original video frame sequence. By constructing different resolution levels of the image and performing optical flow calculations based on the assumptions of constant local window brightness and smooth motion, robust tracking of large displacement motions is achieved.

[0076] In this embodiment, continuous tracking is performed using existing valid sampling point information from the previous video frame, provided the current frame is not the first frame. Specifically, using the pyramid LK optical flow method, even with significant inter-frame jitter, the new positions of valid sampling points in the current frame can be effectively tracked, thereby obtaining a set of candidate sampling points and their corresponding offset vectors.

[0077] Step 202: Based on the inter-frame offset vector of each candidate sampling point, determine at least one valid sampling point from at least one candidate sampling point.

[0078] In this embodiment, reliable feature points representing the rigid motion of the camera device are separated from the candidate sampling points, which are also the valid sampling points of the current frame. Specifically, this is based on the inter-frame offset vectors of each candidate sampling point. In an optional embodiment, motion consistency analysis is performed on the inter-frame offset vectors of each candidate sampling point to obtain candidate sampling points with consistency, which are then used as valid sampling points.

[0079] Step 203: Based on the inter-frame offset vector of at least one valid sampling point, obtain the motion trajectory of at least one valid sampling point.

[0080] In this embodiment, the real-time motion trajectory is updated by connecting the inter-frame offset vector of the effective sampling points calculated in the first original video frame with the historical trajectory formed by the effective sampling points in the previous frame.

[0081] As can be seen from the above technical solutions, the video processing method provided in this application introduces the Pyramid LK optical flow algorithm to achieve robust tracking of large displacements of feature points in non-first frame video frames, obtaining the inter-frame offset vectors of candidate sampling points. Furthermore, by separating effective sampling points representing the rigid motion of the camera device from the candidate points based on motion consistency, interference from independently moving objects within the scene is effectively filtered out. Finally, by integrating the inter-frame offset vectors of the effective sampling points, a device motion trajectory that accurately reflects the device's motion is constructed, thereby achieving accurate tracking of the device's motion.

[0082] In one optional embodiment, if the first original video frame is the first video frame in the first original video frame sequence, or the number of effective sampling points is less than a first quantity threshold, corner detection is performed on the first original video frame to determine multiple key points and obtain the effective sampling points of the first original video frame.

[0083] In this embodiment, when the number of valid sampling points is less than a first threshold, it is determined that the frame has changed significantly, and valid point tracking fails. Therefore, the first original video frame is taken as the first video frame, and the original video frame sequence is re-divided to obtain the second original video frame sequence. It can be understood that the first original video frame is either the first frame in the first original video frame sequence, or, when the number of valid sampling points is less than the first threshold, the first original video frame is always the first video frame. Therefore, corner detection is performed on the first video frame to determine multiple key points, thus obtaining the valid sampling points of the first original video frame.

[0084] In this embodiment, the effective sampling points of the first video frame in each original video frame sequence include all key points, serving as the starting point for effective point tracking.

[0085] Based on this, the specific execution method of step 203 is as follows: if the number of valid sampling points is not less than a first quantity threshold, the motion trajectory of at least one valid sampling point is obtained based on the inter-frame offset vector of at least one valid sampling point. That is, when the number of valid sampling points is not less than the first quantity threshold, the motion trajectory of the valid sampling points is determined, thereby improving the effectiveness and reliability of trajectory tracking.

[0086] Based on the above embodiments, see Figure 3 , Figure 3 This application provides yet another video processing method. Figure 3 The following illustrates the specific implementation process of step 202, which involves determining at least one valid sampling point from at least one candidate sampling point based on the inter-frame offset vector of each candidate sampling point. Figure 3 As shown in the embodiment of this application, a video processing method may include steps 301 to 303, which are described in detail below.

[0087] Step 301: Determine whether camera device movement has occurred based on the inter-frame offset vectors of multiple edge reference points.

[0088] In this embodiment, edge reference points refer to feature points located at the edge of a video frame. Generally, edge reference points belong to the static background, such as distant buildings, the sky, or the ground. Therefore, the inter-frame offset vector of edge reference points is usually caused by the movement of the camera device.

[0089] In this embodiment, based on the inter-frame offset vector of multiple edge reference points, it is determined whether there is frame displacement caused by the movement of the camera device in the current inter-frame.

[0090] In one optional embodiment, a jitter threshold is determined based on factors such as the type of camera equipment, the shooting scene, and stability requirements. The existence of significant global image motion is determined by calculating the average offset of multiple edge reference points or checking the consistency of their motion directions. Optionally, if the average offset is less than the jitter threshold or the motion directions are inconsistent, it is considered that the camera equipment only has minor disturbances; if the average offset is not less than the jitter threshold and the motion directions are consistent, then camera equipment motion is determined to have occurred. This avoids unnecessary image stabilization processing on video frames without significant jitter.

[0091] Step 302: If yes, for each candidate sampling point, determine whether the length of the motion trajectory of the candidate sampling point is consistent with the length of the motion trajectory of the edge reference point.

[0092] In this embodiment, the motion trajectory length of the candidate sampling point refers to the cumulative magnitude of the offset vector of the candidate sampling point within a certain time window, that is, from the first video frame to the current frame, reflecting the total offset of the candidate sampling point accumulated over multiple video frames.

[0093] In this embodiment, if the difference between the trajectory length of the candidate sampling point and the trajectory length of the edge reference point is within the error range, they can be considered consistent. It is understood that within the same time window, the cumulative displacement of feature points driven by the rigid motion of the camera device should be consistent.

[0094] Step 303: If yes, determine the candidate sampling point as a valid sampling point.

[0095] In this embodiment, if the length of the motion trajectory of the candidate sampling point is consistent with the length of the motion trajectory of the edge reference point, it indicates that the candidate sampling point has generated the same motion as the background at the edge of the image, that is, it passes the motion consistency test.

[0096] As can be seen from the above technical solutions, the video processing method provided in this application first uses edge reference points as reference points for frame motion to determine whether there is significant device shake. Then, by comparing the consistency of the cumulative offset between each candidate sampling point and the edge reference point, it distinguishes feature points driven by rigid motion of the camera device from feature points that move independently within the scene. Finally, it selects a set of effective sampling points that can truly reflect device motion, improving the robustness and compensation effect of the image stabilization algorithm for complex dynamic scenes.

[0097] Based on the above embodiments, see Figure 4 , Figure 4 This application provides yet another video processing method. Figure 4 The following illustrates the specific implementation process of step 103, which is determining the device motion trajectory of the camera based on the motion trajectory of at least one valid sampling point: Figure 4 As shown in the embodiment of this application, a video processing method may include steps 401 to 402, which are described in detail below.

[0098] Step 401: Based on the motion trajectory of at least one valid sampling point, obtain the inter-frame motion parameters of the camera device between the first original video frame and the preceding original video frame.

[0099] In this embodiment, the inter-frame motion parameters are a set of geometric transformation parameters used to describe the spatial motion of the camera device from the previous frame (previous original video frame) to the current frame (first original video frame).

[0100] In this embodiment, based on the motion trajectory of at least one valid sampling point, a motion estimation algorithm is used to fit a transformation that best explains the overall offset of all valid sampling points. For example, the least squares method is used to solve for the optimal global motion model, thereby using inter-frame motion parameters to quantify the jitter amplitude and direction of the camera device between two frames.

[0101] Step 402: Update the device motion trajectory of the camera equipment based on the inter-frame motion parameters.

[0102] In this embodiment, a global, smooth device motion trajectory is constructed based on inter-frame motion parameters. Specifically, the inter-frame motion parameters of the current frame are combined or superimposed with the set of inter-frame motion parameters corresponding to historical video frames, thereby extending the device motion trajectory and obtaining a complete motion path from the first video frame to the current video frame.

[0103] Understandably, the device motion trajectory describes the entire motion process of the camera device from the first video frame to the current moment.

[0104] Based on the above embodiments, see Figure 5 , Figure 5 This application provides yet another video processing method. Figure 5 The following illustrates the specific implementation process of step 104, namely, obtaining the inverse compensation parameters for the first original video frame based on the device motion trajectory. Figure 5 As shown in the embodiment of this application, a video processing method may include steps 501 to 503, which are described in detail below.

[0105] Step 501: Based on the device motion trajectory, obtain the shaking characteristics of the camera device within the first time window.

[0106] In this embodiment, the jitter characteristics include jitter frequency and jitter amplitude. The first time window refers to the time window from the first video frame corresponding to the current first original video frame to the first original video frame. The jitter frequency reflects the speed of jitter and can be estimated by performing time-frequency analysis on the device's motion trajectory to distinguish between high-frequency jitter and low-frequency shaking. The jitter amplitude reflects the severity of jitter and can be obtained by calculating statistical quantities such as the maximum value, average value, or variance of the device's motion trajectory within the first time window.

[0107] Step 502: If the jitter feature triggers any compensation condition, based on the device motion trajectory, obtain the inter-frame motion parameters of the camera device between the first original video frame and the preceding original video frame.

[0108] In this embodiment, each compensation condition is used to limit the numerical range of jitter frequency and / or jitter amplitude.

[0109] In this embodiment, jitter compensation is triggered by determining whether the jitter frequency and jitter amplitude fall within the numerical range defined by the compensation conditions.

[0110] For example, the first compensation condition limits the jitter frequency range to a high-frequency range and the jitter amplitude range to a low-amplitude range, while the second compensation condition limits the jitter frequency range to a low-frequency range and the jitter amplitude range to a high-amplitude range. Therefore, when the jitter feature triggers the first compensation condition, it indicates that the current camera device is experiencing high-frequency, low-amplitude jitter; when the jitter feature triggers the second compensation condition, it indicates that the current camera device is experiencing low-frequency, high-amplitude jitter. When the jitter feature triggers either the first or the second compensation condition, jitter compensation is determined to be triggered.

[0111] Step 503: Invert the inter-frame motion parameters to obtain the inverse compensation parameters.

[0112] In this embodiment, inverting the inter-frame motion parameters means performing an inverse matrix transformation on the transformation matrix describing the rigid body motion. For parameters such as translation and rotation, the inverse values ​​are simply taken.

[0113] As can be seen from the above technical solutions, the video processing method provided in this application extracts features such as jitter frequency and jitter amplitude from the device motion trajectory, thereby achieving intelligent identification and quantitative evaluation of jitter types. Furthermore, based on preset compensation conditions, decisions are made only when jitter features significantly affect video stability (such as high-frequency small jitter or low-frequency large jitter), triggering inverse compensation and avoiding unnecessary computational overhead and overcorrection. Finally, by inverting the inter-frame motion parameters, inverse compensation parameters are generated to offset the device jitter corresponding to the first original video frame, thereby canceling or reducing the visual impact of jitter on the screen and achieving a highly efficient and adaptive anti-shake effect.

[0114] In an optional embodiment, step 105, namely, compensating each pixel in the first original video frame based on the reverse compensation parameters to obtain the first target video frame, is specifically implemented as follows: each pixel in the first original video frame is moved according to the offset amount and offset direction indicated by the reverse compensation parameters to obtain the first target video frame.

[0115] In an optional embodiment, this application further includes a cropping and / or padding step for the target video frame, see [link to relevant documentation]. Figure 6 , Figure 6 This application provides yet another video processing method. Figure 6 The specific implementation flow of the cropping and / or padding steps for the target video frame is shown, such as... Figure 6 As shown in the embodiment of this application, a video processing method may include steps 601 to 603, which are described in detail below.

[0116] Step 601: Identify the pixel missing regions in the first target video frame and the effective pixel boundaries adjacent to the pixel missing regions.

[0117] In this embodiment, the pixel missing area is the area in the first target video frame that has no valid image data after reverse compensation, where the original image content is moved out of the frame. It is usually displayed as black or a specified solid color, also known as a black border.

[0118] In this embodiment, the effective pixel boundary refers to the edge contour line formed by the pixels (effective image data) adjacent to the pixel missing area, that is, the dividing line between the content screen and the empty area without content.

[0119] Step 602: Using the pixels on the effective pixel boundary as a reference, interpolate to obtain the filling pixel sequence.

[0120] In this embodiment, a fill pixel sequence is created based on edge information, that is, pixels on the effective pixel boundary, to fill the missing region along a specific direction.

[0121] In this embodiment, a naturally filled pixel sequence is achieved by interpolating based on pixels on the effective pixel boundary. For example, taking the color and texture gradient of multiple adjacent pixels on the boundary as a reference, the effective pixel boundary is used as a reference to extend outward, and a new pixel value is generated by interpolation algorithm to obtain the filled pixel sequence.

[0122] Step 603: Fill the missing pixel region with the filling pixel sequence along the direction perpendicular to the effective pixel boundary and pointing to the missing pixel region to obtain the target output video frame.

[0123] In this embodiment, the filling direction is along the direction perpendicular to the effective pixel boundary and pointing to the pixel missing area. That is, all newly generated pixels extend radially from the existing image edge directly out of the image frame.

[0124] In this embodiment, the filling pixel sequence is assigned to the corresponding positions in the pixel-missing region according to the filling direction described above, to obtain the target output video frame.

[0125] Therefore, after filling and repair, the final output video frame, which eliminates black borders and has a complete and distortion-free image, is the target output video frame.

[0126] As can be seen from the above technical solutions, the video processing method provided in this application identifies the pixel missing regions and their effective pixel boundaries after compensation, performs interpolation operations based on the pixel information of the effective pixel boundaries, and generates a visually coherent and natural sequence of filled content. Finally, the pixel missing regions are filled along a direction perpendicular to the effective pixel boundaries and pointing towards the pixel missing regions, thereby effectively eliminating visual defects such as "black borders" without cropping and while maintaining the original resolution, ultimately outputting high-quality, stable video frames with complete images and smooth transitions.

[0127] It should be noted that, in another optional embodiment, to improve the efficiency of video frame restoration, the effective pixel boundaries can be directly copied and extended to fill the missing pixel areas, resulting in the target output video frame. After filling, if there are redundant areas (pixel areas with unnatural transitions at the frame edges), these redundant areas can be cropped. Alternatively, under certain conditions, such as when the missing pixel area is narrow, it can be directly cropped.

[0128] In conclusion, Figure 7 This application provides a specific implementation of a video processing method. Figure 7 This example illustrates the specific workflow framework for implementing real-time video stream stabilization in a live streaming scenario, such as... Figure 7 As shown, a specific implementation process of this application includes the following four stages:

[0129] I. Data Inflow and Preprocessing Stage.

[0130] like Figure 7As shown, the raw video signal acquired by the camera equipment is transmitted through a broadcast system and input to the video processing system via a hardware interface (such as SDI) or a streaming media transmission protocol (such as NDI / SRT / RTSP / RTMP). A stream adapter adapts the raw video signal from the streaming media transmission path, while a hardware input adapter adapts the raw video signal from the hardware interface path, resulting in the raw video stream. A video decoder then decodes the raw video stream to obtain the raw video frame sequence, which is then sent to the core image stabilization module.

[0131] II. Anti-shake processing stage for the original video frame sequence.

[0132] like Figure 7 As shown, the core image stabilization module tracks effective sampling points in video frames and calculates their inter-frame offset vectors through a frame offset calculation module. Based on the motion trajectories of the effective sampling points, the offset path calculation module calculates and integrates the camera's motion trajectory. The offset frequency calculation module analyzes the device's motion trajectory, extracting jitter features such as frequency and amplitude. Based on the calculated device motion trajectory and jitter features, the 3-axis stabilization module generates inverse compensation parameters and performs pixel-level compensation on the video frames, achieving electronic image stabilization and outputting a stable target video frame sequence. The edge cropping module identifies potential black borders after compensation and ensures a complete target video frame sequence output through padding and / or cropping. The image output module outputs the target video frame sequence.

[0133] In one optional embodiment, combining the above embodiments, the following is a specific implementation process of the image stabilization stage:

[0134] Receive the raw video frame sequence. Each batch of raw video frame sequences has a length of t seconds and contains N consecutive raw video frames. Process frame by frame, starting from the first frame (i.e., the first video frame):

[0135] For the first frame: M1 key points are obtained through corner detection, where each key point may include n edge pixels.

[0136] For the second frame and every subsequent frame:

[0137] The pyramid optical flow method is used to track M1 effective sampling points (for the second frame, the effective sampling points are the key points in the first frame) to obtain the offset vectors of the tracked M2 candidate sampling points. Then, based on the offset vectors of the edge pixels, the frame offset and offset consistency are judged. M3 effective sampling points are obtained from the M2 candidate sampling points, and the offset caused by the object's own movement, such as the sampling point offset caused by the rolling of a football and the running of a player, is filtered out.

[0138] The motion trajectory of the camera device is determined based on the motion trajectory of the effective sampling points, and the jitter characteristics, including amplitude and frequency, are determined based on the motion trajectory. Furthermore, jitter compensation is performed when the jitter characteristics meet the compensation conditions to obtain the compensated target video frame.

[0139] Furthermore, the target video frame has black border areas, which are areas with missing pixels. Therefore, the target video frame is repaired to remove the missing pixels, resulting in the output video frame.

[0140] Perform the above operation on each frame until the last video frame of the original video frame sequence, or until the number of valid sampling points is less than the sampling point threshold, then reread the next original video frame sequence.

[0141] III. Live Signal Output Stage.

[0142] like Figure 7 As shown, the target video frame sequence after image stabilization is re-encoded by the frame encoder to generate the target video stream, which is then output to the live broadcast terminal via the broadcasting system through the streaming media protocol.

[0143] IV. Stability assessment and monitoring phase.

[0144] like Figure 7 As shown, the stability assessment module analyzes the stabilization effect of the core stabilization module based on the target video frame sequence, generates stability assessment data, and transmits the target video frame sequence and stability assessment data to the Web console monitor via HTTPS or RTC. The Web console monitor displays the data visually, facilitating real-time monitoring of the system's processing effect by maintenance personnel.

[0145] The above describes a video processing method provided by an embodiment of this application. The following describes the apparatus for performing the above video processing method.

[0146] Please see Figure 8 , Figure 8 This is a schematic diagram of the structure of a video processing apparatus provided in an embodiment of this application. Figure 8 As shown, the video processing apparatus 800 includes:

[0147] The video frame acquisition unit 801 is used to sequentially acquire the original video frames in the original video stream captured by the camera device;

[0148] The sampling point trajectory tracking unit 802 is used to obtain the motion trajectory of at least one valid sampling point in the first original video frame after obtaining the first original video frame.

[0149] The device motion detection unit 803 is used to determine the device motion trajectory of the camera device based on the motion trajectory of the at least one valid sampling point;

[0150] The compensation parameter determination unit 804 is used to obtain inverse compensation parameters for the first original video frame based on the device motion trajectory.

[0151] The jitter compensation unit 805 is used to compensate each pixel in the first original video frame based on the inverse compensation parameters to obtain the first target video frame.

[0152] In one possible implementation, when the sampling point trajectory tracking unit is used to obtain the motion trajectory of at least one valid sampling point in the first original video frame, it is specifically used for:

[0153] If the first original video frame is not the first video frame in the first original video frame sequence, the pyramid optical flow algorithm is used to trace the effective sampling points of the preceding original video frames in the first original video frame to obtain the inter-frame offset vector of at least one candidate sampling point; wherein, the first original video frame sequence is any original video frame sequence obtained by dividing the original video stream according to the first duration.

[0154] Based on the inter-frame offset vector of each of the candidate sampling points, at least one valid sampling point is determined from the at least one candidate sampling point;

[0155] Based on the inter-frame offset vector of the at least one valid sampling point, the motion trajectory of the at least one valid sampling point is obtained.

[0156] In one possible implementation, when the sampling point trajectory tracking unit obtains the motion trajectory of the at least one valid sampling point based on the inter-frame offset vector of the at least one valid sampling point, it is specifically used for:

[0157] If the number of valid sampling points is determined to be not less than a first quantity threshold, the motion trajectory of the at least one valid sampling point is obtained based on the inter-frame offset vector of the at least one valid sampling point.

[0158] In one possible implementation, the sampling point trajectory tracking unit is also used for:

[0159] If the first original video frame is the first video frame, or the number of valid sampling points is less than the first number threshold, corner detection is performed on the first original video frame to determine multiple key points and obtain the valid sampling points of the first original video frame.

[0160] In one possible implementation, when the sampling point trajectory tracking unit determines the at least one valid sample from the at least one candidate sampling point based on the inter-frame offset vector of each of the candidate sampling points, it is specifically used for:

[0161] Based on the inter-frame offset vector of multiple edge reference points, determine whether camera device movement has occurred;

[0162] If so, for each candidate sampling point, determine whether the length of the motion trajectory of the candidate sampling point is consistent with the length of the motion trajectory of the edge reference point;

[0163] If so, the candidate sampling point is determined as a valid sampling point.

[0164] In one possible implementation, when the device motion detection unit determines the device motion trajectory of the camera device based on the motion trajectory of the at least one valid sampling point, it is specifically used for:

[0165] Based on the motion trajectory of the at least one valid sampling point, the inter-frame motion parameters of the camera device between the first original video frame and the preceding original video frame are obtained.

[0166] The device motion trajectory of the camera is updated based on the inter-frame motion parameters.

[0167] In one possible implementation, when the compensation parameter determination unit obtains the inverse compensation parameters for the first original video frame based on the device motion trajectory, it is specifically used for:

[0168] Based on the motion trajectory of the device, the shaking characteristics of the camera device within a first time window are obtained, and the shaking characteristics include shaking frequency and shaking amplitude;

[0169] If the jitter feature triggers any one of the compensation conditions, based on the device motion trajectory, the inter-frame motion parameters of the camera device between the first original video frame and the preceding original video frame are obtained; wherein, each of the compensation conditions is used to limit the numerical range of the jitter frequency and / or the jitter amplitude;

[0170] The inverse compensation parameters are obtained by inverting the inter-frame motion parameters.

[0171] In one possible implementation, the jitter compensation unit is used to compensate each pixel in the first original video frame based on the inverse compensation parameters to obtain the first target video frame, specifically for:

[0172] Each pixel in the first original video frame is moved according to the offset and offset direction indicated by the inverse compensation parameters to obtain the first target video frame.

[0173] In one possible implementation, a screen restoration unit is also included, configured to: identify a pixel missing region in the first target video frame and an effective pixel boundary adjacent to the pixel missing region; interpolate a filling pixel sequence based on the pixels on the effective pixel boundary; and fill the pixel missing region with the filling pixel sequence along a direction perpendicular to the effective pixel boundary and pointing to the pixel missing region to obtain a target output video frame.

[0174] This application also provides an electronic device. Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device provided in this application includes at least one processor and a memory connected to the processor, wherein: the memory is used to store a computer program; the processor is used to execute the computer program to enable the electronic device to implement the above-described video processing method.

[0175] The electronic devices in the embodiments of this application may include, but are not limited to, fixed terminals such as mobile phones, laptops, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Figure 9 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of this application. Figure 9 As shown, the electronic device may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage device 908 into a random access memory (RAM) 903. When the electronic device is powered on, the RAM 903 also stores various programs and data required for the operation of the electronic device. The processing unit 901, ROM 902, and RAM 903 are interconnected via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0176] Typically, the following devices can be connected to I / O interface 905: input devices 906 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 907 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 908 including, for example, memory cards, hard drives, etc.; and communication devices 909. Communication device 909 allows electronic devices to exchange data via wireless or wired communication with other devices. Although Figure 9 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have instead.

[0177] This application also provides a computer program product including computer-readable instructions, which, when executed on an electronic device, cause the electronic device to implement any of the video processing methods provided in this application.

[0178] This application also provides a computer-readable storage medium that carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any of the video processing methods provided in this application.

[0179] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0180] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0181] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0182] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A video processing method, comprising: Obtain the original video frames from the original video stream captured by the camera device in sequence; After obtaining the first original video frame, the motion trajectory of at least one valid sampling point in the first original video frame is obtained; Based on the motion trajectory of the at least one valid sampling point, the device motion trajectory of the camera is determined; Based on the device's motion trajectory, inverse compensation parameters for the first original video frame are obtained; Based on the inverse compensation parameters, each pixel in the first original video frame is compensated to obtain the first target video frame.

2. The video processing method according to claim 1, obtaining the motion trajectory of at least one valid sampling point in the first original video frame, includes: If the first original video frame is not the first video frame in the first original video frame sequence, the pyramid optical flow algorithm is used to trace the effective sampling points of the preceding original video frames in the first original video frame to obtain the inter-frame offset vector of at least one candidate sampling point; wherein, the first original video frame sequence is any original video frame sequence obtained by dividing the original video stream according to the first duration. Based on the inter-frame offset vector of each of the candidate sampling points, at least one valid sampling point is determined from the at least one candidate sampling point; Based on the inter-frame offset vector of the at least one valid sampling point, the motion trajectory of the at least one valid sampling point is obtained.

3. The video processing method according to claim 2, wherein obtaining the motion trajectory of the at least one valid sampling point based on the inter-frame offset vector of the at least one valid sampling point includes: If the number of valid sampling points is determined to be not less than a first quantity threshold, the motion trajectory of the at least one valid sampling point is obtained based on the inter-frame offset vector of the at least one valid sampling point.

4. The video processing method according to claim 3 further includes: If the first original video frame is the first video frame, or the number of valid sampling points is less than the first number threshold, corner detection is performed on the first original video frame to determine multiple key points and obtain the valid sampling points of the first original video frame.

5. The video processing method according to claim 2, wherein determining at least one valid sampling point from the at least one candidate sampling point based on the inter-frame offset vector of each of the candidate sampling points, includes: Based on the inter-frame offset vector of multiple edge reference points, determine whether camera device movement has occurred; If so, for each candidate sampling point, determine whether the length of the motion trajectory of the candidate sampling point is consistent with the length of the motion trajectory of the edge reference point; If so, the candidate sampling point is determined as a valid sampling point.

6. The video processing method according to claim 1, determining the device motion trajectory of the camera equipment based on the motion trajectory of the at least one valid sampling point, includes: Based on the motion trajectory of the at least one valid sampling point, the inter-frame motion parameters of the camera device between the first original video frame and the preceding original video frame are obtained. The device motion trajectory of the camera is updated based on the inter-frame motion parameters.

7. The video processing method according to claim 1, wherein, based on the device motion trajectory, inverse compensation parameters for the first original video frame are obtained, including: Based on the motion trajectory of the device, the shaking characteristics of the camera device within a first time window are obtained, and the shaking characteristics include shaking frequency and shaking amplitude; If the jitter feature triggers any one of the compensation conditions, based on the device motion trajectory, the inter-frame motion parameters of the camera device between the first original video frame and the preceding original video frame are obtained; wherein, each of the compensation conditions is used to limit the numerical range of the jitter frequency and / or the jitter amplitude; The inverse compensation parameters are obtained by inverting the inter-frame motion parameters.

8. The video processing method according to claim 7, wherein the step of compensating each pixel in the first original video frame based on the inverse compensation parameter to obtain the first target video frame includes: Each pixel in the first original video frame is moved according to the offset and offset direction indicated by the inverse compensation parameters to obtain the first target video frame.

9. The video processing method according to claim 1, further comprising: Identify the pixel missing regions in the first target video frame and the effective pixel boundaries adjacent to the pixel missing regions; Using the pixels on the effective pixel boundary as a reference, interpolation is performed to obtain the filling pixel sequence; The missing pixel region is filled with the filling pixel sequence along a direction perpendicular to the effective pixel boundary and pointing towards the missing pixel region to obtain the target output video frame.

10. A video processing apparatus, characterized in that, include: The video frame acquisition unit is used to sequentially acquire the original video frames in the original video stream captured by the camera device; The sampling point trajectory tracking unit is used to obtain the motion trajectory of at least one valid sampling point in the first original video frame after obtaining the first original video frame. The device motion detection unit is used to determine the device motion trajectory of the camera device based on the motion trajectory of the at least one valid sampling point; The compensation parameter determination unit is used to obtain inverse compensation parameters for the first original video frame based on the device motion trajectory. The jitter compensation unit is used to compensate each pixel in the first original video frame based on the inverse compensation parameters to obtain the first target video frame.

Images