A video processing method, a video processing device and related products

Abstract

The embodiment of the application discloses a video processing method, a video processing device and related products, which are used in the technical field of data processing. In the method, based on the first weight corresponding to the original transformation matrix of the Nth frame and the second weight corresponding to the smoothed transformation matrix of the (N-1)th frame, the original transformation matrix of the Nth frame and the smoothed transformation matrix of the (N-1)th frame are smoothed to obtain the smoothed transformation matrix of the Nth frame; the coordinates of the target display element in the Nth frame are transformed using the smoothed transformation matrix of the Nth frame, the target display element is added to the video picture to be displayed, and the video picture is anti-shake. Only the original transformation matrix of the Nth frame and the smoothed transformation matrix of the (N-1)th frame are used to transform the coordinates of the target display element in the Nth frame, and a deep learning model is not needed, so that the video picture can be anti-shake in real time.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a video processing method, a video processing device, and related products. Background Technology

[0002] As the video recording capabilities of electronic devices have improved, more and more users are using them for video recording. During video recording, especially when users are handheld and taking selfies, the devices often shake or vibrate to some extent, resulting in unstable video footage.

[0003] Current methods typically involve inputting a sequence of captured video frames into a deep learning model. The model then predicts the video frames to replace the jittery frames in the sequence, thus achieving video stabilization. However, deep learning models require large datasets and significant time to train. Furthermore, the models themselves require substantial computational time during inference to predict output video frames, making real-time video stabilization difficult. Summary of the Invention

[0004] This application provides a video processing method, a video processing device, and related products that can stabilize video images in real time.

[0005] This application provides a video processing method, including:

[0006] Obtain the original transformation matrix of the Nth frame in the video, and the smoothed transformation matrix of the (N-1)th frame; where N is greater than or equal to 2; the original transformation matrix of the Nth frame is the coordinate mapping relationship between the face position and the face model in the video frame to be displayed in the Nth frame; the face position in the video frame to be displayed is described by the video frame coordinate system, the coordinates of the face model are described by the calibration coordinate system, and the calibration coordinate system is also used to describe the coordinates of the candidate display elements associated with the face model;

[0007] Determine the first weight corresponding to the original transformation matrix of the Nth frame, and the second weight corresponding to the smoothed transformation matrix of the (N-1)th frame;

[0008] Based on the first weight and the second weight, the original transformation matrix of the Nth frame and the smoothed transformation matrix of the (N-1)th frame are smoothed to obtain the smoothed transformation matrix of the Nth frame.

[0009] Using the smoothed transformation matrix of the Nth frame, the coordinates of the target display element in the video frame to be displayed in the calibration coordinate system are transformed to obtain the target coordinates of the target display element in the video frame coordinate system; the target display element belongs to the candidate display element;

[0010] The target display element is added to the video frame to be displayed according to the target coordinates to obtain the video frame to be displayed in the Nth frame.

[0011] Furthermore, the method also includes:

[0012] Based on the smoothed transformation matrix of the Nth frame and the original transformation matrix of the Nth frame, determine the coordinate offset matrix of the Nth frame;

[0013] Based on the coordinate offset matrix and the first weight, the third weight corresponding to the original transformation matrix of the N+1th frame and the fourth weight corresponding to the smoothed transformation matrix of the Nth frame are determined.

[0014] Based on the third weight and the fourth weight, the original transformation matrix of the (N+1)th frame and the smoothed transformation matrix of the Nth frame are smoothed to obtain the smoothed transformation matrix of the (N+1)th frame.

[0015] Furthermore, determining the third weight corresponding to the original transformation matrix of the (N+1)th frame and the fourth weight corresponding to the smoothed transformation matrix of the Nth frame based on the coordinate offset matrix and the first weight includes:

[0016] Obtain the coordinate offset in the coordinate offset matrix of the Nth frame;

[0017] Within a preset weight range, based on the coordinate offset and the first weight, the third weight corresponding to the original transformation matrix of the (N+1)th frame is obtained;

[0018] Based on the third weight corresponding to the original transformation matrix of the (N+1)th frame, the fourth weight corresponding to the smoothed transformation matrix of the Nth frame is determined.

[0019] Furthermore, when the Nth frame is the second frame, the (N-1)th frame is the first frame;

[0020] Obtaining the smoothed transform matrix of the (N-1)th frame in the video includes:

[0021] Obtain the original transformation matrix of the first frame in the video;

[0022] Based on the original transformation matrix of the first frame and the preset weights corresponding to the original transformation matrix of the first frame, the smoothed transformation matrix of the first frame is obtained.

[0023] Furthermore, obtaining the original transformation matrix of the Nth frame in the video includes:

[0024] Obtain the coordinates of the first vertex of the face in the video frame to be displayed in the video frame coordinate system, and the coordinates of the second vertex of the face model in the calibration coordinate system;

[0025] Based on the mapping relationship between the first vertex coordinates and the second vertex coordinates, the original transformation matrix of the Nth frame is obtained.

[0026] Further, determining the first weight corresponding to the original transformation matrix of the Nth frame and the second weight corresponding to the smoothed transformation matrix of the (N-1)th frame includes:

[0027] Obtain the first weight corresponding to the original transformation matrix of the Nth frame;

[0028] The second weight corresponding to the smoothed transformation matrix of the (N-1)th frame is set to be inversely proportional to the first weight.

[0029] Furthermore, the first weight is: a n The second weight is: 1-a n ;

[0030] The step of smoothing the original transformation matrix of the Nth frame and the smoothed transformation matrix of the (N-1)th frame based on the first weight and the second weight to obtain the smoothed transformation matrix of the Nth frame includes:

[0031] Use the formula: Obtain the smoothed transformation matrix of the Nth frame. Among them, M n The original transformation matrix of the Nth frame is... Let be the smoothed transformation matrix of the (N-1)th frame.

[0032] Furthermore, the target display element in the video frame to be displayed in the Nth frame includes: a face effect texture;

[0033] The step of using the smoothed transformation matrix of the Nth frame to perform coordinate transformation on the coordinates of the target display element in the calibration coordinate system of the Nth frame video frame to obtain the target coordinates of the target display element in the video frame coordinate system includes:

[0034] Multiply the smoothed transformation matrix of the Nth frame by the vertex coordinates of the face effect texture in the calibrated coordinate system to obtain the target coordinates of the face effect texture in the video frame coordinate system.

[0035] This application also provides a video processing apparatus, including:

[0036] The acquisition unit is used to acquire the original transformation matrix of the Nth frame in the video and the smoothed transformation matrix of the (N-1)th frame; wherein N is greater than or equal to 2; the original transformation matrix of the Nth frame is the coordinate mapping relationship between the face position and the face model in the video frame to be displayed in the Nth frame; the face position in the video frame to be displayed is described by the video frame coordinate system, the coordinates of the face model are described by the calibration coordinate system, and the calibration coordinate system is also used to describe the coordinates of the candidate display elements associated with the face model;

[0037] The determining unit is used to determine the first weight corresponding to the original transformation matrix of the Nth frame and the second weight corresponding to the smoothed transformation matrix of the (N-1)th frame.

[0038] A smoothing unit is used to smooth the original transformation matrix of the Nth frame and the smoothed transformation matrix of the (N-1)th frame based on the first weight and the second weight, so as to obtain the smoothed transformation matrix of the Nth frame.

[0039] The transformation unit is used to perform coordinate transformation on the coordinates of the target display element in the calibration coordinate system in the video frame to be displayed in the Nth frame using the smoothed transformation matrix of the Nth frame, so as to obtain the target coordinates of the target display element in the video frame coordinate system; the target display element belongs to the candidate display element;

[0040] An execution unit is configured to add the target display element to the video frame to be displayed according to the target coordinates, thereby obtaining the video frame to be displayed in the Nth frame.

[0041] This application also provides a video processing apparatus, including:

[0042] Processor, memory, and input / output interfaces;

[0043] The memory is either a short-term storage memory or a persistent storage memory;

[0044] The processor is configured to communicate with the memory and execute instructions in the memory to perform the methods described above.

[0045] This application also provides a computer-readable storage medium including instructions that, when executed on a computer, cause the computer to perform the method described above.

[0046] This application also provides a computer program product containing instructions or computer programs, which, when run on a computer, causes the computer to perform the method described above.

[0047] As can be seen from the above technical solutions, the embodiments of this application have the following advantages:

[0048] In this embodiment, based on the first weight corresponding to the original transformation matrix of the Nth frame and the second weight corresponding to the smoothed transformation matrix of the (N-1)th frame, the original transformation matrix of the Nth frame and the smoothed transformation matrix of the (N-1)th frame are smoothed to obtain the smoothed transformation matrix of the Nth frame. Using the smoothed transformation matrix of the Nth frame, the coordinates of the target display element in the video frame to be displayed are transformed in the calibration coordinate system, and the target display element is added to the video frame to be displayed, thereby achieving anti-shake of the video frame of the Nth frame. Only the original transformation matrix of the Nth frame and the smoothed transformation matrix of the (N-1)th frame are needed to transform the coordinates of the target display element in the video frame to be displayed in the Nth frame to achieve anti-shake of the video frame of the Nth frame. No deep learning model is required, and anti-shake of the video frame can be achieved in real time. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings.

[0050] Figure 1 This is a schematic diagram of a video processing communication architecture disclosed in an embodiment of this application;

[0051] Figure 2 This is a flowchart of a video processing method disclosed in an embodiment of this application;

[0052] Figure 3 This is a flowchart of obtaining the smoothed transformation matrix of the (N+1)th frame, as disclosed in an embodiment of this application.

[0053] Figure 4 This is a flowchart of obtaining the smoothed transformation matrix of the first frame, as disclosed in an embodiment of this application;

[0054] Figure 5 This is a flowchart of another video processing method disclosed in an embodiment of this application;

[0055] Figure 6 This is a schematic diagram of a video processing algorithm disclosed in an embodiment of this application;

[0056] Figure 7 This is a schematic diagram of a video processing apparatus disclosed in an embodiment of this application;

[0057] Figure 8This is a schematic diagram of another video processing apparatus disclosed in an embodiment of this application. Detailed Implementation

[0058] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0059] In the description of the embodiments of this application, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing the embodiments of this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the embodiments of this application.

[0060] In the description of the embodiments of this application, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in the embodiments of this application based on the specific circumstances.

[0061] Existing video processing architectures such as Figure 1 As shown, the system includes a video processing device 101 and a video player 102. The video player 102 can be an electronic device capable of playing videos, such as a mobile phone or a computer. The video player 102 has corresponding video files, which can be pre-stored video files or video files recorded and played in real time (such as video files used during video calls or live video streaming). The video processing device 101 can be wired or wirelessly connected to one or more video players 102 to perform image stabilization on the video files displayed on the video players 102.

[0062] Video processing devices often input video frame sequences from video files on a video player into a deep learning model. The model then replaces jittery frames in the sequence with predicted video frames to achieve video stabilization. However, deep learning models require large datasets and significant training time. Furthermore, the inference process involves complex computations to predict output video frames, making real-time video stabilization difficult. Therefore, this application provides a video processing method for real-time video stabilization, such as... Figure 2 As shown, the specific steps include the following:

[0063] 201. Obtain the original transformation matrix of the Nth frame in the video, and the smoothed transformation matrix of the (N-1)th frame.

[0064] In this embodiment, the video processing device can obtain the original transformation matrix of the Nth frame and the smoothed transformation matrix of the (N-1)th frame in the video. The video is a video file with multiple consecutive time-series frames; it can be a real-time captured video file or a video file obtained from the network, and the specific format is not limited here. N is greater than or equal to 2, meaning the Nth frame is the second video frame and the video frames following it. The (N-1)th frame is the frame preceding the Nth frame. If the Nth frame is the third frame (current frame) about to be played in the video, then the (N-1)th frame is the second frame.

[0065] The original transformation matrix of the Nth frame represents the coordinate mapping between the face position and the face model in the video frame to be displayed. The face model contains basic facial elements such as eyes, nose, mouth, and head. The face position in the video frame is described by the video frame coordinate system, while the face model's coordinates are described by the calibration coordinate system, which serves as the reference coordinate system in the video processing device. In other words, the original transformation matrix of the Nth frame is used to transform the coordinates of the face position in the video frame to be displayed and position it in the calibration coordinate system. The calibration coordinate system also describes the coordinates of candidate display elements associated with the face model. These candidate display elements are elements that may exist on the face besides the constituent elements of the face model, such as hats, glasses, and beards. It is understood that the relative positions between the candidate display elements and the face model are fixed. For example, if the candidate display element is a hat, its position is on a key point of the face model's head; if the candidate display element is a beard, its position is on a key point of the face model's mouth, and so on.

[0066] That is, the video processing device can obtain the original transformation matrix of the Nth frame and the smoothed transformation matrix of the (N-1)th frame before the video player displays the video frame N. The original transformation matrix of the Nth frame is then subjected to adaptive smoothing to obtain the smoothed transformation matrix of the Nth frame, thereby achieving anti-shaking of the video frame N. The smoothed transformation matrix of the (N-1)th frame is obtained by adaptively smoothing the original transformation matrix of the (N-1)th frame.

[0067] Obtaining the original transformation matrix of the Nth frame in a video can be achieved by: obtaining the coordinates of the first vertex of the face in the video frame (to be displayed) in the video frame, and the coordinates of the second vertex of the face model in the calibration coordinate system; and obtaining the original transformation matrix of the Nth frame based on the mapping relationship between the first and second vertex coordinates. For example, a face detection model can be used to detect key points of the face in the Nth frame of the video frame. The coordinates of the first vertex corresponding to the key points of the face in the video frame (to be displayed) and the coordinates of the second vertex of the face key points in the calibration coordinate system can be obtained. The first and second vertex coordinates can then be mapped point-to-point, and the original transformation matrix (mapping matrix) of the Nth frame can be calculated using matrix solving methods.

[0068] 202. Determine the first weight corresponding to the original transformation matrix of the Nth frame and the second weight corresponding to the smoothed transformation matrix of the (N-1)th frame.

[0069] After obtaining the original transformation matrix of frame N and the smoothed transformation matrix of frame N-1, the first weight corresponding to the original transformation matrix of frame N and the second weight corresponding to the smoothed transformation matrix of frame N-1 can be determined. That is, corresponding weights can be assigned to the original transformation matrix of frame N and the smoothed transformation matrix of frame N-1. It can be understood that these first and second weights are used to smooth the original transformation matrix of frame N and the smoothed transformation matrix of frame N-1. Generally, the sum of the first and second weights is 1. The first and second weights can be set by the user.

[0070] Preferably, the first weight corresponding to the original transformation matrix of the Nth frame can be obtained, and the second weight corresponding to the smoothed transformation matrix of the (N-1)th frame can be set to be inversely proportional to the first weight. That is, when the first weight corresponding to the original transformation matrix of the Nth frame is large, the second weight corresponding to the smoothed transformation matrix of the (N-1)th frame is small, so that the video processing is biased towards the original transformation matrix of the Nth frame, so as to accelerate the convergence to the original transformation matrix of the Nth frame during video processing and avoid the video frame of the Nth frame from having a delay; or, when the first weight corresponding to the original transformation matrix of the Nth frame is small, the second weight corresponding to the smoothed transformation matrix of the (N-1)th frame is large, so as to bias towards the smoothed transformation matrix of the (N-1)th frame, so as to eliminate the small jitter signal that appears when the video frames of the (N-1)th frame and the Nth frame are played continuously.

[0071] 203. Based on the first weight and the second weight, smooth the original transformation matrix of the Nth frame and the smoothed transformation matrix of the (N-1)th frame to obtain the smoothed transformation matrix of the Nth frame.

[0072] After determining the first weight corresponding to the original transformation matrix of frame N and the second weight corresponding to the smoothed transformation matrix of frame N-1, the original transformation matrix of frame N and the smoothed transformation matrix of frame N-1 can be smoothed based on the first and second weights to obtain the smoothed transformation matrix of frame N. That is, the original transformation matrix of frame N can be multiplied by the first weight, and then the smoothed transformation matrix of frame N-1 can be multiplied by the second weight to obtain the smoothed transformation matrix of frame N.

[0073] 204. Using the smoothed transformation matrix of the Nth frame, perform coordinate transformation on the coordinates of the target display element in the calibration coordinate system in the video frame to be displayed in the Nth frame, and obtain the target coordinates of the target display element in the video frame coordinate system.

[0074] After obtaining the smoothed transformation matrix of the Nth frame, it can be used to transform the coordinates of the target display element in the calibration coordinate system within the Nth frame of the video frame to obtain the target coordinates of the target display element in the video frame coordinate system. This target display element belongs to the candidate display elements. In other words, obtain the coordinates of the target display element in the calibration coordinate system, and then use the smoothed transformation matrix of the Nth frame to transform these coordinates to obtain the target coordinates of the target display element in the video frame coordinate system. For example, if the target display element is a hat, the coordinates of the hat in the calibration coordinate system can be transformed to obtain the target coordinates of the hat in the video frame coordinate system.

[0075] 205. Add the target display element to the video frame to be displayed according to the target coordinates to obtain the video frame to be displayed in the Nth frame.

[0076] After obtaining the target coordinates, the target display elements can be added to the video frame to be displayed according to the target coordinates, resulting in the video frame to be displayed in the Nth frame. This means rendering the target display elements in the target coordinates of the video frame coordinate system. In other words, the target coordinates corresponding to all target display elements in the Nth frame of the video frame can be obtained, unifying the vertex coordinates of all target display elements in the Nth frame of the video frame in the video frame coordinate system, thus avoiding video jitter.

[0077] In this embodiment, based on the first weight corresponding to the original transformation matrix of the Nth frame and the second weight corresponding to the smoothed transformation matrix of the (N-1)th frame, the original transformation matrix of the Nth frame and the smoothed transformation matrix of the (N-1)th frame are smoothed to obtain the smoothed transformation matrix of the Nth frame. Using the smoothed transformation matrix of the Nth frame, the coordinates of the target display element in the video frame to be displayed are transformed in the calibration coordinate system, and the target display element is added to the video frame to be displayed, thereby achieving anti-shake of the video frame of the Nth frame. Only the original transformation matrix of the Nth frame and the smoothed transformation matrix of the (N-1)th frame are needed to transform the coordinates of the target display element in the video frame to be displayed in the Nth frame to achieve anti-shake of the video frame of the Nth frame. No deep learning model is required, and anti-shake of the video frame can be achieved in real time.

[0078] Furthermore, after obtaining the smoothed transformation matrix of the Nth frame, the coordinate offset matrix of the Nth frame can be determined. Based on the coordinate offset matrix of the Nth frame, the smoothed transformation matrix of the (N+1)th frame is obtained to achieve image stabilization for the (N+1)th frame, thus achieving real-time image stabilization. Figure 3 As shown, the specific steps are as follows:

[0079] 301. Based on the smoothed transformation matrix of the Nth frame and the original transformation matrix of the Nth frame, determine the coordinate offset matrix of the Nth frame.

[0080] After obtaining the smoothed transformation matrix of frame N, the coordinate offset matrix of frame N can be determined based on the smoothed transformation matrix of frame N and the original transformation matrix of frame N. Specifically, the coordinate offset matrix of frame N can be obtained by subtracting the original transformation matrix of frame N from the smoothed transformation matrix of frame N.

[0081] The coordinate offset matrix represents the difference between the smoothed transformation matrix of the Nth frame and the original transformation matrix of the Nth frame before smoothing. The smaller the coordinate offset matrix, the smaller the offset of the displayed element in the Nth frame, that is, the smaller the motion amplitude relative to the N-1th frame. The larger the coordinate offset matrix, the larger the offset of the displayed element in the Nth frame, that is, the larger the motion amplitude relative to the N-1th frame.

[0082] 302. Based on the coordinate offset matrix and the first weight, determine the third weight corresponding to the original transformation matrix of the N+1th frame, and the fourth weight corresponding to the smoothed transformation matrix of the Nth frame.

[0083] Next, based on the coordinate offset matrix and the first weight, the third weight corresponding to the original transformation matrix of frame N+1 and the fourth weight corresponding to the smoothed transformation matrix of frame N can be determined. That is, by using the coordinate offset matrix of frame N and the first weight, the degree of attention paid to the original transformation matrix of frame N+1 and the smoothed transformation matrix of frame N is adaptively adjusted when obtaining the smoothed transformation matrix of frame N+1, thereby improving the stability of the video image.

[0084] Generally, if the coordinate offset matrix is ​​larger, the third weight corresponding to the original transformation matrix of frame N+1 is increased, and the fourth weight corresponding to the smoothed transformation matrix of frame N is decreased. This increases the attention paid to the original transformation matrix of frame N+1, meaning that when the motion amplitude of the video image to be displayed is large, it speeds up the convergence to frame N+1 and avoids delays in the video image of frame N+1. Conversely, if the coordinate offset matrix is ​​smaller, the third weight corresponding to the original transformation matrix of frame N+1 is decreased, and the fourth weight corresponding to the smoothed transformation matrix of frame N is increased. This increases the attention paid to the smoothed transformation matrix of frame N, meaning that when the motion amplitude of the video image to be displayed is small, it makes the video image of frame N+1 more continuous with the video image of frame N.

[0085] Specifically, the coordinate offset in the coordinate offset matrix of frame N can be obtained. This can be achieved by summing the absolute values ​​of multiple elements in the coordinate offset matrix of frame N. Within a preset weight range, based on the coordinate offset and the first weight, the third weight corresponding to the original transformation matrix of frame N+1 is obtained. This preset weight range includes an upper and lower weight limit, which can be set manually. The sum of the coordinate offset and the first weight is used to determine if it falls within the preset weight range. If it does, this sum is used as the third weight corresponding to the original transformation matrix of frame N+1. If the sum is greater than the upper weight limit, the upper weight limit is used as the third weight corresponding to the original transformation matrix of frame N+1. If the sum is less than the lower weight limit, the lower weight limit is used as the third weight corresponding to the original transformation matrix of frame N+1. Next, based on the third weight corresponding to the original transformation matrix of frame N+1, the fourth weight corresponding to the smoothed transformation matrix of frame N is determined.

[0086] 303. Based on the third and fourth weights, smooth the original transformation matrix of frame N+1 and the smoothed transformation matrix of frame N to obtain the smoothed transformation matrix of frame N+1.

[0087] After determining the third weight corresponding to the original transformation matrix of frame N+1 and the fourth weight corresponding to the smoothed transformation matrix of frame N, the original transformation matrix of frame N+1 and the smoothed transformation matrix of frame N can be smoothed based on the third weight and the fourth weight to obtain the smoothed transformation matrix of frame N+1. That is, the original transformation matrix of frame N+1 and the smoothed transformation matrix of frame N are assigned the third weight and the fourth weight respectively and then summed to obtain the smoothed transformation matrix of frame N+1.

[0088] Using the smoothed transformation matrix of frame N+1, the coordinates of the target display element in the calibration coordinate system of the video frame to be displayed in frame N+1 are transformed to obtain the target coordinates of the target display element in the video frame coordinate system. The target display element is then added to the video frame to be displayed according to these target coordinates, resulting in the video frame to be displayed in frame N+1, thus achieving anti-shake for the video frame N+1.

[0089] Furthermore, in this embodiment, the smoothed transform matrix of the (N-1)th frame is needed to obtain the smoothed transform matrix of the Nth frame. When the Nth frame is the second frame, the smoothed transform matrix of the first frame needs to be obtained to provide a data basis for the subsequent Nth frame. The process of obtaining the smoothed transform matrix of the first frame is as follows: Figure 4 As shown, the specific steps are as follows:

[0090] 401. Obtain the original transformation matrix of the first frame in the video.

[0091] It is understandable that obtaining the original transformation matrix of the first frame in the video is similar to obtaining the original transformation matrix of the Nth frame in the video in step 201 above, and the specifics will not be repeated here.

[0092] 402. Based on the original transformation matrix of the first frame and the preset weights of the original transformation matrix of the first frame, the smoothed transformation matrix of the first frame is obtained.

[0093] After obtaining the original transformation matrix of the first frame in the video, the smoothed transformation matrix of the first frame can be obtained based on the original transformation matrix of the first frame and the preset weights corresponding to the original transformation matrix of the first frame. The preset weights corresponding to the original transformation matrix of the first frame can be set by the user. That is, the original transformation matrix of the first frame is assigned weights to obtain the smoothed transformation matrix of the first frame.

[0094] Understandably, after obtaining the smoothed transformation matrix of the first frame, the coordinate offset matrix of the first frame can be determined based on the smoothed transformation matrix and the original transformation matrix of the first frame. Based on the coordinate offset matrix and preset weights, the weights corresponding to the original transformation matrix of the second frame and the weights corresponding to the smoothed transformation matrix of the first frame can be determined. The original transformation matrix of the second frame and the smoothed transformation matrix of the first frame are then smoothed to obtain the smoothed transformation matrix of the second frame. The specific process is as follows... Figure 3 Similarly, the specifics will not be elaborated here. It can be seen that after obtaining the smoothed transformation matrix of the first frame, this smoothed transformation matrix of the first frame can be used in the process of determining the smoothed transformation matrix of the subsequent Nth frame, providing a data basis for the subsequent Nth frame.

[0095] The following will combine Figure 5 as well as Figure 6 The video processing procedure is described below, with the specific steps as follows:

[0096] 501. Obtain the original transformation matrix of the Nth frame in the video, and the smoothed transformation matrix of the (N-1)th frame.

[0097] It is understandable that step 501 is similar to step 201 above, and will not be elaborated further here. The original transformation matrix can be the model matrix, and the original transformation matrix of the Nth frame is: M n ∈R( 4×4 ), and the details are as follows:

[0098]

[0099] Where, m 01 m 02 m 10 m 12 m 20 m 22 m is a rotation element. 00 m 11 m 22 To scale the element, m 03 m 13 m 23 For translation elements.

[0100] 502. Determine the first weight corresponding to the original transformation matrix of the Nth frame, and the second weight corresponding to the smoothed transformation matrix of the (N-1)th frame.

[0101] Specifically, the original transformation matrix of frame N-1 can be obtained. Based on the smoothed transformation matrix of frame N-1 and the original transformation matrix of frame N-1, the coordinate offset matrix of frame N-1 is determined. Based on the coordinate offset matrix of frame N-1 and the weights corresponding to the original transformation matrix of frame N-1, the first weight corresponding to the original transformation matrix of frame N and the second weight corresponding to the smoothed transformation matrix of frame N-1 are determined. The specific process is as follows... Figure 3 Similarly, details will not be elaborated here. It is understandable that by using the coordinate offset matrix based on the (N-1)th frame and the weights corresponding to the original transformation matrix of the (N-1)th frame, determining the first weight corresponding to the original transformation matrix of the Nth frame and the second weight corresponding to the smoothed transformation matrix of the (N-1)th frame, the smoothed transformation matrix obtained in the Nth frame can be made more correlated with the smoothed transformation matrix of the (N-1)th frame, further improving the stability of the video image in the Nth frame.

[0102] 503. Based on the first weight and the second weight, smooth the original transformation matrix of the Nth frame and the smoothed transformation matrix of the (N-1)th frame to obtain the smoothed transformation matrix of the Nth frame.

[0103] Specifically, the first weight corresponding to the original transformation matrix of the Nth frame is: a n The smoothed transformation matrix of the (N-1)th frame, according to the corresponding second weight, is: 1-a n That is, the original transformation matrix of the Nth frame and the smoothed transformation matrix of the (N-1)th frame are used as indicators of the smoothed transformation matrix of the Nth frame, and the sum of the weights of the indicators is 1.

[0104] Use the formula: Obtain the smoothed transformation matrix of the Nth frame. That is, the smoothed transformation matrix of the Nth frame after adaptive smoothing; where, M n Let N be the original transformation matrix. This is the smoothed transformation matrix for the (N-1)th frame.

[0105] 504. Multiply the smoothed transformation matrix of the Nth frame by the vertex coordinates of the face effect texture in the calibration coordinate system to obtain the vertex coordinates of the face effect texture in the calibration coordinate system, and obtain the target coordinates of the face effect texture in the video frame coordinate system.

[0106] Furthermore, when users add special effects to a live stream, the face effect texture in the displayed video frame will shake on the face. At this time, the smoothed transformation matrix of the Nth frame can be used to transform the vertex coordinates of the face effect texture in the calibration coordinate system to obtain the target coordinates of the face effect texture in the video frame coordinate system.

[0107] Specifically, the smoothed transformation matrix of the Nth frame is multiplied by the vertex coordinates of the face effect texture in the calibration coordinate system to obtain the vertex coordinates of the face effect texture in the calibration coordinate system, thus obtaining the target coordinates of the face effect texture in the video frame coordinate system. For example, the coordinates of the first vertex (homogeneous coordinates v) of any vertex (x, y, z) of the face effect texture in the calibration coordinate system can be obtained. local (x,y,z,1), the smoothed transformation matrix after the Nth frame. By performing a coordinate transformation, we can obtain the target coordinates v of any vertex of the face effect texture in the video frame coordinate system. world The specific formula is as follows:

[0108]

[0109] 505. Add the face effect texture to the video frame to be displayed according to the target coordinates to obtain the video frame to be displayed in the Nth frame.

[0110] After obtaining the target coordinates of the face effect texture in the video frame coordinate system, the face effect texture is added to the video frame to be displayed according to the target coordinates, resulting in the video frame to be displayed in the Nth frame. That is, the face effect texture is rendered based on the target coordinates to obtain the effect video frame to be displayed in the Nth frame, thereby achieving smooth optimization of the time sequence frames, avoiding jitter of the face effect texture, and improving the stability of the effect video frame.

[0111] Furthermore, after obtaining the smoothed transformation matrix of the Nth frame in step 503, the smoothed transformation matrix of the Nth frame and the original transformation matrix of the Nth frame can be used to obtain the smoothed transformation matrix of the (N+1)th frame, as follows:

[0112] Based on the smoothed transformation matrix of frame N and the original transformation matrix of frame N, the coordinate offset matrix of frame N is determined. This coordinate offset matrix is ​​the offset matrix E of frame N before and after smoothing. n ∈R( 4×4 ),Right now

[0113] To obtain the coordinate offset in the coordinate offset matrix of frame N, specifically, you can do so in the offset matrix E. n To select a specific element, use the following formula:

[0114] e n =|e 00 |+|e 01 |+|e 10 |+|e 11 |+|e 03 |+|e 13 |-b, obtain the coordinate offset e of the coordinate offset matrix. n , where eij For the offset matrix E n The element in row i and column j, where b is a fixed compensation value.

[0115] Next, within the preset weight range, based on the coordinate offset and the first weight, the third weight corresponding to the original transformation matrix of the (N+1)th frame is obtained. Specifically, the third weight α corresponding to the original transformation matrix of the (N+1)th frame can be obtained using the following formula. n+1 :

[0116] α n+1 =min(max(α) n +β×e n (minimal), maximal)

[0117] Where β is the scaling factor for the coordinate offset, and minimal and maximal are the upper and lower bounds of the weights, respectively. The fourth weight corresponding to the smoothed transformation matrix of the Nth frame is: 1-α n+1 .

[0118] It can be seen that if the coordinate offset matrix (i.e., the offset matrix E) n The smaller the value of α, the greater the third weight α corresponding to the original transformation matrix of the (N+1)th frame. n+1 The smaller the value, the lower the corresponding fourth weight (1-α) of the smoothed transformation matrix of the Nth frame. n+1 The larger the coordinate offset matrix (i.e., the offset matrix E), the more continuous the video image and the better it is at eliminating minute jitter signals; n The larger the value of α, the larger the third weight α of the original transformation matrix of the (N+1)th frame. n+1 The larger the value, the greater the fourth weight (1-α) of the smoothed transformation matrix of the Nth frame. n+1 The smaller the value, the faster the convergence to the (N+1)th frame, and the less delay there will be in the video footage of the (N+1)th frame.

[0119] Based on the third and fourth weights, the original transformation matrix of frame N+1 and the smoothed transformation matrix of frame N are smoothed to obtain the smoothed transformation matrix of frame N+1. This can be achieved using the formula: Obtain the smoothed transformation matrix of the (N+1)th frame. The video frame of frame N+1 is obtained by using the smoothed transformation matrix of frame N+1.

[0120] This application also provides a video processing apparatus, such as... Figure 7 As shown, it includes:

[0121] The acquisition unit 701 is used to acquire the original transformation matrix of the Nth frame in the video and the smoothed transformation matrix of the (N-1)th frame; wherein N is greater than or equal to 2; the original transformation matrix of the Nth frame is the coordinate mapping relationship between the face position and the face model in the video frame to be displayed in the Nth frame; the face position in the video frame to be displayed is described by the video frame coordinate system, the coordinates of the face model are described by the calibration coordinate system, and the calibration coordinate system is also used to describe the coordinates of the candidate display elements associated with the face model;

[0122] The determining unit 702 is used to determine the first weight corresponding to the original transformation matrix of the Nth frame and the second weight corresponding to the smoothed transformation matrix of the (N-1)th frame.

[0123] Smoothing unit 703 is used to smooth the original transformation matrix of the Nth frame and the smoothed transformation matrix of the (N-1)th frame based on the first weight and the second weight, so as to obtain the smoothed transformation matrix of the Nth frame.

[0124] Transformation unit 704 is used to perform coordinate transformation on the coordinates of the target display element in the calibration coordinate system in the video frame to be displayed in the Nth frame using the smoothed transformation matrix of the Nth frame, so as to obtain the target coordinates of the target display element in the video frame coordinate system; the target display element belongs to the candidate display element;

[0125] The execution unit 705 is used to add the target display element to the video frame to be displayed according to the target coordinates, so as to obtain the video frame to be displayed in the Nth frame.

[0126] This application embodiment also provides a video processing apparatus 800, such as... Figure 8 As shown, the video processing apparatus 800 of this application embodiment may include one or more processors CPU (central processing units) 801 and memory 802, wherein the memory 802 stores one or more applications or data.

[0127] The memory 802 can be volatile or persistent storage. The program stored in the memory 802 can include one or more modules, each module including a series of instruction operations on the electronic device. Furthermore, the processor 801 can be configured to communicate with the memory 802 and execute the series of instruction operations stored in the memory 802 on the video processing device 800.

[0128] The video processing device 800 may also include one or more power supplies 805, one or more wired or wireless network interfaces 804, one or more input / output interfaces 803, and / or one or more operating systems, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0129] The processor 801 can perform the operations performed in the aforementioned specific method embodiments, which will not be described in detail here.

[0130] This application also provides a computer-readable storage medium including instructions that, when executed on a computer, cause the computer to perform the method described above.

[0131] This application also provides a computer program product containing instructions or computer programs, which, when run on a computer, causes the computer to perform the method described above.

[0132] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0133] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0134] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0135] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0136] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0137] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0138] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0139] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0140] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0141] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A video processing method, characterized in that, include: Obtain the original transformation matrix of the Nth frame in the video, and the smoothed transformation matrix of the (N-1)th frame; where N is greater than or equal to 2; the original transformation matrix of the Nth frame is the coordinate mapping relationship between the face position and the face model in the video frame to be displayed in the Nth frame; the face position in the video frame to be displayed is described by the video frame coordinate system, the coordinates of the face model are described by the calibration coordinate system, and the calibration coordinate system is also used to describe the coordinates of the candidate display elements associated with the face model; Determine the first weight corresponding to the original transformation matrix of the Nth frame, and the second weight corresponding to the smoothed transformation matrix of the (N-1)th frame; Based on the first weight and the second weight, the original transformation matrix of the Nth frame and the smoothed transformation matrix of the (N-1)th frame are smoothed to obtain the smoothed transformation matrix of the Nth frame. Using the smoothed transformation matrix of the Nth frame, the coordinates of the target display element in the video frame to be displayed in the calibration coordinate system are transformed to obtain the target coordinates of the target display element in the video frame coordinate system; the target display element belongs to the candidate display element; The target display element is added to the video frame to be displayed according to the target coordinates to obtain the video frame to be displayed in the Nth frame.

2. The video processing method according to claim 1, characterized in that, The method further includes: Based on the smoothed transformation matrix of the Nth frame and the original transformation matrix of the Nth frame, determine the coordinate offset matrix of the Nth frame; Based on the coordinate offset matrix and the first weight, the third weight corresponding to the original transformation matrix of the N+1th frame and the fourth weight corresponding to the smoothed transformation matrix of the Nth frame are determined. Based on the third weight and the fourth weight, the original transformation matrix of the (N+1)th frame and the smoothed transformation matrix of the Nth frame are smoothed to obtain the smoothed transformation matrix of the (N+1)th frame.

3. The video processing method according to claim 2, characterized in that, The step of determining the third weight corresponding to the original transformation matrix of the (N+1)th frame and the fourth weight corresponding to the smoothed transformation matrix of the Nth frame based on the coordinate offset matrix and the first weight includes: Obtain the coordinate offset in the coordinate offset matrix of the Nth frame; Within a preset weight range, based on the coordinate offset and the first weight, the third weight corresponding to the original transformation matrix of the (N+1)th frame is obtained; Based on the third weight corresponding to the original transformation matrix of the (N+1)th frame, the fourth weight corresponding to the smoothed transformation matrix of the Nth frame is determined.

4. The video processing method according to claim 1, characterized in that, When the Nth frame is the second frame, the (N-1)th frame is the first frame; Obtaining the smoothed transform matrix of the (N-1)th frame in the video includes: Obtain the original transformation matrix of the first frame in the video; Based on the original transformation matrix of the first frame and the preset weights corresponding to the original transformation matrix of the first frame, the smoothed transformation matrix of the first frame is obtained.

5. The video processing method according to claim 1, characterized in that, Obtaining the original transformation matrix of the Nth frame in the video includes: Obtain the coordinates of the first vertex of the face in the video frame to be displayed in the video frame coordinate system, and the coordinates of the second vertex of the face model in the calibration coordinate system; Based on the mapping relationship between the first vertex coordinates and the second vertex coordinates, the original transformation matrix of the Nth frame is obtained.

6. The video processing method according to claim 1, characterized in that, Determining the first weight corresponding to the original transformation matrix of the Nth frame and the second weight corresponding to the smoothed transformation matrix of the (N-1)th frame includes: Obtain the first weight corresponding to the original transformation matrix of the Nth frame; The second weight corresponding to the smoothed transformation matrix of the (N-1)th frame is set to be inversely proportional to the first weight.

7. The video processing method according to claim 1, characterized in that, The first weight is: a n The second weight is: 1-a n ; The step of smoothing the original transformation matrix of the Nth frame and the smoothed transformation matrix of the (N-1)th frame based on the first weight and the second weight to obtain the smoothed transformation matrix of the Nth frame includes: Use the formula: Obtain the smoothed transformation matrix of the Nth frame. Among them, M n The original transformation matrix of the Nth frame is... Let be the smoothed transformation matrix of the (N-1)th frame.

8. The video processing method according to claim 1, characterized in that, The target display elements in the video frame to be displayed in the Nth frame include: face effect textures; The step of using the smoothed transformation matrix of the Nth frame to perform coordinate transformation on the coordinates of the target display element in the calibration coordinate system of the Nth frame video frame to obtain the target coordinates of the target display element in the video frame coordinate system includes: Multiply the smoothed transformation matrix of the Nth frame by the vertex coordinates of the face effect texture in the calibrated coordinate system to obtain the target coordinates of the face effect texture in the video frame coordinate system.

9. A video processing apparatus, characterized in that, include: The acquisition unit is used to acquire the original transformation matrix of the Nth frame in the video and the smoothed transformation matrix of the (N-1)th frame; wherein N is greater than or equal to 2; the original transformation matrix of the Nth frame is the coordinate mapping relationship between the face position and the face model in the video frame to be displayed in the Nth frame; the face position in the video frame to be displayed is described by the video frame coordinate system, the coordinates of the face model are described by the calibration coordinate system, and the calibration coordinate system is also used to describe the coordinates of the candidate display elements associated with the face model; The determining unit is used to determine the first weight corresponding to the original transformation matrix of the Nth frame and the second weight corresponding to the smoothed transformation matrix of the (N-1)th frame. A smoothing unit is used to smooth the original transformation matrix of the Nth frame and the smoothed transformation matrix of the (N-1)th frame based on the first weight and the second weight, so as to obtain the smoothed transformation matrix of the Nth frame. The transformation unit is used to perform coordinate transformation on the coordinates of the target display element in the calibration coordinate system in the video frame to be displayed in the Nth frame using the smoothed transformation matrix of the Nth frame, so as to obtain the target coordinates of the target display element in the video frame coordinate system; the target display element belongs to the candidate display element; An execution unit is configured to add the target display element to the video frame to be displayed according to the target coordinates, thereby obtaining the video frame to be displayed in the Nth frame.

10. A video processing apparatus, characterized in that, include: Processor, memory, and input / output interfaces; The memory is either a short-term storage memory or a persistent storage memory; The processor is configured to communicate with the memory and execute instructions in the memory to perform the method of any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, Includes instructions that, when executed on a computer, cause the computer to perform the method as described in any one of claims 1 to 8.

12. A computer program product comprising instructions or a computer program, characterized in that, When the computer program product is run on a computer, it causes the computer to perform the method as described in any one of claims 1 to 8.

Images