Video processing methods, apparatus, computer equipment and computer-readable storage media

Abstract

The present application discloses a video processing method, device, computer equipment and computer-readable storage medium, the method comprising: obtaining shooting information of a first video frame and first correction information of a second video frame, the second video frame being a historical video frame; determining first stabilization information of the first video frame based on the shooting information; performing correction processing on the first stabilization information based on the shooting information and the first correction information to obtain a target video frame of the first video frame. The method of the present application can simultaneously realize uniform speed stabilization of video frames and alignment of stabilization results with the horizon, thereby improving the stabilization effect of video frames.

Description

Technical Field

[0001] The present application relates to the field of video processing technology, and in particular to a video processing method, apparatus, computer equipment and computer-readable storage medium. Background Art

[0002] With the rapid development of mobile terminals, more and more users shoot videos with handheld devices. Videos shot with handheld devices often have the problem of picture distortion. Therefore, it is necessary to perform video stabilization on the shot videos. Video stabilization refers to the process of converting the original shot video into a stabilized video through special cropping and interpolation methods, so that the stabilized video playback content presents uniform motion. However, the existing video stabilization methods have the problem of poor video stabilization effect. Summary of the invention

[0003] The embodiments of the present application provide a video processing method, apparatus, computer device and computer-readable storage medium, which can improve the stabilization effect of video frames.

[0004] The technical solution adopted by the present invention to solve the problem is as follows:

[0005] On the one hand, the present application provides a video processing method, comprising:

[0006] Acquire shooting information of a first video frame and first correction information of a second video frame, where the second video frame is a historical video frame;

[0007] Determining first stabilization information of a first video frame based on the shooting information;

[0008] The first stabilization information is corrected based on the shooting information and the first correction information to obtain a target video frame of the first video frame.

[0009] In some embodiments of the present application, determining first stabilization information of a first video frame based on the shooting information includes:

[0010] Determining second stabilization information of the first video frame based on the shooting information;

[0011] Acquire a rotation angle of the second stabilization information and a rotation angle of the shooting information;

[0012] The rotation angle of the second stabilization information is constrained based on the rotation angle of the shooting information to obtain the first stabilization information of the first video frame.

[0013] In some embodiments of the present application, the first stabilization information is corrected based on the shooting information and the first correction information to obtain a target video frame of the first video frame, including:

[0014] Based on the shooting information, determine frame state information of the first video frame, where the frame state information includes a static state and a non-static state;

[0015] If the frame state information is in a static state, fine-tuning and resetting the first stabilization information to obtain third stabilization information and first mapping information;

[0016] Determining target stabilization information for the first video frame based on the third stabilization information, the first mapping information, and the first correction information;

[0017] Based on the target stabilization information, a target video frame of the first video frame is determined.

[0018] In some embodiments of the present application, the first correction information includes a horizon alignment state, a horizon misalignment state, and a horizon alignment fine-tuning state. When the first correction information is the horizon alignment state, the target stabilization information of the first video frame is determined based on the third stabilization information, the first mapping information, and the first correction information, including:

[0019] Updating the third stabilization information to obtain updated third stabilization information;

[0020] Acquire gravity data of the first video frame, and determine a gravity calibration angle based on the gravity data;

[0021] Determining fourth stabilization information and a correction angle based on the updated third stabilization information, the first mapping information, and the gravity calibration angle;

[0022] Based on the correction angle, determining a first interpolation ratio;

[0023] The updated third stabilization information and the fourth stabilization information are interpolated based on the first interpolation ratio to obtain target stabilization information of the first video frame.

[0024] In some embodiments of the present application, when the first correction information is not in a horizon alignment state, determining target stabilization information of the first video frame based on the third stabilization information, the first mapping information, and the first correction information includes:

[0025] Determining fifth stabilization information and a mapping tilt angle based on the first mapping information, the third stabilization information, and the gravity calibration angle;

[0026] If the first correction information is a horizon misalignment state and the mapping tilt angle is greater than a preset first angle threshold, determining the third stabilization information as the target stabilization information of the first video frame;

[0027] If the first correction information is in a horizon alignment fine-tuning state or the mapping tilt angle is not greater than the first angle threshold, the fifth stabilization information is updated, and the updated fifth stabilization information is determined as the target stabilization information of the first video frame.

[0028] In some embodiments of the present application, after correcting the first stabilization information based on the shooting information and the first correction information to obtain a target video frame of the first video frame, the method includes:

[0029] Acquire second correction information of the stabilized video frame, the second correction information including a horizon alignment state, a horizon misalignment state, and a horizon alignment fine-tuning state;

[0030] If the second correction information is a horizon alignment state, the stabilized video frame is post-processed to correct the horizon and prevent crossing the boundary, so as to obtain third mapping information;

[0031] If the second correction information is not in the horizon alignment state or the horizon correction anti-crossing post-processing fails, performing stabilization anti-crossing post-processing on the stabilized video frame to obtain fourth mapping information;

[0032] Based on the third mapping information or the fourth mapping information, a post-processed video frame of the first video frame is determined.

[0033] In some implementation schemes of the present application, the stabilized video frame is post-processed for horizon correction and anti-crossing to obtain third mapping information, including:

[0034] Determine fifth mapping information based on the stabilized video frame, and obtain the cross-border condition of the stabilized cropping area of ​​the first video frame;

[0035] Decomposing the fifth mapping information based on the cross-boundary condition to obtain a rotation angle;

[0036] updating the rotation angle, and updating the fifth mapping information based on the updated rotation angle to obtain updated fifth mapping information;

[0037] Determine, based on the updated fifth mapping information, a cross-border condition of the stabilized cropping area of ​​the first video frame;

[0038] When the cross-border situation does not meet the preset conditions, the rotation angle is updated, and the step of updating the fifth mapping information based on the rotation angle to obtain the updated fifth mapping information is continued until the cross-border situation meets the preset conditions to obtain the third mapping information.

[0039] In a second aspect, an embodiment of the present invention further provides a video processing device, including:

[0040] An information acquisition unit, used to acquire shooting information of a first video frame and first correction information of a second video frame, where the second video frame is a historical video frame;

[0041] a parameter determination unit, configured to determine first stabilization information of a first video frame based on the shooting information;

[0042] A correction processing unit, configured to perform correction processing on the first stabilization information based on the shooting information and the first correction information to obtain a target video frame of the first video frame;

[0043] The post-processing unit is used to perform anti-crossing post-processing on the stabilized video frame to obtain a target video frame of the first video frame.

[0044] In a third aspect, the present application further provides a computer device, the computer device comprising:

[0045] one or more processors;

[0046] Memory; and

[0047] One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the processor to implement any one of the video processing methods in the first aspect.

[0048] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, and the computer program is loaded by a processor to execute the steps in any one of the video processing methods in the first aspect.

[0049] The beneficial effects of the present invention are as follows: first stabilization information of the first video frame is determined based on the shooting information, and the first stabilization information is corrected based on the shooting information and the first correction information. This can not only achieve uniform-speed stabilization of the first video frame, but also align the stabilization result with the horizon, thereby improving the stabilization effect of the video frame. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0051] Figure 1 is a flow chart of a video processing method provided by an embodiment of the present invention;

[0052] Figure 2 is a logical schematic diagram of a video stabilization transformation provided by an embodiment of the present invention;

[0053] Figure 3 is a logical schematic diagram of the horizon alignment state transition provided by an embodiment of the present invention;

[0054] Figure 4 is a schematic diagram of a stabilized clipping region boundary distance provided by an embodiment of the present invention;

[0055] Figure 5 is a principle block diagram of a video processing device provided by an embodiment of the present invention;

[0056] Figure 6 It is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0057] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0058] In the description of the present application, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the drawings, which are only for the convenience of describing the present application and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present application. In addition, the terms "first", "second", "third" and the like are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, features defined as "first", "second", "third", etc. may explicitly or implicitly include one or more features. In the description of the present application, "multiple" means two or more, and "several" means one or more, unless otherwise clearly and specifically defined.

[0059] In this application, the word "exemplary" is used to mean "used as an example, illustration, or description." Any embodiment described in this application as "exemplary" is not necessarily to be construed as being preferred or advantageous over other embodiments. The following description is given to enable any technician in the field to implement and use the present application. In the following description, details are listed for the purpose of explanation. It should be understood that a person of ordinary skill in the art can recognize that the present application can be implemented without using these specific details. In other instances, well-known structures and processes will not be elaborated in detail to avoid obscuring the description of the present application with unnecessary details. Therefore, the present application is not intended to be limited to the embodiments shown, but is consistent with the widest scope consistent with the principles and features disclosed in the present application.

[0060] It should be noted that since the method of the embodiment of the present application is executed in a computer device, the processing objects of each computer device exist in the form of data or information. For example, time is actually time information. It can be understood that if size, quantity, position, etc. are mentioned in subsequent embodiments, they are all corresponding data for processing by the computer device. The details will not be repeated here.

[0061] The content of this application is further illustrated below through the description of embodiments in conjunction with the accompanying drawings.

[0062] This embodiment provides a video processing method, such as Figure 1 As shown in , the method includes:

[0063] Step S10: acquiring shooting information of a first video frame and first correction information of a second video frame, wherein the second video frame is a historical video frame.

[0064] The video to be processed is a video that needs to be stabilized, for example, a video shot by a handheld device. The video to be processed includes multiple video frames, the first video frame and the second video frame are two frames of video in the video to be processed, and the second video frame is a historical video frame of the first video frame, for example, the second video frame is the previous frame of the first video frame, such as the first video frame is the i-th frame of video in the video to be processed, and the second video frame is the i-1-th frame of video in the video to be processed.

[0065] like Figure 2 As shown in FIG. 1 , the first video frame (i.e., the first video frame) of the video to be processed is represented as F1, the second video frame is represented as F2, the third video frame is represented as F3, the fourth video frame is represented as F4, and so on. There is a mapping transformation relationship between two adjacent video frames, and the mapping transformation relationship can be represented by a motion parameter M. For example, the mapping transformation relationship between F1 and F2 is represented by a motion parameter M1, and the mapping transformation relationship between F2 and F3 is represented by a motion parameter M2, ..., F t-1 With F t The mapping transformation relationship between them is determined by the motion parameter M t express.

[0066] The shooting information of the first video frame is the motion parameter that characterizes the mapping transformation relationship between the first video frame and the second video frame. The image content of the first video frame and the image content of the second video frame can be approximately represented by a two-dimensional plane mapping transformation based on the shooting information. t , the second video frame is F t-1 , the shooting information is M t , then F t ≈f(F t-1 , M t ), F t-1≈f(F t , M t -1 ), where the function f(·) represents the projective transformation.

[0067] In a specific implementation, the shooting information of the first video frame can be obtained by obtaining the rotation speed of a rotating device (such as a gyroscope). Due to the sampling frequency limitation of the rotating device, it is impossible to obtain the rotation speed of the rotating device at any time. In actual applications, the rotation speed of the rotating device at t i-1 to i The shooting information of the first video frame is approximately calculated by using n rotation speeds at the moment. The calculation formula of the shooting information is: w k =(x k y k z k ), w k represents the kth rotation speed.

[0068] The first correction information is the horizon alignment information of the stabilized video frame corresponding to the second video frame. The first correction information includes a horizon alignment state, a horizon misalignment state, and a horizon alignment fine-tuning state. The conversion logic of the three alignment states is as follows: Figure 3 As shown, when the stabilized cropping rotation angle is close to the maximum cropping rotation threshold, the picture may repeatedly trigger the horizon aligned state and the horizon misaligned state, thereby causing jitter. This embodiment can achieve smooth switching between the horizon aligned state and the horizon misaligned state by setting the horizon alignment fine-tuning state.

[0069] Specifically, the conditions for the current frame to enter the horizon alignment state include: ① The previous frame is in the horizon alignment state, and the stabilized clipping angle after the current frame is aligned is less than the maximum rotation threshold θ max ;② The previous frame is in the horizon alignment fine-tuning state, and the horizon alignment angle deviation of the current frame is less than a small angle deviation threshold θ small The conditions for the current frame to enter the horizon misaligned state include: ① The previous frame is in the horizon aligned state or the horizon aligned fine-tuned state, and the stabilized crop rotation angle of the current frame after alignment is greater than the maximum rotation threshold θ max ; ② The previous frame is in the state of horizon misalignment, and the stabilized crop rotation angle after alignment of the current frame is greater than a slightly larger rotation threshold θ1. The conditions for the current frame to enter the horizon alignment fine-tuning state include: ① The previous frame is in the state of horizon misalignment, and the stabilized crop rotation angle after alignment of the current frame is less than a slightly larger rotation threshold θ1; ② The previous stabilized frame is in the state of horizon alignment, and the horizon alignment angle deviation of the current frame is greater than a smaller angle deviation threshold.

[0070] Step S20: Determine first stabilization information of the first video frame based on the shooting information.

[0071] Continue to refer to Figure 2 As shown, S1 represents the stabilized video frame of F1, S2 represents the stabilized video frame of F2, ..., S n Indicates F n There is also a projection transformation relationship between two adjacent stabilized video frames. The projection transformation relationship between two adjacent stabilized video frames can be expressed by the stabilization parameter K. For example, F t Stabilized video frame S t The image content and F t-1 Stabilized video frame S t-1 The image contents can be compared with each other by using the stabilization parameter K t The two-dimensional plane projective transformation is approximately represented by S t ≈f(S t-1 , K t ), S t-1 ≈f(S t , K t -1 ), where the function f(·) represents the projective transformation.

[0072] The first stabilization information is a stabilization parameter that characterizes a mapping transformation relationship between a stabilized video frame of the first video frame and a stabilized video frame of the second video frame, determined based on the shooting information. After acquiring the shooting information of the first video frame, this embodiment determines the first stabilization information of the first video frame based on the shooting information, and uniform speed stabilization of the first video frame can be achieved based on the first stabilization information.

[0073] In a specific implementation, step S20 includes:

[0074] Step S21, determining second stabilization information of the first video frame based on the shooting information;

[0075] Step S22, obtaining a rotation angle of the second stabilization information and a rotation angle of the shooting information;

[0076] Step S23: constraining the rotation angle of the second stabilization information based on the rotation angle of the shooting information to obtain the first stabilization information of the first video frame.

[0077] The video processing scenarios include video preview and video recording. When the video processing scenario is video preview, the delay frame is 0, and the shooting information of the video frame after the first video frame in the video to be processed is unknown. Then, based on the shooting information, the step of determining the second stabilization information of the first video frame specifically includes: obtaining the stabilization information of the second video frame and a preset second interpolation ratio; interpolating the stabilization information and shooting information of the second video frame based on the second interpolation ratio to obtain the second stabilization information of the first video frame. Among them, the formula for determining the second stabilization information is: represents the second stabilization information, ρ1 represents the second interpolation ratio, K i-1 represents the stabilization information of the second video frame, M i Indicates the shooting information of the first video frame.

[0078] In another specific implementation, when the video processing scenario is video recording, since the stabilization effect does not need to be presented in real time, n-frame delay can be used for stabilization. At this time, the operating parameters of the n frames after the first video frame in the video frame to be processed can be regarded as known quantities. Then, based on the shooting information, the step of determining the second stabilization information of the first video frame specifically includes: obtaining the shooting information of the in-th video frame and the third video frame in the video to be processed, wherein the third video frame is the video frame other than the first video frame from the i-n+1-th video frame to the i+n-th video frame in the video to be processed, and i represents the frame number to which the first video frame belongs in the video to be processed; based on the shooting information of the in-th video frame, the third video frame and the shooting information of the first video frame, determining the i+n-th video frame; based on the i+n-th video frame and the in-th video frame, determining the second stabilization information of the first video frame. Among them, the determination process of the i+n-th video frame can be expressed as: The determination process of the second stabilization information can be expressed as: F i+n Indicates the i+nth video frame, F i-n Indicates the inth video frame, represents the second stabilization information of the first video frame, F i Represents the first video frame.

[0079] Considering that the original video frame suddenly stops from continuous motion, the speed of the stabilized video frame will continue to move according to the interpolation inertia. In order to always ensure that the motion amplitude of the stabilized video frame is less than or equal to the motion amplitude of the original video frame, after determining the second stabilized information of the first video frame, this embodiment obtains the rotation angle of the second stabilized information and the rotation angle of the shooting information; constrains the rotation angle of the second stabilized information based on the rotation angle of the shooting information, and converts the constrained second stabilized information into a quaternion form to obtain the first stabilized information of the first video frame. Among them, the process of constraining the rotation angle of the second stabilized information can be expressed as: Indicates the second stabilization information The rotation angle of Axis_M i .angle indicates shooting information M i The rotation angle of .

[0080] In a specific implementation, the step of obtaining the rotation angle of the second stabilization information and the rotation angle of the shooting information specifically includes: converting the second stabilization information into the second stabilization information in the form of a rotation axis, and converting the shooting information into the shooting information in the form of a rotation axis; obtaining the rotation angle of the second stabilization information based on the second stabilization information in the form of a rotation axis, and obtaining the rotation angle of the shooting information based on the shooting information in the form of a rotation axis. The second stabilization information in the form of a rotation axis can be expressed as: The shooting information in the form of rotation axis can be expressed as: i →Axis_M i .

[0081] It should be noted that, from the aforementioned determination process of the second stabilization information, when the video processing scene is video recording, the second stabilization information is Time has come The quaternion of motion at time t i-n Indicates F i-n Imaging time of the first line, Indicates F i-n The rolling shutter time, i.e. F i-n The difference between the imaging time of the last line and the first line of the picture, t i+n Indicates F i+n Imaging time of the first line, Indicates F i+n The rolling shutter time, i.e. F i+n Therefore, before constraining the rotation angle of the second stabilization information based on the rotation angle of the shooting information, it is necessary to scale the rotation angle of the second stabilization information to convert the rotation angle of the second stabilization information into Time has come The rotation amplitude at the moment. Among them, the scaling process of the rotation angle of the second stabilization information can be expressed as: t i Indicates F i That is, the imaging time of the first line of the first video frame, Indicates the rolling shutter time of the first video frame, i.e. F i The difference between the imaging time of the last line and the first line of the picture.

[0082] Step S30: Correct the first stabilization information based on the shooting information and the first correction information to obtain a target video frame of the first video frame.

[0083] The target video frame is a stabilized video frame of the first video frame. Considering that although the first stabilization information can achieve a video stabilization effect, it cannot ensure that the stabilization result is aligned with the horizon, after determining the first stabilization information, this embodiment corrects the first stabilization information based on the shooting information and the first correction information to obtain a stabilized video frame of the first video frame. Correcting the first stabilization information based on the shooting information and the first correction information can not only achieve a video stabilization effect, but also ensure that the stabilization result is aligned with the horizon.

[0084] In a specific implementation, step S30 includes:

[0085] Step S31: determining frame state information of the first video frame based on the shooting information, where the frame state information includes a static state and a non-static state;

[0086] Step S32: if the frame state information is in a static state, fine-tune and reset the first stabilization information to obtain third stabilization information and first mapping information;

[0087] Step S33, determining target stabilization information of the first video frame based on the third stabilization information, the first mapping information and the first correction information;

[0088] Step S34: Determine a target video frame of the first video frame based on the target stabilization information.

[0089] The frame status information is used to characterize whether the first video frame is in a stationary state relative to the second video frame. The first video frame and the second video frame are obtained through the camera module. Based on the shooting information, the step of determining the frame status information of the first video frame specifically includes: obtaining the internal reference matrix of the camera module and the coordinates of the image points in the second video frame; determining the coordinate mapping information based on the internal reference matrix and the shooting information; determining the coordinates of the image points in the first video frame based on the coordinates of the image points in the second video frame and the coordinate mapping information; determining the displacement information based on the coordinates of the image points in the second video frame and the coordinates of the image points in the first video frame; and determining the frame status information of the first video frame based on the displacement information. Among them, the formula for determining the coordinate mapping information is: Di =A*M i *A -1 , the coordinates of the image point in the first video frame can be expressed as: p i =D i *p i-1 , D i represents the coordinate mapping information, A represents the internal parameter matrix, p i represents the coordinates of the image point in the first video frame, p i-1 Represents the coordinates of the image point in the second video frame.

[0090] In a specific implementation, in order to measure whether the first video frame is in a static state, a displacement threshold is pre-set in this embodiment. After the displacement information is determined, the displacement information is compared with the displacement threshold. When the displacement information is less than the displacement threshold, the first video frame is judged to be in a static state; otherwise, the first video frame is judged to be in a non-static state. For example, the displacement threshold is τ i , the displacement threshold is μ, when τ i <μ, it is determined that the first video frame is in a stationary state; otherwise, it is determined that the first video frame is in a non-stationary state.

[0091] like Figure 4 As shown, the stabilized cropping area of ​​the first video frame is composed of 2L triangles, where L represents the number of rolling shutter row blocks, and each rolling shutter row block is composed of 2 triangles. In order to ensure that the stabilized cropping area does not cross the boundary, it is necessary to ensure that the minimum distance between the vertices of the stabilized cropping area and the top (t), bottom (b), left (l), and right (r) sides of the first video frame is greater than 0. When min(l, r) or min(t, b) is less than the preset distance threshold, it triggers fine-tuning and resetting of the first stabilization information.

[0092] Specifically, when min(l,r) and min(t,b) are both less than the preset distance threshold, the x and y axes need to be reset at the same time, and the steps of fine-tuning and resetting the first stabilization information specifically include: determining the sixth mapping information based on the first stabilization information; constructing the first mapping point pair based on the four corner points of the stabilized cropping area of ​​the first video frame; performing least squares estimation based on the first mapping point pair to obtain a first estimated value; eliminating the horizontal and vertical displacements of the first estimated value to obtain the first decomposition parameter of the sixth mapping information; determining the second decomposition parameter of the sixth mapping information based on the first decomposition parameter; adjusting the rotation angle of the second decomposition parameter to obtain the updated second decomposition parameter; determining the updated sixth mapping information based on the updated second decomposition parameter and the first decomposition parameter; determining the third stabilization information based on the updated sixth mapping information. The calculation formula for the sixth mapping information is: The first mapping point pair can be expressed as: The process of determining the second decomposition parameter can be expressed as The process of adjusting the rotation angle of the second decomposition parameter can be expressed as , the updated sixth mapping information can be expressed as: represents the sixth mapping information, Indicates the L / 2th row of the first video frame, S i-1 represents a stabilized video frame of the second video frame, represents the first stabilization information, A represents the internal parameter matrix, (x i ,y i |i∈[0,3]) represents the four corner points of the stabilized clipping area; Represents the first mapping point pair, R′ z_i represents the first decomposition parameter, represents the second decomposition parameter, ∈ represents the preset trace value, represents the updated second decomposition parameter, represents the updated sixth decomposition parameter.

[0093] Further, when only min(l,r) is less than the preset distance threshold, the x-axis needs to be reset, and the steps of fine-tuning and resetting the first stabilization information specifically include: determining the sixth mapping information based on the first stabilization information; constructing the second mapping point pair based on the center point of the stabilized cropping area of ​​the first video frame; performing least squares estimation based on the second mapping point pair to obtain the third decomposition parameter; adjusting the rotation angle of the third decomposition parameter to obtain the updated third decomposition parameter; determining the updated sixth mapping information based on the updated third decomposition parameter and the fourth decomposition parameter; determining the third stabilization information based on the updated sixth mapping information. Among them, the second mapping point pair can be expressed as: The process of adjusting the rotation angle of the third decomposition parameter can be expressed as , the updated sixth mapping information can be expressed as: represents the sixth mapping information, represents the second mapping point pair, A represents the internal parameter matrix, (width / 2, height / 2) represents the center point of the stabilized clipping area, represents the third decomposition parameter, represents the updated sixth mapping information, represents the updated third decomposition parameter, Represents the fourth decomposition parameter.

[0094] Furthermore, when only min(b, t) is less than the preset distance threshold, the y-axis needs to be reset, and the steps of fine-tuning and resetting the first stabilization information specifically include: determining the sixth mapping information based on the first stabilization information; constructing a second mapping point pair based on the center point of the stabilized cropping area of ​​the first video frame; performing least squares estimation based on the second mapping point pair to obtain the fifth decomposition parameter; adjusting the rotation angle of the fifth decomposition parameter to obtain the updated fifth decomposition parameter; determining the updated sixth mapping information based on the updated fifth decomposition parameter and the sixth decomposition parameter; determining the third stabilization information based on the updated sixth mapping information. The process of adjusting the rotation angle of the fifth decomposition parameter can be expressed as , the process of determining the updated sixth mapping information can be expressed as: represents the fifth decomposition parameter, represents the updated sixth mapping information, represents the updated fifth decomposition parameter, Represents the sixth decomposition parameter.

[0095] In a specific implementation, the first correction information includes a horizon alignment state, a horizon misalignment state, and a horizon alignment fine-tuning state. When the first correction information is the horizon alignment state, step S33 includes:

[0096] Step S331, updating the third stabilization information to obtain updated third stabilization information;

[0097] Step S332: Obtaining gravity data of the first video frame, and determining a gravity calibration angle based on the gravity data;

[0098] Step S333, determining fourth stabilization information and a correction angle based on the updated third stabilization information, the first mapping information and the gravity calibration angle;

[0099] Step S334: determining a first interpolation ratio based on the correction angle;

[0100] Step S335 : interpolate the updated third stabilization information and the fourth stabilization information based on the first interpolation ratio to obtain target stabilization information of the first video frame.

[0101] If the previous frame is in the horizon alignment state, when processing the current frame, if the z-axis rotation amplitude of the stabilized video frame of the current frame is 0, the stabilized video frame of the current frame is still in the horizon alignment state. Therefore, in this embodiment, when the first correction information is in the horizon alignment state, the third stabilization information is updated to obtain the updated third stabilization information. Among them, the step of updating the third stabilization information specifically includes: performing Euler decomposition on the third stabilization information, removing the z-axis rotation component motion, and obtaining the updated first stabilization information. Specifically, when removing the z-axis rotation component motion, the z-axis rotation amount obtained after the decomposition is compared with 0 and π. If it is closer to 0, the z-axis rotation amount is set to 0, and if it is closer to π, the z-axis rotation amount is set to π.

[0102] Gravity data format is g t =(x t y t z t ), where t represents the sampling time corresponding to the gravity data. t ,y t 、z t They represent the three projection components of the gravity data on the 3D coordinate axis. Let the gravity data of each frame be The device must satisfy the formula for vertical screen shooting: The device must satisfy the formula for horizontal screen shooting: Among them, v represents the threshold parameter. By setting the threshold parameter, it can be ensured that there is a certain angle between the device screen and the horizon. If the gravity data of the video frame does not satisfy the above formula, the mobile phone screen is nearly parallel to the horizon or the angle is too small, and the horizon correction angle cannot be calculated.

[0103] Furthermore, if the device is photographed in portrait orientation, the y component value of the projected gravity data is 0, that is, the rotation angle satisfies the formula: If the device is shooting in landscape mode, the x-component value of the gravity data after projection is 0, that is, the rotation angle satisfies the formula: That is, the gravity calibration angle θ may be determined based on the gravity data.

[0104] The step of determining the fourth stabilization information and the correction angle based on the updated third stabilization information, the first mapping information and the gravity calibration angle specifically includes: determining the correction angle based on the first mapping information and the gravity calibration angle; determining the first alignment matrix based on the correction angle; determining the fourth stabilization information based on the first alignment matrix and the updated third stabilization information. The first alignment matrix can be expressed as: The calculation formula of the fourth stabilization information is: H θ′represents the first alignment matrix, θ′ represents the correction angle, represents the fourth stabilization information, Indicates the updated third stabilization information.

[0105] In a specific implementation, based on the first mapping information and the gravity calibration angle, the step of determining the correction angle specifically includes: calculating the row block number of the midpoint endpoint of the stabilized video frame of the first video frame in the first video frame based on the first mapping information, and then calculating the position information of the midpoint endpoint of the first video frame according to the mapping information corresponding to the row block number; determining the mapping tilt angle based on the position information of the midpoint endpoint of the first video frame; determining the correction angle based on the mapping tilt angle and the gravity calibration angle. Wherein, the calculation formula of the correction angle is: θ′=θ-θ m , θ′ represents the correction angle, θ represents the gravity calibration angle, θ m Indicates the mapping tilt angle.

[0106] In order to achieve that the picture is always consistent with the correction direction of the first frame under any angle rotation, before determining the correction angle based on the mapping tilt angle and the gravity calibration angle, the mapping tilt angle θ m Make corresponding adjustments. Specifically, when 360 stabilization is not enabled, θ m =θ m , 360 stabilization is enabled and the angle between the first frame correction direction and the current frame correction direction is but Enable 360 ​​stabilization and the angle between the first frame correction direction and the current frame correction direction is but If 360 stabilization is enabled and the angle between the first frame correction direction and the current frame correction direction is π, then θ m =θ m +π.

[0107] In a specific implementation, based on the correction angle, the step of determining the first interpolation ratio specifically includes: comparing the correction angle with a preset second angle threshold and a third angle threshold; when the correction angle is less than the second angle threshold, it is considered that the correction angle is very small, and considering that the previous frame is in the gravity alignment direction, the first interpolation ratio is set to 0; when the correction angle is greater than the second angle threshold and less than the third angle threshold, the first interpolation ratio is set to When the correction angle is greater than the third angle threshold, the first interpolation ratio is set to 1, that is, Wherein, θ′ represents the correction angle, α represents the second angle threshold, β represents the third angle threshold, and ρ1 represents the first interpolation ratio.

[0108] The process of interpolating the updated third stabilization information and the fourth stabilization information based on the first interpolation ratio can be expressed as: Ki represents the target stabilization information, ρ1 represents the first interpolation ratio, Indicates the updated third stabilization information, Indicates the fourth stabilization information.

[0109] In a specific implementation, when the first correction information is not in the horizon alignment state, step S33 includes:

[0110] Step S336, determining fifth stabilization information and a mapping tilt angle based on the first mapping information, the third stabilization information and the gravity calibration angle;

[0111] Step S337: if the first correction information is a horizon misalignment state and the mapping tilt angle is greater than a preset first angle threshold, determine the third stabilization information as the target stabilization information of the first video frame;

[0112] Step S338: If the first correction information is in the horizon alignment fine-tuning state or the mapping tilt angle is not greater than the first angle threshold, the fifth stabilization information is updated, and the updated fifth stabilization information is determined as the target stabilization information of the first video frame.

[0113] In step S336, the fifth stabilization information and the mapping tilt angle are determined based on the first mapping information, the third stabilization information and the gravity calibration angle, which is the same as the step of determining the fourth stabilization information and the mapping tilt angle based on the updated third stabilization information, the first mapping information and the gravity calibration angle in the aforementioned step S333. For details, please refer to the specific steps of the aforementioned step S333, and this application will not repeat them here.

[0114] In this embodiment, when it is determined that the first correction information is in a state of misaligned horizon and the mapping tilt angle is greater than a preset first angle threshold, it is considered that the stabilized cropping shape of the first video frame is likely to cause out-of-bounds jitter. If the alignment fine-tuning process is entered, it is likely to trigger out-of-bounds jitter. Therefore, the correction information of the first video frame is determined to be in a state of misaligned horizon, and the third stabilization information is determined as the target stabilization information of the first video frame.

[0115] Further, when it is determined that the first correction information is in a horizon alignment fine-tuning state or the mapping tilt angle is not greater than the first angle threshold, the fifth stabilization information is updated, and the updated fifth stabilization information is determined as the target stabilization information of the first video frame.

[0116] Specifically, when the frame state information of the first video frame is not a static state, the step of updating the fifth stabilization information specifically includes: converting the fifth stabilization information into the form of a rotation axis; constraining the rotation angle of the fifth stabilization information in the form of a rotation axis based on the shooting information in the form of a rotation axis to obtain the constrained fifth stabilization information; converting the constrained fifth stabilization information into a quaternion form to obtain the updated fifth stabilization information.

[0117] Further, when the frame state information of the first video frame is in a static state, the step of updating the fifth stabilization information specifically includes: updating the correction angle based on a preset threshold parameter to obtain an updated correction angle; and determining the updated fifth stabilization information based on the updated correction angle. The process of updating the correction angle based on the threshold parameter can be expressed as: θ′ represents the correction angle, and ν represents the threshold parameter.

[0118] It should be noted that the step of determining the updated fifth stabilization information based on the updated correction angle is the same as the step of determining the fourth stabilization information based on the correction angle. For details, please refer to the step of determining the fourth stabilization information based on the correction angle, and this application will not repeat them here.

[0119] In a specific implementation, step S30 includes:

[0120] Step S41, obtaining second correction information of the stabilized video frame, the second correction information including a horizon alignment state, a horizon misalignment state, and a horizon alignment fine-tuning state;

[0121] Step S42: if the second correction information is in the horizon alignment state, perform horizon correction and anti-crossing post-processing on the stabilized video frame to obtain third mapping information;

[0122] Step S43: If the second correction information is not in the horizon alignment state or the horizon correction anti-crossing post-processing fails, perform stabilization anti-crossing post-processing on the stabilized video frame to obtain fourth mapping information;

[0123] Step S44: determine a post-processed video frame of the first video frame based on the third mapping information or the fourth mapping information.

[0124] Considering that the stabilized video frame obtained by horizon correction processing cannot ensure that the picture content of the stabilized frame comes from the original video frame, after obtaining the stabilized video frame, the present application performs anti-cross-border post-processing on the stabilized video frame to ensure that the picture content of the stabilized video frame comes from the original video frame.

[0125] The second correction information is the alignment information of the stabilized video frame, and the second correction information includes the horizon alignment status. When the stabilized video frame is post-processed to prevent crossing the boundary, the second correction information of the stabilized video frame is first obtained, and it is determined whether the second correction information is in the horizon alignment status. When the second correction information is in the horizon alignment status, the stabilized video frame is post-processed to prevent crossing the boundary by horizon correction to obtain third mapping information, and the post-processed video frame of the first video frame is determined based on the third mapping information; if the second correction information is not the horizon correction information or the post-processing to prevent crossing the boundary by horizon correction fails, the stabilized video frame is post-processed to prevent crossing the boundary by stabilization to obtain fourth mapping information, and the post-processed video frame of the first video frame is determined based on the fourth mapping information.

[0126] In a specific implementation, step S42 includes:

[0127] Step S421: determining fifth mapping information based on the stabilized video frame, and obtaining the crossing condition of the stabilized cropping area of ​​the first video frame;

[0128] Step S422: Decompose the fifth mapping information based on the crossing-border condition to obtain a rotation angle;

[0129] Step S423: updating the rotation angle, and updating the fifth mapping information based on the updated rotation angle to obtain updated fifth mapping information;

[0130] Step S424: determining the crossing of the stabilized cropping area of ​​the first video frame based on the updated fifth mapping information;

[0131] Step S425: When the cross-border situation does not meet the preset conditions, the rotation angle is updated, and the step of updating the fifth mapping information based on the rotation angle to obtain the updated fifth mapping information is continued until the cross-border situation meets the preset conditions to obtain the third mapping information.

[0132] It can be seen from the aforementioned steps that the out-of-bounds situations of the stabilized cropped area of ​​the first video frame include three types, namely, left-right out-of-bounds, top-bottom out-of-bounds, and top-bottom and left-right out-of-bounds. Specifically, when the out-of-bounds situation is left-right out-of-bounds or top-bottom out-of-bounds, the step of decomposing the fifth mapping information based on the out-of-bounds situation specifically includes: constructing a third mapping point pair based on the center point of the stabilized cropped area of ​​the first video frame and the fifth mapping information; determining the second alignment matrix based on the third mapping point pair, the center point of the stabilized cropped area and the internal parameter matrix; determining the rotation angle based on the second alignment matrix. Among them, the step of constructing the third mapping point pair based on the center point of the stabilized cropped area of ​​the first video frame and the fifth mapping information is the same as the aforementioned step of constructing the second mapping point pair based on the center point of the stabilized cropped area of ​​the first video frame. Specifically, reference can be made to the aforementioned step of constructing the second mapping point pair based on the center point of the stabilized cropped area of ​​the first video frame, and this application will not repeat it here.

[0133] Specifically, when the crossing of the boundary is left or right, When the crossing situation is left or right crossing, represents the third mapping point pair, A represents the internal parameter matrix, (width / 2, height / 2) represents the center point of the stabilized clipping area, and represents the second alignment matrix, θ x and θ y Indicates the rotation angle. Therefore, when the out-of-bounds situation is left or right, the second alignment matrix can be used to Determine the rotation angle θ x , when the cross-border situation is upper and lower cross-border, you can use the second alignment matrix Determine the rotation angle θ y .

[0134] Furthermore, when the crossing-over situation is the crossing-over of up and down and left and right, the step of decomposing the fifth mapping information based on the crossing-over situation specifically includes: constructing a fourth mapping point pair based on the four corner points of the stabilized cropping area of ​​the first video frame and the fifth mapping information; determining a third alignment matrix based on the fourth mapping point pair, the four corner points of the stabilized cropping area and the internal parameter matrix; and determining a rotation angle based on the third alignment matrix. Among them, the step of constructing the fourth mapping point pair based on the four corner points of the stabilized cropping area of ​​the first video frame and the fifth mapping information is the same as the step of constructing the first mapping point pair based on the four corner points of the stabilized cropping area of ​​the first video frame, and can be specifically referred to the step of constructing the first mapping point pair based on the four corner points of the stabilized cropping area of ​​the first video frame.

[0135] Specifically, when the crossing of the boundary is up and down and left and right, represents the fourth mapping point pair, A represents the internal parameter matrix, Indicates the four corner points of the stabilized clipping area. represents the third alignment matrix, θ z Indicates the rotation angle. Therefore, when the out-of-bounds situation is up and down and left and right, you can use the third alignment matrix Determine the rotation angle θ z .

[0136] In a specific implementation, the step of updating the fifth mapping information based on the updated rotation angle specifically includes: updating the fifth mapping information based on the updated rotation angle to obtain first updated mapping information; updating the fifth mapping information based on the first updated mapping information and combining the formula Determine a second stabilized video frame; based on the second stabilized video frame and in combination with the formula Determine second updated mapping information; determine the tilt mapping angle based on the second updated mapping information; determine a fourth alignment matrix based on the tilt mapping angle and the gravity calibration angle; determine a fourth alignment matrix based on the fourth alignment matrix and in combination with the formula Determine the updated fifth mapping information. Wherein, based on the second updated mapping information, the step of determining the tilted mapping angle is the same as the step of determining the mapping tilt angle based on the first mapping information, and the specific step can refer to the step of determining the mapping tilt angle based on the first mapping information, and this application will not repeat it here.

[0137] Furthermore, when the crossing condition is a left-right crossing, the formula for updating the fifth mapping information based on the updated rotation angle is: When the crossing condition is crossing the upper and lower boundaries, the formula for updating the fifth mapping information based on the rotation angle is: When the crossing situation is the crossing of up and down and left and right, the formula for updating the fifth mapping information based on the rotation angle is:

[0138] In a specific implementation, the step of determining the crossing of the stabilized clipping area of ​​the first video frame based on the updated fifth mapping information includes: determining the initial drawing point and the clipping matrix; for each point in the initial drawing point, determining a mapping value based on the updated fifth mapping information, and determining the rolling curtain row block idx to which the point belongs based on the mapping value. R , and continue to execute the mapping information based on the corresponding rolling shutter row block Determine the mapping value, and determine the rolling shutter row block idx to which the point belongs based on the mapping value R until the estimated rolling shutter row blocks no longer change; based on The initial drawing point is transformed with the clipping matrix to obtain the clipping grid point; based on the clipping grid point and the initial drawing point, the boundary distance of the drawing grid point is determined; based on the boundary distance, the cross-border situation of the stabilized clipping area of ​​the first video frame is determined. The initial drawing point can be expressed as: The clipping matrix can be expressed as: The clipping grid points can be expressed as: T represents the initial drawing point, H crop Represents the clipping matrix.

[0139] Furthermore, the boundary distance is calculated as follows: l = min(x)|(x,y)∈T, t = min(y)|(x,y)∈T, r = width-1-max(x)|(x,y)∈T, b = height-1-max(y)|(x,y)∈T, {x∈[0,width-1], y∈[0,height-1]|(x,y)∈T new Specifically, when min(l,r)<0, it indicates that there is a left-right out-of-bounds situation, when min(t,b)<0, it indicates that there is a top-bottom out-of-bounds situation, and when min(l,r)<0 and min(t,b)<0, it indicates that there is a left-right and top-bottom out-of-bounds situation.

[0140] The step of updating the rotation angle specifically includes: determining a search step length, and updating the rotation angle based on the search step length. When the crossing of the boundary is left or right crossing of the boundary and the rotation angle is updated for the first time, the search step length The update formula for the rotation angle is: θ x =θ x -step1, step1 represents the initial search step length, θ x Indicates the rotation angle; when the out-of-bounds situation is left or right out-of-bounds and the rotation angle is not updated for the first time, if min(l,r)>0, step>ε, The update formula for the rotation angle is: θ x =θ x -step3, step2 represents the previous search step, step3 represents the current search step, and ε represents the preset step threshold; when the out-of-bounds situation is left or right out-of-bounds and the rotation angle is not updated for the first time, if min(l,r)<0 is satisfied, the update formula of the rotation angle is: θ x =max(0,θ x -step2).

[0141] Furthermore, when the out-of-bounds situation is up and down and the rotation angle is updated for the first time, the search step length is The update formula for the rotation angle is: θ y =θ y -step1, step1 represents the initial search step length, θy Indicates the rotation angle; when the out-of-bounds situation is up and down and the rotation angle is not updated for the first time, if min(t,b)>0, step>ε, The update formula for the rotation angle is: θ y =θ y -step3, step2 represents the previous search step, step3 represents the current search step, and ε represents the preset step threshold; when the out-of-bounds situation is left or right out-of-bounds and the rotation angle is not updated for the first time, if min(t,b)<0 is satisfied, the update formula of the rotation angle is: θ y =max(0,θ y -step2).

[0142] Furthermore, when the out-of-bounds situation is left-right and up-down out-of-bounds and the rotation angle is updated for the first time, the search step length is The update formula for the rotation angle is: step1 represents the initial search step length, Indicates the rotation angle; when the out-of-bounds situation is left-right and up-down out-of-bounds and the rotation angle is not updated for the first time, if min(t,b)>0, min(l,r)>0, step>ε, The update formula for the rotation angle is: step2 represents the previous search step, step3 represents the current search step, and ε represents the preset step threshold. When the out-of-bounds situation is left-right and up-down out-of-bounds and the rotation angle is not updated for the first time, if min(t,b)<0 or min(l,r)<0 is satisfied, the update formula of the rotation angle is:

[0143] As mentioned in the above steps, the out-of-bounds situations include left-right out-of-bounds, top-bottom out-of-bounds, and left-right and top-bottom out-of-bounds. Therefore, there are three ways to determine whether the out-of-bounds situation meets the preset conditions. Specifically, when the out-of-bounds situation is left-right out-of-bounds, determining whether the out-of-bounds situation meets the preset conditions includes: determining whether min(l,r) is greater than or equal to zero, and whether the angle adjustment step is less than the preset step threshold. When min(l,r) ≥ 0 and the angle adjustment step < ε, it is determined that the adjustment is successful and the rotation angle update is completed; when When the corresponding x-axis rotation amplitude is 0, min(l,r)<0, it is determined that the adjustment has failed and the rotation angle update ends; otherwise, it is determined that the out-of-bounds condition of the first video frame does not meet the preset condition.

[0144] Furthermore, when the out-of-bounds situation is an up-down out-of-bounds situation, judging whether the out-of-bounds situation meets the preset conditions includes: judging whether min(t,b) is greater than or equal to zero, and whether the angle adjustment step step is less than the preset step threshold. When min(t,b)≥0 and the angle adjustment step step<ε, it is determined that the adjustment is successful and the rotation angle update is completed; when When the corresponding y-axis rotation amplitude is 0 and min(t,b)<0, it is determined that the adjustment has failed and the rotation angle update ends; otherwise, it is determined that the out-of-bounds situation does not meet the preset conditions.

[0145] Furthermore, when the out-of-bounds situation is left-right and up-down, judging whether the out-of-bounds situation meets the preset conditions includes: judging whether min(l,r) and min(t,b) are greater than or equal to zero, and whether the angle adjustment step step is less than the preset step threshold. When min(l,r)≥0, min(t,b)≥0 and the angle adjustment step step<ε, it is determined that the adjustment is successful and the initial rotation angle update is completed; when When the corresponding z-axis rotation amplitude is 0, min(l,r)<0 or min(t,b)<0, the adjustment is determined to have failed and the rotation angle update ends; otherwise, it is determined that the out-of-bounds situation does not meet the preset conditions.

[0146] In a specific implementation, the difference between the stabilization anti-crossing post-processing step and the horizon anti-crossing post-processing step is that before performing the anti-crossing post-processing according to the aforementioned horizon anti-crossing post-processing step, it is first determined whether l+r<0 or t+b<0 is satisfied. If l+r<0 or t+b<0 is satisfied, the z-axis rotation component is adjusted, and after the z-axis rotation component is adjusted, it is determined whether l+r<0 or t+b<0 is still satisfied. If l+r<0 or t+b<0 is still satisfied, the x-axis or y-axis rotation component is adjusted. After the x-axis or y-axis rotation component is adjusted, the anti-crossing post-processing is performed according to the aforementioned horizon anti-crossing post-processing step. If l+r<0 or t+b<0 is still satisfied after the processing, then according to Generate the corresponding rotation axis and adjust The rotation amplitude is until it reaches 0. When the rotation amplitude is 0, l+r<0 or t+b<0 is still satisfied, then let That is, the rolling shutter mapping of all row blocks is the same as that of the center row block.

[0147] In a specific implementation, the step of adjusting the z-axis rotation component specifically includes: Decompose to get the z-axis rotation angle; determine the reset angle based on the z-axis rotation angle; adjust the z-axis rotation angle to be closer to the reset angle to determine whether there is a rotation angle that satisfies l+r≥0 and t+b≥0. If there is a rotation angle that satisfies l+r≥0 and t+b≥0, return the updated If it does not exist, return the original The formula for determining the reset angle is: Represents the reset angle, θ z Indicates the z-axis rotation angle.

[0148] In a specific implementation, the step of adjusting the x-axis or y-axis rotation component specifically includes: Decompose and get the rotation angle Adjustment Move closer to zero and determine whether there is a rotation angle that satisfies l+r≥0 and t+b≥0. If there is a rotation angle that satisfies l+r≥0 and t+b≥0, return the updated If it does not exist, return the original

[0149] In a specific implementation, in step S44, after determining the post-processed video frame of the first video frame based on the third mapping information or the fourth mapping information, the method may further include: rendering the post-processed video frame to obtain a target rendered image. The rendering method includes but is not limited to GPU rendering, CPU rendering, etc.

[0150] In a specific implementation, when GPU rendering is used, the step of rendering the post-processed video frame specifically includes: obtaining vertex data V of the post-processed video frame; ij and vertex data V ij The corresponding texture coordinate T ij ; Determine the vertex data V ij The row block to which it belongs; based on the vertex data V ij Mapping information of the row block to which it belongs Vertex data V ij Perform vertex transformation to obtain the transformed vertex data Based on the transformed vertex data and texture coordinates T ij Draw the rendered image to obtain the target rendered image of the post-processed video frame. Among them, the texture coordinates T ij It can be expressed as: The rendered image consists of 2*L*C triangles, where i and j represent vertex data V ij The row and column of the render grid to which it belongs. L represents the number of rows in the render grid, and C represents the number of columns in the render grid.

[0151] In order to better implement the video processing method in the embodiment of the present application, based on the video processing method, the embodiment of the present application also provides a video processing device, such as Figure 5 As shown, the video processing device includes:

[0152] An information acquisition unit 610 is used to acquire shooting information of a first video frame and first correction information of a second video frame, where the second video frame is a historical video frame;

[0153] A parameter determination unit 620, configured to determine first stabilization information of a first video frame based on the shooting information;

[0154] The correction processing unit 630 is used to perform correction processing on the first stabilization information based on the shooting information and the first correction information to obtain a target video frame of the first video frame.

[0155] In an embodiment of the present application, first stabilization information of the first video frame is determined based on the shooting information, and the first stabilization information is corrected based on the shooting information and the first correction information. This not only enables uniform-speed stabilization of the first video frame, but also enables the stabilization result to be aligned with the horizon, thereby improving the stabilization effect of the first video frame.

[0156] In some embodiments of the present application, the parameter determination unit 620 is specifically used to:

[0157] Determining second stabilization information of the first video frame based on the shooting information;

[0158] Acquire a rotation angle of the second stabilization information and a rotation angle of the shooting information;

[0159] The rotation angle of the second stabilization information is constrained based on the rotation angle of the shooting information to obtain the first stabilization information of the first video frame.

[0160] In some embodiments of the present application, the correction processing unit 630 is specifically used to:

[0161] Based on the shooting information, determine frame state information of the first video frame, where the frame state information includes a static state and a non-static state;

[0162] If the frame state information is in a static state, fine-tuning and resetting the first stabilization information to obtain third stabilization information and first mapping information;

[0163] Determining target stabilization information for the first video frame based on the third stabilization information, the first mapping information, and the first correction information;

[0164] Based on the target stabilization information, a target video frame of the first video frame is determined.

[0165] In some embodiments of the present application, the first correction information includes a horizon alignment state, a horizon misalignment state, and a horizon alignment fine-tuning state. When the first correction information is the horizon alignment state, the correction processing unit 630 is further configured to:

[0166] Updating the third stabilization information to obtain updated third stabilization information;

[0167] Acquire gravity data of the first video frame, and determine a gravity calibration angle based on the gravity data;

[0168] Determining fourth stabilization information and a correction angle based on the updated third stabilization information, the first mapping information, and the gravity calibration angle;

[0169] Based on the correction angle, determining a first interpolation ratio;

[0170] The updated third stabilization information and the fourth stabilization information are interpolated based on the first interpolation ratio to obtain target stabilization information of the first video frame.

[0171] In some embodiments of the present application, when the first correction information is not in the horizon alignment state, the correction processing unit 630 is further configured to:

[0172] Determining fifth stabilization information and a mapping tilt angle based on the first mapping information, the third stabilization information, and the gravity calibration angle;

[0173] If the first correction information indicates a horizon misalignment state and the mapping tilt angle is greater than a preset first angle threshold, determining the third stabilization information as target stabilization information of the first video frame;

[0174] If the first correction information is in a horizon alignment fine-tuning state or the mapping tilt angle is not greater than the first angle threshold, the fifth stabilization information is updated, and the updated fifth stabilization information is determined as the target stabilization information of the first video frame.

[0175] In some embodiments of the present application, the video processing device further includes:

[0176] A parameter acquisition unit, used to acquire second correction information of the stabilized video frame, the second correction information including a horizon alignment state, a horizon misalignment state, and a horizon alignment fine-tuning state;

[0177] A first post-processing unit is configured to perform horizon correction and anti-crossing post-processing on the stabilized video frame if the second correction information is in a horizon alignment state, so as to obtain third mapping information;

[0178] A second post-processing unit is configured to perform stabilization and anti-crossing post-processing on the stabilized video frame to obtain fourth mapping information if the second correction information is not in the horizon alignment state or the horizon correction and anti-crossing post-processing fails;

[0179] The video determination unit is used to determine a post-processed video frame of the first video frame based on the third mapping information or the fourth mapping information.

[0180] In some embodiments of the present application, the first post-processing unit is specifically used for:

[0181] Determine fifth mapping information based on the stabilized video frame, and obtain the cross-border condition of the stabilized cropping area of ​​the first video frame;

[0182] Decomposing the fifth mapping information based on the cross-boundary condition to obtain a rotation angle;

[0183] updating the rotation angle, and updating the fifth mapping information based on the updated rotation angle to obtain updated fifth mapping information;

[0184] Determine, based on the updated fifth mapping information, a cross-border condition of the stabilized cropping area of ​​the first video frame;

[0185] When the cross-border situation does not meet the preset conditions, the rotation angle is updated, and the step of updating the fifth mapping information based on the rotation angle to obtain the updated fifth mapping information is continued until the cross-border situation meets the preset conditions to obtain the third mapping information.

[0186] The embodiment of the present application further provides a computer device, which integrates any one of the video processing devices provided in the embodiment of the present application, and the computer device includes:

[0187] one or more processors;

[0188] Memory; and

[0189] One or more applications, wherein the one or more applications are stored in the memory and are configured to execute, by the processor, the steps of the video processing method in any of the above-mentioned video processing method embodiments.

[0190] The present application also provides a computer device that integrates any video processing device provided in the present application. Figure 6 As shown, it shows a schematic diagram of the structure of the computer device involved in the embodiment of the present application, specifically:

[0191] The computer device may include one or more processing core processors 801, one or more computer-readable storage media memories 802, a power supply 803, an input unit 804 and other components. Those skilled in the art will appreciate that Figure 6 The computer device structure shown in the figure does not constitute a limitation on the computer device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. Among them:

[0192] The processor 801 is the control center of the computer device. It uses various interfaces and lines to connect various parts of the entire computer device. By running or executing software programs and / or modules stored in the memory 802 and calling data stored in the memory 802, it executes various functions of the computer device and processes data, thereby monitoring the computer device as a whole. Optionally, the processor 801 may include one or more processing cores; preferably, the processor 801 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 801.

[0193] The memory 802 can be used to store software programs and modules. The processor 801 executes various functional applications and data processing by running the software programs and modules stored in the memory 802. The memory 802 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 802 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 802 may also include a memory controller to provide the processor 801 with access to the memory 802.

[0194] The computer device also includes a power supply 803 for supplying power to each component. Preferably, the power supply 803 can be logically connected to the processor 801 through a power management system, so that the power management system can manage charging, discharging, power consumption and other functions. The power supply 803 can also include one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, power status indicators and other arbitrary components.

[0195] The computer device may further include an input unit 804, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0196] Although not shown, the computer device may further include a display unit, etc., which will not be described in detail herein. Specifically in this embodiment, the processor 801 in the computer device will load the executable files corresponding to the processes of one or more application programs into the memory 802 according to the following instructions, and the processor 801 will run the application programs stored in the memory 802, thereby realizing various functions, as follows:

[0197] Acquire shooting information of a first video frame and first correction information of a second video frame, where the second video frame is a historical video frame;

[0198] Determining first stabilization information of a first video frame based on the shooting information;

[0199] The first stabilization information is corrected based on the shooting information and the first correction information to obtain a target video frame of the first video frame.

[0200] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0201] To this end, the embodiment of the present application provides a computer-readable storage medium, which may include: a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc. A computer program is stored thereon, and the computer program is loaded by a processor to execute the steps in any video processing method provided in the embodiment of the present application. For example, the computer program loaded by the processor can execute the following steps:

[0202] Acquire shooting information of a first video frame and first correction information of a second video frame, where the second video frame is a historical video frame of the first video frame;

[0203] Determining first stabilization information of a first video frame based on the shooting information;

[0204] The first stabilization information is corrected based on the shooting information and the first correction information to obtain a target video frame of the first video frame.

[0205] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the detailed description of other embodiments above, and will not be repeated here.

[0206] In specific implementation, the above units or structures can be implemented as independent entities, or can be arbitrarily combined to be implemented as the same or several entities. The specific implementation of the above units or structures can refer to the previous method embodiments, which will not be repeated here.

[0207] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0208] The above is a detailed introduction to a video processing method, device, computer equipment and computer-readable storage medium provided in the embodiments of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the ideas of the present application, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A video processing method, characterized in that: include: Acquire shooting information of a first video frame and first correction information of a second video frame, where the second video frame is a historical video frame; Based on the shooting information, determining first stabilization information of the first video frame; The first stabilization information is corrected based on the shooting information and the first correction information to obtain a target video frame of the first video frame.

2. The method according to claim 1, characterized in that The determining, based on the shooting information, first stabilization information of the first video frame includes: Based on the shooting information, determining second stabilization information of the first video frame; Acquire a rotation angle of the second stabilization information and a rotation angle of the shooting information; The rotation angle of the second stabilization information is constrained based on the rotation angle of the shooting information to obtain the first stabilization information of the first video frame.

3. The method according to claim 1, characterized in that The correcting the first stabilization information based on the shooting information and the first correction information to obtain a target video frame of the first video frame includes: Based on the shooting information, determine frame state information of the first video frame, where the frame state information includes a stationary state and a non-stationary state; If the frame state information is in a static state, fine-tuning and resetting the first stabilization information to obtain third stabilization information and first mapping information; Determining target stabilization information of the first video frame based on the third stabilization information, the first mapping information, and the first correction information; Based on the target stabilization information, a target video frame of the first video frame is determined.

4. The method according to claim 3, characterized in that The first correction information includes a horizon alignment state, a horizon misalignment state, and a horizon alignment fine-tuning state. When the first correction information is the horizon alignment state, determining the target stabilization information of the first video frame based on the third stabilization information, the first mapping information, and the first correction information includes: Updating the third stabilization information to obtain updated third stabilization information; Acquire gravity data of the first video frame, and determine a gravity calibration angle based on the gravity data; determining fourth stabilization information and a correction angle based on the updated third stabilization information, the first mapping information, and the gravity calibration angle; Based on the correction angle, determining a first interpolation ratio; The updated third stabilization information and the fourth stabilization information are interpolated based on the first interpolation ratio to obtain target stabilization information of the first video frame.

5. The method according to claim 4, characterized in that When the first correction information is not in a horizon alignment state, determining target stabilization information of the first video frame based on the third stabilization information, the first mapping information, and the first correction information includes: determining fifth stabilization information and a mapping tilt angle based on the first mapping information, the third stabilization information and the gravity calibration angle; If the first correction information is a horizon misalignment state and the mapping tilt angle is greater than a preset first angle threshold, determining the third stabilization information as the target stabilization information of the first video frame; If the first correction information is in a horizon alignment fine-tuning state or the mapping tilt angle is not greater than the first angle threshold, the fifth stabilization information is updated, and the updated fifth stabilization information is determined as the target stabilization information of the first video frame.

6. The method according to claim 1, characterized in that After the first stabilization information is corrected based on the shooting information and the first correction information to obtain a target video frame of the first video frame, the method includes: Acquire second correction information of the target video frame, the second correction information including a horizon alignment state, a horizon misalignment state, and a horizon alignment fine-tuning state; If the second correction information is in a horizon alignment state, performing horizon correction and anti-crossing post-processing on the target video frame to obtain third mapping information; If the second correction information is not in the horizon alignment state or the horizon correction anti-crossing post-processing fails, performing stabilization anti-crossing post-processing on the target video frame to obtain fourth mapping information; Based on the third mapping information or the fourth mapping information, a post-processed video frame of the first video frame is determined.

7. The method according to claim 6, characterized in that The step of performing horizon correction and anti-crossing post-processing on the target video frame to obtain third mapping information includes: Determine fifth mapping information based on the target video frame, and obtain the crossing condition of the stabilized cropping area of ​​the first video frame; Decomposing the fifth mapping information based on the crossing condition to obtain a rotation angle; updating the rotation angle, and updating the fifth mapping information based on the updated rotation angle to obtain updated fifth mapping information; Determining, based on the updated fifth mapping information, a cross-border condition of the stabilized cropping area of ​​the first video frame; When the out-of-bounds situation does not meet the preset conditions, the rotation angle is updated, and the step of updating the fifth mapping information based on the rotation angle to obtain the updated fifth mapping information is continued until the out-of-bounds situation meets the preset conditions to obtain the third mapping information.

8. A video processing device, characterized in that: include: An information acquisition unit, used to acquire shooting information of a first video frame and first correction information of a second video frame, where the second video frame is a historical video frame; a parameter determination unit, configured to determine first stabilization information of the first video frame based on the shooting information; A correction processing unit is used to perform correction processing on the first stabilization information based on the shooting information and the first correction information to obtain a target video frame of the first video frame.

9. A computer device, characterized in that: The computer device comprises: one or more processors; Memory; and One or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the processor to implement the video processing method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and the computer program is loaded by a processor to execute the steps in the video processing method according to any one of claims 1 to 7.

Images