Video processing methods, apparatus, electronic devices and readable storage media

Abstract

This application discloses a video processing method, apparatus, electronic device, and readable storage medium, belonging to the field of image processing technology. When the first and third frames are clear frames and the second frame is a blurred frame, the method involves acquiring first shooting data corresponding to the first frame, second shooting data corresponding to the second frame, and third shooting data corresponding to the third frame; determining a first rotation matrix and a second rotation matrix based on the first, second, and third shooting data; determining a first mapped image based on the first frame and the first rotation matrix, and determining a second mapped image based on the third frame and the second rotation matrix; and replacing the blurred image region in the second frame with the first and second mapped images to obtain the target image.

Description

Technical Field

[0001] This application belongs to the field of image processing technology, and specifically relates to a video processing method, apparatus, electronic device, and readable storage medium. Background Technology

[0002] When shooting video at night, a longer exposure time is chosen to obtain sufficient light. However, choosing a longer exposure time will result in blurry frames in the captured video. Especially when a blurry frame suddenly appears in a series of clear frames, it significantly affects the viewing experience.

[0003] In related technical solutions, increasing the stabilization angle of optical image stabilization can improve blurry frames. However, the stabilization angle of optical image stabilization is limited and cannot guarantee that every frame is a clear frame. Another approach is to reduce the stabilization strength of electronic image stabilization, causing some shaking in the captured video to reduce the impact of blurry frames on the viewing experience. However, this method sacrifices the overall stability of the video, thus reducing the viewing experience. Summary of the Invention

[0004] The purpose of this application is to provide a video processing method, apparatus, electronic device, and readable storage medium that can solve the problem of a blurry frame suddenly appearing in a series of clear frames.

[0005] In a first aspect, embodiments of this application provide a video processing method, comprising: when a first frame image and a third frame image are clear frames and a second frame image is a blurred frame, acquiring first shooting data corresponding to the first frame image, second shooting data corresponding to the second frame image, and third shooting data corresponding to the third frame image, wherein the first frame image, the second frame image, and the third frame image are three consecutive frames in a video, and the first shooting data, the second shooting data, and the third shooting data all include Hall data and gyroscope data; determining a first rotation matrix and a second rotation matrix based on the first shooting data, the second shooting data, and the third shooting data; wherein the first rotation matrix is ​​used to represent the offset between the first frame image and the second frame image, and the second rotation matrix is ​​used to represent the offset between the third frame image and the second frame image; determining a first mapped image based on the first frame image and the first rotation matrix, and determining a second mapped image based on the third frame image and the second rotation matrix; and replacing the blurred image region in the second frame image based on the first mapped image and the second mapped image to obtain a target image.

[0006] Secondly, embodiments of this application provide a video processing apparatus, comprising: an acquisition module, configured to acquire first shooting data corresponding to the first frame image, second shooting data corresponding to the second frame image, and third shooting data corresponding to the third frame image when the first frame image and the third frame image are clear frames and the second frame image is a blurred frame, wherein the first frame image, the second frame image, and the third frame image are three consecutive frames in a video, and the first shooting data, the second shooting data, and the third shooting data all include Hall data and gyroscope data; a first determination module, configured to determine a first rotation matrix and a second rotation matrix based on the first shooting data, the second shooting data, and the third shooting data; wherein the first rotation matrix is ​​used to represent the offset between the first frame image and the second frame image, and the second rotation matrix is ​​used to represent the offset between the third frame image and the second frame image; a second determination module, configured to determine a first mapped image based on the first frame image and the first rotation matrix, and to determine a second mapped image based on the third frame image and the second rotation matrix; and a processing module, configured to replace the blurred image region in the second frame image based on the first mapped image and the second mapped image to obtain a target image.

[0007] Thirdly, embodiments of this application provide an electronic device including a processor and a memory, the memory storing a program or instructions that can run on the processor, the program or instructions implementing the steps of the method as described in the first aspect when executed by the processor.

[0008] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, and when the program or instructions are executed by a processor, they implement the steps of the method as described in the first aspect.

[0009] Fifthly, embodiments of this application provide a chip, which includes a processor and a communication interface. The communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the method as described in the first aspect.

[0010] In a sixth aspect, embodiments of this application provide a computer program product stored in a storage medium, which is executed by at least one processor to implement the method as described in the first aspect.

[0011] This application proposes a video processing method. In this method, if the first and third frames are clear frames and the second frame is blurry in three consecutive frames, the method acquires first shooting data corresponding to the first frame, second shooting data corresponding to the second frame, and third shooting data corresponding to the third frame. A first rotation matrix and a second rotation matrix are then determined using these three data. The first rotation matrix represents the offset between the first and second frames. Therefore, the first rotation matrix can be used to infer the content in the first frame that is identical to the content in the second frame, i.e., the first mapped image. Similarly, the second rotation matrix can be used to infer the content in the third frame that is identical to the content in the second frame, i.e., the second mapped image. Since the first and third frames are clear frames, the first and second mapped images are obviously also clear. By replacing the blurry image area in the second frame with the first and second mapped images, blurry frames in the video can be eliminated, thereby improving the video viewing experience. Attached Figure Description

[0012] Figure 1 A flowchart illustrating the video processing method provided in an embodiment of this application is shown;

[0013] Figure 2 A schematic diagram of the (N-1)th frame image in an embodiment of this application is shown;

[0014] Figure 3 A schematic diagram of the Nth frame image in an embodiment of this application is shown;

[0015] Figure 4 A schematic diagram of the (N+1)th frame image in an embodiment of this application is shown;

[0016] Figure 5 A schematic diagram of the first mapped image in an embodiment of this application is shown;

[0017] Figure 6 A schematic diagram of the second mapped image in an embodiment of this application is shown;

[0018] Figure 7 A schematic diagram of the replaced Nth frame image in an embodiment of this application is shown;

[0019] Figure 8 A schematic diagram of the completed first mapping image in an embodiment of this application is shown;

[0020] Figure 9 A schematic diagram of the completed second mapping image in an embodiment of this application is shown;

[0021] Figure 10A schematic diagram of the (N-1)th frame image in an embodiment of this application is shown;

[0022] Figure 11 A schematic diagram of the first mapped image in an embodiment of this application is shown;

[0023] Figure 12 A schematic diagram of the Nth frame image in an embodiment of this application is shown;

[0024] Figure 13 A schematic diagram of the image before offset in an embodiment of this application is shown;

[0025] Figure 14 A schematic diagram of the offset image in an embodiment of this application is shown;

[0026] Figure 15 This illustration shows a schematic diagram of the Nth frame image after fine-tuning in an embodiment of this application;

[0027] Figure 16 A schematic diagram showing the relationship between attitude angle and focal length in an embodiment of this application is shown;

[0028] Figure 17 A schematic diagram showing the first detection area, exposure time, start exposure time, and end exposure time in an embodiment of this application is provided.

[0029] Figure 18 This paper shows a schematic diagram of the overall flow of the video processing method provided in an embodiment of this application;

[0030] Figure 19 A schematic flowchart of the video processing apparatus provided in an embodiment of this application is shown;

[0031] Figure 20 This is a schematic block diagram of the electronic device in the embodiments of this application;

[0032] Figure 21 This is a schematic diagram of the hardware structure of the electronic device in the embodiments of this application. Detailed Implementation

[0033] The embodiments of this application will now be clearly described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of, and not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0034] The terms "first," "second," etc., used in this application's specification are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class, without limiting the number of objects; for example, a first object can be one or more. Furthermore, in the specification, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects have an "or" relationship.

[0035] The video processing method, apparatus, electronic device, and readable storage medium provided in this application will be described in detail below with reference to the accompanying drawings and through specific embodiments and application scenarios.

[0036] In one embodiment, such as Figure 1 As shown, a video processing method is proposed, including:

[0037] Step 102: If the first frame and the third frame are clear frames and the second frame is a blurry frame, obtain the first shooting data corresponding to the first frame, the second shooting data corresponding to the second frame, and the third shooting data corresponding to the third frame. The first frame, the second frame, and the third frame are three consecutive frames in the video. The first shooting data, the second shooting data, and the third shooting data all include Hall effect data and gyroscope data.

[0038] In some embodiments of this application, the first frame, the second frame, and the third frame are three consecutive frames in a captured video. They can be the first three frames in the video or any three consecutive frames in the video.

[0039] For example, if the video includes a sequentially consecutive first frame, second frame, third frame, fourth frame, ..., N-1th frame, Nth frame and N+1th frame, then the first frame, second frame and third frame can be the first frame, second frame and third frame respectively, or the N-1th frame, Nth frame and N+1th frame respectively.

[0040] Step 104: Determine a first rotation matrix and a second rotation matrix based on the first shooting data, the second shooting data, and the third shooting data. The first rotation matrix is ​​used to represent the offset between the first frame image and the second frame image, and the second rotation matrix is ​​used to represent the offset between the third frame image and the second frame image.

[0041] Step 106: Determine the first mapped image based on the first frame image and the first rotation matrix, and determine the second mapped image based on the third frame image and the second rotation matrix.

[0042] Step 108: Replace the blurred image region in the second frame image with the first mapped image and the second mapped image to obtain the target image.

[0043] This application proposes a video processing method. In this method, if the first and third frames are clear frames and the second frame is blurry in three consecutive frames, the method acquires first shooting data corresponding to the first frame, second shooting data corresponding to the second frame, and third shooting data corresponding to the third frame. A first rotation matrix and a second rotation matrix are then determined using these three data. The first rotation matrix represents the offset between the first and second frames. Therefore, the first rotation matrix can be used to infer the content in the first frame that is identical to the second frame, i.e., the first mapped image. Similarly, the second rotation matrix is ​​used to infer the content in the third frame that is identical to the second frame, i.e., the second mapped image. Since the first and third frames are clear frames, the first and second mapped images are obviously also clear. By replacing the blurry image area in the second frame with the first and second mapped images, blurry frames in the video can be eliminated, thereby improving the video viewing experience.

[0044] In some embodiments of this application, determining a first rotation matrix and a second rotation matrix based on first shooting data, second shooting data, and third shooting data specifically includes: determining a first position offset and a second position offset compensated by optical image stabilization for the first frame image based on the first shooting data; determining a third position offset and a fourth position offset compensated by optical image stabilization for the second frame image based on the second shooting data; determining a fifth position offset and a sixth position offset compensated by optical image stabilization for the third frame image based on the third shooting data; obtaining a first product value of the second position offset and a conversion coefficient, a second product value of the fourth position offset and a conversion coefficient, and a third product value of the sixth position offset and a conversion coefficient; obtaining a first difference between the first position offset and the first product value, a second difference between the third position offset and the second product value, and a third difference between the fifth position offset and the third product value; determining a first rotation matrix based on the first difference and the second difference, and determining a second rotation matrix based on the second difference and the third difference; wherein, the conversion coefficient is the scaling factor corresponding to the Hall data when converting Hall data into gyroscope data.

[0045] In this embodiment, the positional offset when capturing each frame of an image and the positional offset compensated by optical image stabilization for each frame of an image can be determined based on the shooting data. Specifically, the first positional offset and the second positional offset are the positional offsets corresponding to the first frame of an image, and the third positional offset and the fourth positional offset are the positional offsets corresponding to the second frame of an image. Therefore, after obtaining the first positional offset, the second positional offset, the third positional offset, and the fourth positional offset, the offset between the first frame of an image and the second frame of an image, that is, the first difference and the second difference, can be known based on the first positional offset, the second positional offset, the third positional offset, and the fourth positional offset. Then, the first rotation matrix can be determined based on the first difference and the second difference.

[0046] Similarly, the fifth position offset when the third frame image is captured and the sixth position offset that the optical image stabilization has compensated for in the third frame image can be obtained. Then, based on the third position offset, the fourth position offset, the fifth position offset and the sixth position offset, the offset between the second frame image and the third frame image, that is, the second difference and the third difference, can be known. Then, based on the second difference and the third difference, the second rotation matrix can be determined.

[0047] In this process, by determining the first rotation matrix and the second rotation matrix, the first frame image can be mapped to the second frame image, and the third frame image can be mapped to the second frame image, thus obtaining a clear second frame image.

[0048] In the above embodiments, the set conversion coefficient can convert the positional offset compensated by optical image stabilization into the positional offset when the image is captured. In this process, the data source can be unified to ensure the accuracy of the first rotation matrix and the second rotation matrix.

[0049] In some embodiments of this application, the first image capture data includes: multiple sets of first gyroscope data, multiple first timestamps corresponding one-to-one with the multiple sets of first gyroscope data, multiple sets of first Hall data, multiple second timestamps corresponding one-to-one with the multiple sets of first Hall data, each set of first gyroscope data including the rotation rate of the x-axis, the rotation rate of the y-axis, and the rotation rate of the z-axis in a spatial Cartesian coordinate system, and each set of first Hall data including the rotation rate of the x-axis, the rotation rate of the y-axis, and the rotation rate of the z-axis in a spatial Cartesian coordinate system; the second image capture data includes: multiple sets of second gyroscope data, multiple third timestamps corresponding one-to-one with the multiple sets of second gyroscope data, multiple sets of second Hall data, and multiple fourth timestamps corresponding one-to-one with the multiple sets of second Hall data. Each set of second gyroscope data includes the rotation rates of the x-axis, y-axis, and z-axis in a Cartesian coordinate system; each set of second Hall data includes the rotation rates of the x-axis, y-axis, and z-axis in a Cartesian coordinate system. The third set of captured data includes: multiple sets of third gyroscope data, multiple fifth timestamps corresponding one-to-one with the multiple sets of third gyroscope data, multiple sets of third Hall data, multiple sixth timestamps corresponding one-to-one with the multiple sets of third Hall data, each set of third gyroscope data including the rotation rates of the x-axis, y-axis, and z-axis in a Cartesian coordinate system, and each set of third Hall data including the rotation rates of the x-axis, y-axis, and z-axis in a Cartesian coordinate system.

[0050] Based on the first shooting data, a first position offset and a second position offset compensated by optical image stabilization for the first frame image are determined. Based on the second shooting data, a third position offset and a fourth position offset compensated by optical image stabilization for the second frame image are determined. Based on the third shooting data, a fifth position offset and a sixth position offset compensated by optical image stabilization for the third frame image are determined. Specifically, this includes: determining the first position offset based on multiple sets of first gyroscope data and multiple first timestamps; determining the second position offset based on multiple sets of first Hall data and multiple second timestamps; determining the third position offset based on multiple sets of second gyroscope data and multiple third timestamps; determining the fourth position offset based on multiple sets of second Hall data and multiple fourth timestamps; determining the fifth position offset based on multiple sets of third gyroscope data and multiple fifth timestamps; and determining the sixth position offset based on multiple sets of third Hall data and multiple sixth timestamps.

[0051] In this embodiment, during the shooting process, it takes a period of time to capture each frame of the image, and the shooting data will change during this period of time.

[0052] In the embodiments of this application, multiple sets of first gyroscope data, multiple first timestamps corresponding one-to-one with the multiple sets of first gyroscope data, multiple sets of first Hall data, and multiple second timestamps corresponding one-to-one with the multiple sets of first Hall data are acquired, so as to determine the first position offset and the second position offset based on the acquired multiple sets of data. In this process, the accuracy of the first position offset and the second position offset can be improved.

[0053] Specifically, let n be the number of frames obtained in the (N-1)th frame. N-1 Group 1 gyroscope data and n N-1 Group 1 Hall effect data, where the first gyroscope data, also known as Gyro data, is represented as... The first Hall data, also known as Hall data, is represented as in, This represents the Gyro data for the i-th group of x, y, and z axes. This represents the timestamp of the i-th group of Gyro data. This represents the Hall data for the i-th group of x, y, and z axes. This represents the timestamp of the i-th group of Hall data.

[0054] Then the first position offset s corresponding to the N-1th frame image N-1 The formula is as follows:

[0055]

[0056] The second position offset h corresponding to the N-1th frame image N-1 The formula is as follows:

[0057]

[0058] Among them, the (N-1)th frame image, which is also the first frame image in this application, This is the timestamp of the (i+1)th group of Gyro data, also known as the first timestamp. This is the timestamp of the (i+1)th group of Hall data, also known as the second timestamp.

[0059] Similarly, the second position offset s corresponding to the Nth frame image can be obtained. N and the third position offset h N The Nth frame image is also the second frame image in this application.

[0060] Based on this, the image offset between the two frames can be calculated as follows:

[0061] ΔX N,N-1 =s N -ch N -(s N x1-chN-1 )

[0062] Where, Δx N,N-1 represents the offset between the Nth frame and the (N-1)th frame, and c is the conversion coefficient.

[0063] Specifically, when photographing an object at infinity, it is assumed that the movement of the electronic device is only rotation, not translation; therefore, the first rotation matrix R... N-1,N The relationship between the image offset of frame N and frame N-1 is as follows:

[0064] s N -ch N =R N-1，N (s N-1 -ch N-1 )

[0065] Among them, R N-1，N The expression is as follows:

[0066]

[0067] Where, γ N-1,N β N-1，N α N-1,N R is the first rotation matrix N-1，N The three offsets in Let α represent the components along the x, y, and z axes, and let cosα, sinα, cosβ, sinβ, cosγ, and sinγ be the sin and cosine function values ​​of α, β, and γ, respectively. α, β, and γ are the unknowns to be solved.

[0068] Specifically, s N -ch N It is a 3×1 vector with three values, R. N-1，N It is a 3×3 matrix, s N-1 -ch N-1 It is a 3×1 vector, so R N-1，N (s N-1 -ch N-1 ) is also a 3×1 vector.

[0069] s N -ch N =R N-1，N (s N-1 -ch N-1 )

[0070] In the above equation, both the left and right sides are 3×1 vectors. Therefore, the three equations can be solved to obtain the three unknowns α, β, and γ, and thus the first rotation matrix R. N-1,N .

[0071] Similarly, the second rotation matrix R can be calculated. N+1，N .

[0072] Let I be the image of the (N-1)th frame. N-1 The image in frame N+1 is I N+1 Then the actual image when mapping the (N-1)th frame image to the Nth frame image is I. N-1，N That is, the first mapped image in this application.

[0073] Then, I N-1，N =R N-1,N I N-1 .

[0074] Similarly, we can obtain the true image I when the (N+1)th frame image is mapped to the Nth frame image. N+1，N That is, the second mapped image in this application.

[0075] Then, I N+1，N =R N+1,N I N+1 .

[0076] In the N+1 and N-1 frames, the missing parts are filled with black borders.

[0077] Figure 2 A schematic diagram of the (N-1)th frame image in an embodiment of this application is shown. Figure 3 This illustration shows a schematic diagram of the Nth frame image in an embodiment of this application. Figure 4 This illustration shows a schematic diagram of the (N+1)th frame image in an embodiment of this application. Figure 5 A schematic diagram of the first mapped image in an embodiment of this application is shown. Figure 6 A schematic diagram of the second mapping image proposed in an embodiment of this application is shown. Figure 7 A schematic diagram of the replaced Nth frame image in an embodiment of this application is shown, wherein, Figure 5 The black borders of the first mapped image shown are padded after offset, resulting in the image shown below. Figure 8 The image shown, Figure 8 This illustration shows a schematic diagram of the completed first mapping image in an embodiment of this application. Similarly, Figure 6 The black borders of the second mapped image shown are padded after offset, resulting in the following: Figure 9 The image shown, Figure 9 A schematic diagram of the completed second mapping image in an embodiment of this application is shown.

[0078] like Figure 2 , Figure 3 , Figure 4 , Figure 5 , Figure 6 , Figure 7 , Figure 8 and Figure 9 As shown, Figure 2 , Figure 3 , Figure 4 After applying the video processing method proposed in this application to the image shown, the result is as follows: Figure 5 The first mapped image shown and as Figure 6 The second mapped image shown, and as... Figure 7 The image shown is the replaced Nth frame, and the first and second mapped images, after completion, are as follows: Figure 8 and Figure 9 The image shown.

[0079] For example, Figure 10 A schematic diagram of the (N-1)th frame image in an embodiment of this application is shown. Figure 11 A schematic diagram of the first mapped image in an embodiment of this application is shown. Figure 12 A schematic diagram of the Nth frame image in an embodiment of this application is shown, as follows: Figure 10 , Figure 11 and Figure 12 As shown, it can be done through Figure 10 and Figure 12 Calculated Figure 11 .

[0080] In some embodiments of this application, a target image is obtained by replacing blurred image regions in a second frame image with a first mapped image and a second mapped image. Specifically, this includes: obtaining a first affine function between the partitioned first mapped image and the partitioned second frame image, and a second affine function between the partitioned second mapped image and the partitioned second frame image; determining a first linear optimization model based on the first affine function and a first constraint condition; determining a second linear optimization model based on the second affine function and the second constraint condition; solving the first linear optimization model to obtain a first offset, and solving the second linear optimization model to obtain a second offset; matching the first mapped image and the second frame image based on the first offset, and matching the second mapped image and the second frame image based on the second offset; and replacing blurred image regions in the second frame image with the matched first mapped image and second mapped image to obtain the target image.

[0081] In some embodiments of this application, the method further includes: performing mesh partitioning on the first mapped image, the second mapped image, and the second frame image using a preset mesh to obtain the partitioned first image, the partitioned second image, and the partitioned second frame image.

[0082] In this embodiment, a first offset and a second offset are calculated so that the first and second mapped images can be fine-tuned based on the first and second offsets, thereby eliminating distortion caused by shooting from different positions and reducing the distortion of the target image.

[0083] Specifically, when When I = 1, it means that the gradient at that point is completely consistent with the gradient of the Nth frame image, where I = 1. N (i,j) is the pixel with coordinates (i,j) in the Nth frame of the image. It is an affine function. It is the product of the pixel with coordinates (i,j) in the first mapped image and the affine function, that is, when When it is at its minimum, it represents the first mapped image I. N-1,N With the Nth frame image I N Alignment has been maximized, among which, The affine function representing the grid containing the pixel coordinate (i,j) in the image is shown in the following expression:

[0084]

[0085] c 00 c 01 c 02 c 10 c 11 and c 12 Here, i and j are the affine parameters, and i and j are the pixel coordinates. Each grid has its own affine parameters. To ensure grid continuity, the intersections of two grids must be equal, i.e., there is a constraint, which is the first constraint condition in this application, specifically expressed as:

[0086]

[0087] in, and This represents the values ​​at gridx and gridx+1. and The values ​​at gridy and gridy+1 are represented, w and h are the values ​​of i and j, gridx represents the x-coordinate of the grid point after meshing, and gridy represents the y-coordinate of the grid point after meshing.

[0088] The first linear optimization model is expressed as follows:

[0089]

[0090] By solving the first linear optimization model, the first offset is obtained. Similarly, by solving the second linear optimization model, the second offset is obtained.

[0091] Specifically, the first linear optimization model is a linear programming model. Since the first and second mapping images have been determined, the solution range can be set, and the local optimal solution can be found by enumeration to obtain the first offset.

[0092] After solving the first linear optimization model, we can obtain the part that is consistent between the first mapped image and the second frame image. Therefore, we can replace the corresponding image in the second frame image with the first mapped image. Similarly, we can replace the corresponding image in the second frame image with the second mapped image to obtain the replaced second frame image, so that the second frame image is transformed from a blurry frame to a clear frame.

[0093] For example, if the input image size is 1920×1080 and it is divided into a 27×35 grid, then the length and width of a grid are approximately 54 and 40 respectively. Therefore, by enumerating the values ​​between x-25 and x+25, and y-20 and y+20 of the grid points, the minimum value can be found, and the grid point positions on the image after fine matching can be determined. Then, the color of each pixel in each grid can be calculated based on the grid point positions to obtain the image after fine matching.

[0094] Figure 13 A schematic diagram of the image before offset in an embodiment of this application is shown. Figure 14 A schematic diagram of the offset image in an embodiment of this application is shown.

[0095] Image changes within a grid can be controlled by fixing points on the grid, such as... Figure 13 and Figure 14 As shown, image distortion is altered by offsetting, thereby achieving fine image matching.

[0096] Figure 15 A schematic diagram of the replaced Nth frame image in an embodiment of this application is shown, as follows. Figure 15 As shown, compared to Figure 3 The image shown demonstrates a significant improvement in image clarity.

[0097] In some embodiments of this application, before obtaining the first shooting data corresponding to the first frame image, the second shooting data corresponding to the second frame image, and the third shooting data corresponding to the third frame image, the video processing method further includes: obtaining the shooting focal length corresponding to the second frame image and obtaining the angle corresponding to the second frame image, wherein the angle corresponding to the second frame image is the attitude angle without optical image stabilization compensation; obtaining the product value of the angle corresponding to the second frame image and the shooting focal length; and determining that the second frame image is a blurred frame if the product value is greater than a comparison threshold.

[0098] In this embodiment, the shooting focal length corresponding to the second frame image and the angle corresponding to the second frame image are obtained in order to calculate the product value of the shooting focal length corresponding to the second frame image and the angle corresponding to the second frame image. The product value can objectively characterize the blurring of the second frame image. By setting a comparison threshold, it is determined whether the second frame image is a blurry frame based on the comparison result of the product value and the comparison threshold.

[0099] During this process, blurry frames in the video can be accurately identified and automatically replaced to improve the video viewing experience.

[0100] Specifically, Figure 16 A schematic diagram showing the relationship between attitude angle and focal length in an embodiment of this application is shown, such as... Figure 16 As shown, the angle corresponding to the second frame image, that is, the attitude angle, is the angle of lens rotation during the exposure time. The focal length and the amount of blur are geometrically related as the two legs of a right triangle, and there is a corresponding relationship of trigonometric function tan(attitude angle) = blur amount / focal length. Since the attitude angle at the millisecond level is very small, it can be approximated as attitude angle = blur amount / focal length. Then, the blur amount = attitude angle × focal length, where the blur amount is also the product value in this embodiment.

[0101] In some embodiments of this application, the comparison threshold can be set according to actual usage needs.

[0102] For example, at a resolution of 1080p, a product value between 4 and 10 pixels is considered to be heavily blurred, and a product value greater than 10 pixels is also considered to be heavily blurred, where a pixel is a unit of area.

[0103] In some embodiments of this application, obtaining the angle corresponding to the second frame image specifically includes: determining a first detection region in the second frame image, the first detection region having a corresponding start exposure time and end exposure time; determining a first starting posture based on the start exposure time, exposure time, and gyroscope data in the second shooting data; determining a first ending posture based on the end exposure time, exposure time, and gyroscope data in the second shooting data; determining a second starting posture based on the start exposure time, exposure time, and Hall data in the second shooting data; determining a second ending posture based on the end exposure time, exposure time, and Hall data in the second shooting data; determining a first posture difference based on the first ending posture and the first starting posture; determining a second posture difference based on the second ending posture and the second starting posture; and determining the difference between the first posture difference and the second posture difference as the angle corresponding to the second frame image.

[0104] In this embodiment, a first detection region can be selected in the second frame image. The first detection region corresponds to the start exposure time and the end exposure time. Based on the start exposure time and the end exposure time, the gyroscope data in the second shooting data and the Hall data in the second shooting data are calculated to obtain the first attitude difference and the second attitude difference. The difference between the first attitude difference and the second attitude difference is used as the angle corresponding to the second frame image.

[0105] During this process, the corresponding angle of the second frame image can be automatically calculated based on gyroscope data and Hall data, and the attitude angle that is not compensated by optical image stabilization can be accurately evaluated.

[0106] Specifically, the first initial posture is GyroPos t0 The first ending posture is indicated by GyroPos. t1 This indicates that GyroPos t0 The expression is as follows:

[0107]

[0108] GyroPos t1 The expression is as follows:

[0109]

[0110] Among them, ROI t1 It's the moment the exposure ends, ROI t0 It is the initial exposure time, Te xp It is the exposure time, and gyro(t) is the gyroscope data that changes with time t.

[0111] The first pose difference GyroDiffROI is then expressed as follows:

[0112] GyroDiffROI = GyroPos t1 -GyroPos t0

[0113] Using the same principle, the second pose difference can be calculated, and then the angle corresponding to the second frame image can be calculated.

[0114] In the above embodiments, the size of the first detection region is half the size of the second frame image. Therefore, in the above embodiments, in the ROI... t0 Add T on the basis exp / 2, in ROI t1 Reduce T on the basis exp / 2.

[0115] Figure 17A schematic diagram illustrating the first detection area, exposure time, start exposure time, and end exposure time in an embodiment of this application is shown, as follows: Figure 17 As shown, the first detection region can be extracted from the second frame image, thereby calculating the angle corresponding to the second frame image.

[0116] Figure 18 This document illustrates the overall flowchart of the video processing method provided in an embodiment of this application. Figure 18 As shown, by running the video processing method proposed in this application, the replaced second frame image, i.e., I~(N), can be obtained.

[0117] The video processing method provided in this application can be executed by a video processing device. This application uses a video processing device executing the video processing method as an example to illustrate the video processing device provided in this application.

[0118] In some embodiments of this application, such as Figure 19 As shown, a video processing apparatus 1900 is provided, comprising: an acquisition module 1902, configured to acquire first shooting data corresponding to the first frame image, second shooting data corresponding to the second frame image, and third shooting data corresponding to the third frame image when the first frame image and the third frame image are clear frames and the second frame image is a blurry frame, wherein the first frame image, the second frame image, and the third frame image are three consecutive frames in a video, and the first shooting data, the second shooting data, and the third shooting data all include Hall effect data and gyroscope data; a first determination module 1904, configured to determine a first rotation matrix and a second rotation matrix based on the first shooting data, the second shooting data, and the third shooting data; wherein the first rotation matrix is ​​used to represent the offset between the first frame image and the second frame image, and the second rotation matrix is ​​used to represent the offset between the third frame image and the second frame image; a second determination module 1906, configured to determine a first mapped image based on the first frame image and the first rotation matrix, and to determine a second mapped image based on the third frame image and the second rotation matrix; and a processing module 1908, configured to replace the blurry image region in the second frame image based on the first mapped image and the second mapped image to obtain a target image.

[0119] In this embodiment, if the first and third frames are clear frames and the second frame is blurry in three consecutive frames, the system acquires first shooting data corresponding to the first frame, second shooting data corresponding to the second frame, and third shooting data corresponding to the third frame. A first rotation matrix is ​​determined using the first and second shooting data, and a second rotation matrix is ​​determined using the second and third shooting data. The first rotation matrix represents the offset between the first and second frames. Therefore, the first rotation matrix can be used to infer the content in the first frame that is identical to the content in the second frame, i.e., the first mapped image. Similarly, the second rotation matrix can be used to infer the content in the third frame that is identical to the content in the second frame, i.e., the second mapped image. Since the first and third frames are clear frames, the first and second mapped images are obviously also clear. By replacing the blurry image area in the second frame with the first and second mapped images, blurry frames in the video can be eliminated, thereby improving the video viewing experience.

[0120] In some embodiments of this application, the first determining module 1904 is specifically configured to: determine a first position offset and a second position offset compensated by optical image stabilization for the first frame image based on the first shooting data; determine a third position offset and a fourth position offset compensated by optical image stabilization for the second frame image based on the second shooting data; determine a fifth position offset and a sixth position offset compensated by optical image stabilization for the third frame image based on the third shooting data; obtain a first product value of the second position offset and the conversion coefficient, a second product value of the fourth position offset and the conversion coefficient, and a third product value of the sixth position offset and the conversion coefficient; obtain a first difference between the first position offset and the first product value, a second difference between the third position offset and the second product value, and a third difference between the fifth position offset and the third product value; determine a first rotation matrix based on the first difference and the second difference, and determine a second rotation matrix based on the second difference and the third difference; wherein, the conversion coefficient is the scaling factor corresponding to the Hall data when converting Hall data into gyroscope data.

[0121] In some embodiments of this application, the first image capture data includes: multiple sets of first gyroscope data, multiple first timestamps corresponding one-to-one with the multiple sets of first gyroscope data, multiple sets of first Hall data, multiple second timestamps corresponding one-to-one with the multiple sets of first Hall data, each set of first gyroscope data including the rotation rate of the x-axis, the rotation rate of the y-axis, and the rotation rate of the z-axis in a spatial rectangular coordinate system, and each set of first Hall data including the rotation rate of the x-axis, the rotation rate of the y-axis, and the rotation rate of the z-axis in a spatial rectangular coordinate system; the second image capture data includes: multiple sets of second gyroscope data, multiple third timestamps corresponding one-to-one with the multiple sets of second gyroscope data, multiple sets of second Hall data, multiple fourth timestamps corresponding one-to-one with the multiple sets of second Hall data, each set of second gyroscope data including the rotation rate of the x-axis, the rotation rate of the y-axis, and the rotation rate of the z-axis in a spatial rectangular coordinate system, and each set of second Hall data including the rotation rate of the x-axis, the rotation rate of the y-axis, and the rotation rate of the z-axis in a spatial rectangular coordinate system; the third image capture data includes: multiple The system comprises: a set of third gyroscope data, multiple fifth timestamps corresponding one-to-one with the sets of third gyroscope data, multiple sets of third Hall data, multiple sixth timestamps corresponding one-to-one with the sets of third Hall data, and each set of third gyroscope data including the rotation rates of the x-axis, y-axis, and z-axis in a spatial Cartesian coordinate system; and each set of third Hall data including the rotation rates of the x-axis, y-axis, and z-axis in a spatial Cartesian coordinate system. The first determining module 1904 is specifically used for: determining a first position offset based on the sets of first gyroscope data and multiple first timestamps; determining a second position offset based on the sets of first Hall data and multiple second timestamps; determining a third position offset based on the sets of second gyroscope data and multiple third timestamps; determining a fourth position offset based on the sets of second Hall data and multiple fourth timestamps; determining a fifth position offset based on the sets of third gyroscope data and multiple fifth timestamps; and determining a sixth position offset based on the sets of third Hall data and multiple sixth timestamps.

[0122] In some embodiments of this application, the processing module 1908 is specifically configured to: obtain a first affine function between the partitioned first mapped image and the partitioned second frame image, and a second affine function between the partitioned second mapped image and the partitioned second frame image; determine a first linear optimization model based on the first affine function and a first constraint condition; determine a second linear optimization model based on the second affine function and the second constraint condition; solve the first linear optimization model to obtain a first offset, and solve the second linear optimization model to obtain a second offset; match the first mapped image and the second frame image based on the first offset, and match the second mapped image and the second frame image based on the second offset; replace the blurred image region in the second frame image with the matched first mapped image and the second mapped image to obtain a target image.

[0123] In some embodiments of this application, before acquiring the first shooting data corresponding to the first frame image, the second shooting data corresponding to the second frame image, and the third shooting data corresponding to the third frame image, the acquisition module 1902 is further configured to: acquire the shooting focal length corresponding to the second frame image, and acquire the angle corresponding to the second frame image, wherein the angle corresponding to the second frame image is the attitude angle without optical image stabilization compensation; and acquire the product value of the angle corresponding to the second frame image and the shooting focal length; the processing module 1908 is further configured to: determine that the second frame image is a blurred frame if the product value is greater than a comparison threshold.

[0124] In some embodiments of this application, the acquisition module 1902 is specifically used for: determining a first detection region in a second frame image, the first detection region having a corresponding start exposure time and end exposure time; determining a first starting posture based on the start exposure time, exposure time, and gyroscope data in the second shooting data; determining a first ending posture based on the end exposure time, exposure time, and gyroscope data in the second shooting data; determining a second starting posture based on the start exposure time, exposure time, and Hall data in the second shooting data; determining a second ending posture based on the end exposure time, exposure time, and Hall data in the second shooting data; determining a first posture difference based on the first ending posture and the first starting posture; determining a second posture difference based on the second ending posture and the second starting posture; and determining the difference between the first posture difference and the second posture difference as the angle corresponding to the second frame image.

[0125] The video processing device in this application embodiment can be an electronic device or a component within an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television set (TV), ATM, or self-service machine, etc. This application embodiment does not specifically limit the device.

[0126] The video processing device in this application embodiment can be a device with an operating system. This operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit the specific operating system used.

[0127] The video processing device provided in this application embodiment can achieve... Figure 1 The various processes implemented in the video processing method embodiments can achieve the same technical effect, and will not be described again here to avoid repetition.

[0128] like Figure 20 As shown, this application embodiment also provides an electronic device 2000, including a processor 2002 and a memory 2004. The memory 2004 stores a program or instructions that can run on the processor 2002. When the program or instructions are executed by the processor 2002, they implement the various steps of the above-described video processing method embodiment and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0129] It should be noted that the electronic devices in the embodiments of this application include the aforementioned mobile electronic devices and non-mobile electronic devices.

[0130] Figure 21 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of this application.

[0131] like Figure 21 As shown, the electronic device 2100 includes, but is not limited to, components such as: radio frequency unit 2101, network module 2102, audio output unit 2103, input unit 2104, sensor 2105, display unit 2106, user input unit 2107, interface unit 2108, memory 2109, and processor 2110.

[0132] Those skilled in the art will understand that the electronic device 2100 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 2110 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 21 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0133] In some embodiments of this application, the processor 2110 is configured to: acquire first shooting data corresponding to the first frame image, second shooting data corresponding to the second frame image, and third shooting data corresponding to the third frame image when the first frame image and the third frame image are clear frames and the second frame image is a blurry frame, wherein the first frame image, the second frame image, and the third frame image are three consecutive frames in a video, and the first shooting data, the second shooting data, and the third shooting data all include Hall data and gyroscope data; determine a first rotation matrix and a second rotation matrix based on the first shooting data, the second shooting data, and the third shooting data; wherein the first rotation matrix is ​​used to represent the offset between the first frame image and the second frame image, and the second rotation matrix is ​​used to represent the offset between the third frame image and the second frame image; determine a first mapped image based on the first frame image and the first rotation matrix, and determine a second mapped image based on the third frame image and the second rotation matrix; and replace the blurry image region in the second frame image with the first mapped image and the second mapped image to obtain a target image.

[0134] In some embodiments of this application, the processor 2110 is specifically configured to: determine a first position offset and a second position offset compensated by optical image stabilization for the first frame image based on the first shooting data; determine a third position offset and a fourth position offset compensated by optical image stabilization for the second frame image based on the second shooting data; determine a fifth position offset and a sixth position offset compensated by optical image stabilization for the third frame image based on the third shooting data; obtain a first product value of the second position offset and a conversion coefficient, a second product value of the fourth position offset and a conversion coefficient, and a third product value of the sixth position offset and a conversion coefficient; obtain a first difference between the first position offset and the first product value, a second difference between the third position offset and the second product value, and a third difference between the fifth position offset and the third product value; determine a first rotation matrix based on the first difference and the second difference, and determine a second rotation matrix based on the second difference and the third difference; wherein, the conversion coefficient is the scaling factor corresponding to the Hall data when converting Hall data into gyroscope data.

[0135] In some embodiments of this application, the first image capture data includes: multiple sets of first gyroscope data, multiple first timestamps corresponding one-to-one with the multiple sets of first gyroscope data, multiple sets of first Hall data, multiple second timestamps corresponding one-to-one with the multiple sets of first Hall data, each set of first gyroscope data including the rotation rate of the x-axis, the rotation rate of the y-axis, and the rotation rate of the z-axis in a spatial Cartesian coordinate system, and each set of first Hall data including the rotation rate of the x-axis, the rotation rate of the y-axis, and the rotation rate of the z-axis in a spatial Cartesian coordinate system; the second image capture data includes: multiple sets of second gyroscope data, multiple third timestamps corresponding one-to-one with the multiple sets of second gyroscope data, multiple sets of second Hall data, multiple fourth timestamps corresponding one-to-one with the multiple sets of second Hall data, each set of second gyroscope data including the rotation rate of the x-axis, the rotation rate of the y-axis, and the rotation rate of the z-axis in a spatial Cartesian coordinate system, and each set of second Hall data including the rotation rate of the x-axis, the rotation rate of the y-axis, and the rotation rate of the z-axis in a spatial Cartesian coordinate system; the third image capture data includes... The processor 2110 is specifically used to: determine a first position offset based on multiple sets of first gyroscope data and multiple first timestamps; determine a second position offset based on multiple sets of first Hall data and multiple second timestamps; determine a third position offset based on multiple sets of second gyroscope data and multiple third timestamps; determine a fourth position offset based on multiple sets of second Hall data and multiple fourth timestamps; determine a fifth position offset based on multiple sets of third gyroscope data and multiple fifth timestamps; and determine a sixth position offset based on multiple sets of third Hall data and multiple sixth timestamps.

[0136] In some embodiments of this application, the processor 2110 is specifically configured to: obtain a first affine function between the partitioned first mapped image and the partitioned second frame image, and a second affine function between the partitioned second mapped image and the partitioned second frame image; determine a first linear optimization model based on the first affine function and a first constraint condition; determine a second linear optimization model based on the second affine function and a second constraint condition; solve the first linear optimization model to obtain a first offset, and solve the second linear optimization model to obtain a second offset; match the first mapped image and the second frame image based on the first offset, and match the second mapped image and the second frame image based on the second offset; replace the blurred image region in the second frame image with the matched first mapped image and the second mapped image to obtain a target image.

[0137] In some embodiments of this application, before acquiring the first shooting data corresponding to the first frame image, the second shooting data corresponding to the second frame image, and the third shooting data corresponding to the third frame image, the processor 2110 is further configured to: acquire the shooting focal length corresponding to the second frame image, and acquire the angle corresponding to the second frame image, wherein the angle corresponding to the second frame image is the attitude angle without optical image stabilization compensation; acquire the product value of the angle corresponding to the second frame image and the shooting focal length; and determine that the second frame image is a blurred frame if the product value is greater than a comparison threshold.

[0138] In some embodiments of this application, the processor 2110 is specifically configured to: determine a first detection region in a second frame image, the first detection region having a corresponding start exposure time and end exposure time; determine a first starting posture based on the start exposure time, exposure time, and gyroscope data in the second shooting data; determine a first ending posture based on the end exposure time, exposure time, and gyroscope data in the second shooting data; determine a second starting posture based on the start exposure time, exposure time, and Hall data in the second shooting data; determine a second ending posture based on the end exposure time, exposure time, and Hall data in the second shooting data; determine a first posture difference based on the first ending posture and the first starting posture; determine a second posture difference based on the second ending posture and the second starting posture; and determine the difference between the first posture difference and the second posture difference as the angle corresponding to the second frame image.

[0139] In this embodiment, if the first and third frames are detected as clear frames and the second frame is a blurry frame in three consecutive frames, the system acquires the first shooting data corresponding to the first frame, the second shooting data corresponding to the second frame, and the third shooting data corresponding to the third frame. A first rotation matrix and a second rotation matrix are then determined using these three data. The first rotation matrix represents the offset between the first and second frames. Therefore, the first rotation matrix can be used to infer the content in the first frame that is identical to the content in the second frame, i.e., the first mapped image. Similarly, the second rotation matrix can be used to infer the content in the third frame that is identical to the content in the second frame, i.e., the second mapped image. Since the first and third frames are clear frames, the first and second mapped images are obviously also clear. By replacing the blurry image area in the second frame with the first and second mapped images, blurry frames in the video can be eliminated, thereby improving the video viewing experience.

[0140] It should be understood that, in this embodiment, the input unit 2104 may include a graphics processing unit (GPU) 21041 and a microphone 21042. The GPU 21041 processes image data of still images or videos obtained by an image capture device (such as a camera) in video capture mode or image capture mode. The display unit 2106 may include a display panel 21061, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, etc. The user input unit 2107 includes at least one of a touch panel 21071 and other input devices 21072. The touch panel 21071 is also called a touch screen. The touch panel 21071 may include a touch detection device and a touch controller. Other input devices 21072 may include, but are not limited to, physical keyboards, function keys (such as volume control buttons, power buttons, etc.), trackballs, mice, and joysticks, which will not be described in detail here.

[0141] The memory 2109 can be used to store software programs and various data. The memory 2109 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, the memory 2109 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 2109 in the embodiments of this application includes, but is not limited to, these and any other suitable types of memory.

[0142] Processor 2110 may include one or more processing units; optionally, processor 2110 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into processor 2110.

[0143] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described video processing method embodiments and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0144] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0145] This application also provides a chip, which includes a processor and a communication interface. The communication interface and the processor are coupled. The processor is used to run programs or instructions to implement the various processes of the above-described video processing method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0146] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0147] This application provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the various processes of the video processing method embodiments described above, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0148] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0149] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the embodiments of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of this application.

[0150] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A video processing method, characterized in that, include: When the first and third frames are clear frames and the second frame is a blurry frame, the first shooting data corresponding to the first frame, the second shooting data corresponding to the second frame, and the third shooting data corresponding to the third frame are obtained. The first frame, the second frame, and the third frame are three consecutive frames in the video. The first shooting data, the second shooting data, and the third shooting data all include Hall effect data and gyroscope data. A first rotation matrix and a second rotation matrix are determined based on the first shooting data, the second shooting data, and the third shooting data; wherein, the first rotation matrix is ​​used to represent the offset between the first frame image and the second frame image, and the second rotation matrix is ​​used to represent the offset between the third frame image and the second frame image; A first mapped image is determined based on the first frame image and the first rotation matrix, and a second mapped image is determined based on the third frame image and the second rotation matrix; The target image is obtained by replacing the blurred image region in the second frame image with the first mapped image and the second mapped image; The determination of the first rotation matrix and the second rotation matrix based on the first shooting data, the second shooting data, and the third shooting data specifically includes: Based on the first shooting data, a first position offset and a second position offset that has been compensated for by optical image stabilization when shooting the first frame image are determined. Based on the second shooting data, a third position offset and a fourth position offset that has been compensated for by optical image stabilization when shooting the second frame image are determined. Based on the third shooting data, a fifth position offset and a sixth position offset that has been compensated for by optical image stabilization when shooting the third frame image are determined. Obtain the first product of the second position offset and the conversion coefficient, the second product of the fourth position offset and the conversion coefficient, and the third product of the sixth position offset and the conversion coefficient; Obtain the first difference between the first position offset and the first product value, the second difference between the third position offset and the second product value, and the third difference between the fifth position offset and the third product value; A first rotation matrix is ​​determined based on the first difference and the second difference, and a second rotation matrix is ​​determined based on the second difference and the third difference; The conversion coefficient is the scaling factor corresponding to the Hall data when converting the Hall data into gyroscope data.

2. The video processing method according to claim 1, characterized in that, The first captured data includes: multiple sets of first gyroscope data, multiple first timestamps corresponding one-to-one with the multiple sets of first gyroscope data, multiple sets of first Hall data, multiple second timestamps corresponding one-to-one with the multiple sets of first Hall data, each set of first gyroscope data including the rotation rate of the x-axis, the rotation rate of the y-axis, and the rotation rate of the z-axis in a spatial rectangular coordinate system, and each set of first Hall data including the rotation rate of the x-axis, the rotation rate of the y-axis, and the rotation rate of the z-axis in a spatial rectangular coordinate system; The second captured data includes: multiple sets of second gyroscope data, multiple third timestamps corresponding one-to-one with the multiple sets of second gyroscope data, multiple sets of second Hall data, multiple fourth timestamps corresponding one-to-one with the multiple sets of second Hall data, each set of second gyroscope data including the rotation rate of the x-axis, the rotation rate of the y-axis, and the rotation rate of the z-axis in a spatial rectangular coordinate system, and each set of second Hall data including the rotation rate of the x-axis, the rotation rate of the y-axis, and the rotation rate of the z-axis in a spatial rectangular coordinate system; The third captured data includes: multiple sets of third gyroscope data, multiple fifth timestamps corresponding one-to-one with the multiple sets of third gyroscope data, multiple sets of third Hall data, multiple sixth timestamps corresponding one-to-one with the multiple sets of third Hall data, each set of third gyroscope data including the rotation rate of the x-axis, the rotation rate of the y-axis, and the rotation rate of the z-axis in a spatial rectangular coordinate system, and each set of third Hall data including the rotation rate of the x-axis, the rotation rate of the y-axis, and the rotation rate of the z-axis in a spatial rectangular coordinate system; The steps of determining a first positional offset and a second positional offset compensated by optical image stabilization for the first frame image based on the first shooting data, determining a third positional offset and a fourth positional offset compensated by optical image stabilization for the second frame image based on the second shooting data, and determining a fifth positional offset and a sixth positional offset compensated by optical image stabilization for the third frame image based on the third shooting data, include: The first position offset is determined based on multiple sets of the first gyroscope data and multiple first timestamps. The second position offset is determined based on multiple sets of the first Hall data and multiple second timestamps; The third position offset is determined based on multiple sets of the second gyroscope data and multiple third timestamps. The fourth position offset is determined based on multiple sets of the second Hall data and multiple fourth timestamps. The fifth position offset is determined based on multiple sets of the third gyroscope data and multiple fifth timestamps. The sixth position offset is determined based on multiple sets of the third Hall data and multiple sixth timestamps.

3. The video processing method according to claim 1 or 2, characterized in that, The step of replacing the blurred image region in the second frame image based on the first mapped image and the second mapped image to obtain the target image specifically includes: Obtain the first affine function between the partitioned first mapped image and the partitioned second frame image, and the second affine function between the partitioned second mapped image and the partitioned second frame image; Based on the first affine function and the first constraint, a first linear optimization model is determined; Based on the second affine function and the second constraint, the second linear optimization model is determined; Solve the first linear optimization model to obtain the first offset, and solve the second linear optimization model to obtain the second offset; Matching the first mapped image and the second frame image based on the first offset, and matching the second mapped image and the second frame image based on the second offset; The target image is obtained by replacing the blurred image region in the second frame image with the matched first and second mapped images.

4. The video processing method according to claim 1 or 2, characterized in that, Before acquiring the first shooting data corresponding to the first frame image, the second shooting data corresponding to the second frame image, and the third shooting data corresponding to the third frame image, the method further includes: Obtain the shooting focal length corresponding to the second frame image, and obtain the angle corresponding to the second frame image. The angle corresponding to the second frame image is the attitude angle before optical image stabilization compensation. Obtain the product of the angle corresponding to the second frame image and the shooting focal length; If the product value is greater than the comparison threshold, the second frame image is determined to be the blurred frame.

5. The video processing method according to claim 4, characterized in that, The step of obtaining the angle corresponding to the second frame image specifically includes: A first detection region is determined in the second frame image, and the first detection region has a corresponding start exposure time and end exposure time; The first initial posture is determined based on the initial exposure time, the exposure time, and the gyroscope data in the second shooting data; The first ending posture is determined based on the end exposure time, the exposure time, and the gyroscope data in the second shooting data; The second initial posture is determined based on the initial exposure time, the exposure time, and the Hall data in the second shooting data; The second ending posture is determined based on the end exposure time, the exposure time, and the Hall data in the second shooting data; The first posture difference is determined based on the first ending posture and the first starting posture; The second attitude difference is determined based on the second ending attitude and the second starting attitude; The difference between the first attitude difference and the second attitude difference is determined as the angle corresponding to the second frame image.

6. A video processing apparatus, characterized in that, include: The acquisition module is used to acquire, when the first frame image and the third frame image are clear frames and the second frame image is a blurry frame, the first shooting data corresponding to the first frame image, the second shooting data corresponding to the second frame image, and the third shooting data corresponding to the third frame image, wherein the first frame image, the second frame image, and the third frame image are three consecutive frames in the video, and the first shooting data, the second shooting data, and the third shooting data all include Hall data and gyroscope data; A first determining module is configured to determine a first rotation matrix and a second rotation matrix based on the first shooting data, the second shooting data, and the third shooting data; wherein the first rotation matrix is ​​used to represent the offset between the first frame image and the second frame image, and the second rotation matrix is ​​used to represent the offset between the third frame image and the second frame image; The second determining module is used to determine a first mapped image based on the first frame image and the first rotation matrix, and to determine a second mapped image based on the third frame image and the second rotation matrix; The processing module is used to replace the blurred image regions in the second frame image based on the first mapped image and the second mapped image to obtain the target image; The first determining module is specifically used for: Based on the first shooting data, a first position offset and a second position offset that has been compensated for by optical image stabilization when shooting the first frame image are determined. Based on the second shooting data, a third position offset and a fourth position offset that has been compensated for by optical image stabilization when shooting the second frame image are determined. Based on the third shooting data, a fifth position offset and a sixth position offset that has been compensated for by optical image stabilization when shooting the third frame image are determined. Obtain the first product of the second position offset and the conversion coefficient, the second product of the fourth position offset and the conversion coefficient, and the third product of the sixth position offset and the conversion coefficient; Obtain the first difference between the first position offset and the first product value, the second difference between the third position offset and the second product value, and the third difference between the fifth position offset and the third product value; A first rotation matrix is ​​determined based on the first difference and the second difference, and a second rotation matrix is ​​determined based on the second difference and the third difference; The conversion coefficient is the scaling factor corresponding to the Hall data when converting the Hall data into gyroscope data.

7. An electronic device, characterized in that, It includes a processor and a memory, the memory storing a program or instructions that can run on the processor, the program or instructions being executed by the processor to implement the steps of the video processing method as described in any one of claims 1 to 5.

8. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the video processing method as described in any one of claims 1 to 5.

Images