Video anti-shake display method and device, terminal equipment and program product

Abstract

The embodiment of the invention provides a video anti-shake display method and device, terminal equipment and a program product, and the method comprises the steps: determining the initial anti-shake intensity of a current frame based on the obtained angular velocity data of a sensor; wherein the current frame is any frame except the first frame; determining statistical value information based on the angular velocity data and at least two preset sliding windows; wherein the sizes of all the preset sliding windows in the at least two preset sliding windows are different; updating the initial anti-shake intensity based on the statistical value information, and determining the final anti-shake intensity of the current frame; determining a final reference attitude of the current frame based on the final anti-shake intensity and the angular velocity data; and carrying out projection and mapping processing on the final reference attitude to obtain and display an image of the current frame after anti-shake. According to the scheme, a better anti-shake effect can be achieved.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular, to a video anti-shake display method, apparatus, terminal device, and program product. Background Art

[0002] Currently, in the related art, when using a relatively long recognition window to judge a motion scene, there are problems of high latency and low accuracy. Either it detects the common motion techniques used for conventional short-focus lenses, which is not applicable to the above-mentioned long-focus lenses for the recognition and judgment of static holding, fine-tuning the viewing angle, and large-scale movement. At the same time, the long-focus lens itself is more sensitive to picture jitter, so the requirement for accurate recognition of the motion scene is also higher. Some existing methods for determining the motion scene cannot meet this requirement, and correspondingly, the stability or followability of long-focus anti-shake cannot meet the user's expectations, resulting in poor anti-shake effects. Summary of the Invention

[0003] Embodiments of the present application are expected to provide a video anti-shake display method, apparatus, terminal device, and program product, which can improve the stability or followability of long-focus anti-shake and achieve better anti-shake effects.

[0004] The technical solution of the present application is implemented as follows:

[0005] In a first aspect, an embodiment of the present application provides a video anti-shake display method, including:

[0006] Based on the obtained angular velocity data of the sensor, determining the initial anti-shake intensity of the current frame; wherein, the current frame is any frame other than the first frame;

[0007] Based on the angular velocity data and at least two preset sliding windows, determining statistical value information; wherein, the size of each of the at least two preset sliding windows is different;

[0008] Based on the statistical value information, updating the initial anti-shake intensity to determine the final anti-shake intensity of the current frame;

[0009] Based on the final anti-shake intensity and the angular velocity data, determining the final reference pose of the current frame;

[0010] Performing projection and mapping processing on the final reference pose to obtain the anti-shake image of the current frame and displaying it.

[0011] In a second aspect, an embodiment of the present application provides a video anti-shake display apparatus, which includes: a determination unit, an update unit, and a display unit; wherein,

[0012] The determining unit is configured to determine an initial anti-shake intensity of the current frame based on the acquired angular velocity data of the sensor; wherein, the current frame is any frame other than the first frame; and determine statistical value information based on the angular velocity data and at least two preset sliding windows; wherein, the size of each of the at least two preset sliding windows is different.

[0013] The updating unit is configured to update the initial anti-shake intensity based on the statistical value information to determine a final anti-shake intensity of the current frame.

[0014] The determining unit is further configured to determine a final reference pose of the current frame based on the final anti-shake intensity and the angular velocity data.

[0015] The display unit is configured to perform projection and mapping processing on the final reference pose to obtain an image after anti-shake of the current frame and display it.

[0016] In a third aspect, an embodiment of the present application provides a terminal device, which includes: a processor and a memory; wherein,

[0017] The memory is configured to store a computer program.

[0018] The processor is configured to call and run the computer program from the memory to execute the method as described in the first aspect.

[0019] In a fourth aspect, an embodiment of the present application provides a computer program product, including a computer program or instruction for causing a processor to execute the method as described in the first aspect.

[0020] An embodiment of the present application provides a video anti-shake display method, apparatus, device, and computer program product. The method includes: determining an initial anti-shake intensity of a current frame based on the acquired angular velocity data of a sensor, where the current frame is any frame other than the first frame; determining statistical value information based on the angular velocity data and at least two preset sliding windows, where the size of each of the at least two preset sliding windows is different; updating the initial anti-shake intensity based on the statistical value information to determine a final anti-shake intensity of the current frame; determining a final reference pose of the current frame based on the final anti-shake intensity and the angular velocity data; performing projection and mapping processing on the final reference pose to obtain an anti-shake image of the current frame and display it. In the above solution, on the one hand, by calculating statistical value information by combining angular velocity data with different preset sliding windows, since the size of each preset sliding window is different, more abundant statistical value information can be determined. On the other hand, updating the initial anti-shake intensity based on the statistical value information to determine the final anti-shake intensity of the current frame can make the final anti-shake intensity more accurate. Thus, based on the final anti-shake intensity and the angular velocity data, the final reference pose of the current frame is determined; performing projection and mapping processing on the final reference pose to obtain an anti-shake image of the current frame can improve the stability or followability in different motion states, thereby achieving a better anti-shake effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings herein are incorporated into the specification and form a part of this specification. These drawings illustrate embodiments consistent with the present application and, together with the specification, are used to explain the technical solutions of the present application. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0022] The flowchart shown in the drawings is only an exemplary illustration and does not necessarily include all contents and operations / steps, nor does it necessarily need to be executed in the described order. For example, some operations / steps can be decomposed, while some operations / steps can be combined or partially combined. Therefore, the actual execution order may change according to the actual situation.

[0023] Figure 1 FIG. is a schematic diagram of a scenario of a video anti-shake display method provided by an embodiment of the present application;

[0024] Figure 2 FIG. is an optional flowchart of a video anti-shake display method provided by an embodiment of the present application Figure 1 ;

[0025] Figure 3An optional process schematic diagram of a video anti-shake display method provided by an embodiment of the present application Figure 2 ;

[0026] Figure 4 An optional process schematic diagram of a video anti-shake display method provided by an embodiment of the present application Figure 3 ;

[0027] Figure 5 An optional process schematic diagram of a video anti-shake display method provided by an embodiment of the present application Figure 4 ;

[0028] Figure 6 An optional process schematic diagram of a video anti-shake display method provided by an embodiment of the present application Figure 5 ;

[0029] Figure 7 An optional process schematic diagram of a video anti-shake display method provided by an embodiment of the present application Figure 6 ;

[0030] Figure 8 A structural schematic diagram of a video anti-shake display device provided by an embodiment of the present application;

[0031] Figure 9 A structural schematic diagram of a terminal device provided by an embodiment of the present application. Detailed implementation manners

[0032] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the following will further describe the specific technical solutions of the present application in detail with reference to the accompanying drawings in the embodiments of the present application. The following embodiments are used to illustrate the present application but are not intended to limit the scope of the present application.

[0033] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0034] In the following descriptions, references to "some embodiments", "this embodiment", "embodiments of the present application", and examples, etc., describe subsets of all possible embodiments. However, it can be understood that "some embodiments" can be the same subset or different subsets of all possible embodiments and can be combined with each other without conflict.

[0035] If similar descriptions such as "first / second" appear in the application documents, the following description shall be added. In the following description, the terms "first / second / third" only distinguish similar objects and do not represent a specific order for the objects. Understandably, "first / second / third" can be interchanged in a specific order or sequence when permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0036] With the rapid development of science and technology, people increasingly use smartphones to take pictures and record their lives. At the same time, in order to meet more diverse shooting needs, mobile phone manufacturers have entered the era of multi-camera, and the mainstream lenses include wide-angle, ultra-wide-angle, telephoto, portrait lenses, etc. Among them, the telephoto lens can extend the vision to capture distant objects more than others, making the subject more prominent. Therefore, it has gradually become an important highlight in the mobile phone market and is widely used by user groups.

[0037] As Figure 1 shown, for a telephoto lens (such as a narrow-angle lens) at high magnification zoom, with the same degree of mobile phone shake, compared with a short-focus lens (such as a wide-angle or ultra-wide-angle lens), the degree of shake of the picture will be amplified. Taking a telephoto lens with a focal length of 1000mm as an example, when the gimbal device shakes horizontally by 0.1°, the shake amplitude at the wide-angle end accounts for 0.45% of the entire picture, and the picture shake is not obvious; but when the lens is at the telephoto segment, the shake amplitude accounts for 22.00% of the entire picture, and the picture shake becomes larger subjectively.

[0038] In addition, compared with traditional short-focus lenses, users will use methods such as static holding, panning, walking, and running to take pictures. Due to the "telephoto" characteristics of telephoto lenses, users often use several common techniques such as static holding, fine-tuning the viewing angle to frame, and significantly moving to switch the shooting target. Different EIS anti-shake methods should be adopted for different techniques to more adaptively output an anti-shake effect that meets user needs. For example, when statically holding, the anti-shake intensity is increased to make the picture completely stable, and when the user subjectively moves the picture, the anti-shake intensity is weakened to increase the followability of the picture.

[0039] Currently, in related technologies, there are methods that use time-domain or frequency-domain feature points to identify the shooting motion mode, which require a relatively large recognition window, such as more than 1s, and cannot output the recognition result in real time; some methods use camera attitude parameters to construct a sliding time window for statistical analysis of target data, and then classify to obtain accumulated camera motion scenarios, such as static holding, panning, walking, running, etc.

[0040] The technical solution of the present invention identifies three motion scenarios for telephoto lenses and provides a method for calculating the anti-shake intensity in different motion scenarios, which can achieve a real-time, stable, followable, and smooth telephoto anti-shake effect.

[0041] To address the drawbacks of related technologies, an embodiment of the present application provides a video anti-shake display method, including: determining an initial anti-shake intensity for the current frame based on the acquired angular velocity data; where the current frame is any frame other than the first frame; determining statistical value information based on the angular velocity data and at least two preset sliding windows; where the size of each of the at least two preset sliding windows is different; updating the initial anti-shake intensity based on the statistical value information to determine the final anti-shake intensity for the current frame; determining the final reference pose for the current frame based on the final anti-shake intensity and the angular velocity data; performing projection and mapping processing on the reference pose to obtain the anti-shake image for the current frame and display it. On the one hand, by calculating statistical value information through the angular velocity data in combination with different preset sliding windows, since the size of each preset sliding window is different, more abundant statistical value information can be determined. On the other hand, updating the initial anti-shake intensity based on the statistical value information to determine the final anti-shake intensity for the current frame can make the final anti-shake intensity more accurate, and thus, based on the final anti-shake intensity and the angular velocity data, determine the final reference pose for the current frame; performing projection and mapping processing on the final reference pose to obtain the anti-shake image for the current frame can improve the stability or followability in different motion states, thereby achieving a better anti-shake effect.

[0042] An embodiment of the present application provides a video anti-shake display method, Figure 2 which is an optional process schematic for a video anti-shake display method provided by an embodiment of the present application Figure 1 and will be described in conjunction with Figure 2 the steps shown.

[0043] S101. Determine an initial anti-shake intensity for the current frame based on the acquired angular velocity data of the sensor; where the current frame is any frame other than the first frame.

[0044] In some embodiments of the present application, the video anti-shake display method is applicable to the anti-shake scenario of a telephoto lens in different motion states.

[0045] In some embodiments of the present application, the execution subject of the video anti-shake display method is a terminal device, which can be a mobile phone or a tablet computer, and the embodiments of the present application do not make specific limitations in this regard.

[0046] In some embodiments of the present application, the angular velocity data is collected through a sensor on the terminal device to directly obtain the angular velocity data. The initial anti-shake intensity is data directly determined based on the angular velocity data without anti-shake processing.

[0047] It should be noted that the sensor can be a gyroscope.

[0048] In some embodiments of the present application, the terminal device obtains angular velocity data through a sensor, integrates the angular velocity data, determines the attitude difference between the current frame and the previous frame, and converts the attitude difference into Euler angles, thereby determining the initial anti-shake intensity.

[0049] In some embodiments of the present application, based on the angular velocity data, integration is performed to obtain the attitude information corresponding to each frame of data; based on the attitude information corresponding to each frame of data, the attitude difference between the current frame and the previous frame is determined; and the attitude difference is converted into three-axis Euler angles in three-dimensional space; wherein, the three-axis Euler angles include a first Euler angle, a second Euler angle, and a third Euler angle; based on the first Euler angle, the second Euler angle, and the third Euler angle, the main direction axis of the current frame is determined; based on the main direction axis of the current frame and the main direction axis of the obtained previous frame, the cumulative Euler angle change value is determined; based on the cumulative Euler angle change value, a preset cumulative Euler angle threshold, and a preset anti-shake intensity, calculations are performed to determine the initial anti-shake intensity of the current frame.

[0050] It should be noted that the current frame is any frame other than the first frame.

[0051] S102. Determine statistical value information based on the angular velocity data and at least two preset sliding windows; wherein, the size of each of the at least two preset sliding windows is different.

[0052] In some embodiments of the present application, the size of each of the at least two preset sliding windows is different, and a preset sliding window refers to an image sequence of different numbers of frames.

[0053] In some embodiments of the present application, the at least two preset sliding windows include a first preset sliding window, a second preset sliding window, a third preset sliding window, and a fourth preset sliding window.

[0054] It should be noted that the present application does not make a specific limitation on the number of the at least two preset sliding windows.

[0055] Exemplarily, the first preset sliding window is an image sequence of 10 frames, the second preset sliding window is an image sequence of 15 frames, the third preset sliding window is an image sequence of 20 frames, and the fourth preset sliding window is an image sequence of 60 frames.

[0056] In some embodiments of the present application, the statistical value information includes pixel offset, cumulative pixel offset corresponding to at least two preset sliding windows, average pixel offset, cumulative vector offset, acceleration / deceleration state value, picture pixel offset, and picture vector offset. Among them, the acceleration / deceleration state value includes a first acceleration / deceleration state value, a second acceleration state value, a third acceleration state value, and a third deceleration state value.

[0057] It should be noted that the first acceleration / deceleration state value includes a first acceleration state value and a first deceleration state value.

[0058] In some embodiments of the present application, the terminal device can integrate based on the angular velocity data to obtain the attitude information corresponding to each frame of data. According to the attitude information corresponding to each frame of data, operations and conversions are performed to determine the pixel offset. Based on the pixel offset, operations are performed through at least two preset sliding windows to determine the statistical value information.

[0059] In some embodiments of the present application, integration is performed based on the angular velocity data to obtain the attitude information corresponding to each frame of data. Based on the attitude information corresponding to each frame of data, the attitude difference between the current frame and the previous frame is determined; and the attitude difference is converted into a pixel offset; wherein the pixel offset includes a first pixel offset and a second pixel offset. Based on the pixel offset and at least two preset sliding windows, the statistical value information is determined.

[0060] S103. Update the initial anti-shake intensity based on the statistical value information to determine the final anti-shake intensity of the current frame.

[0061] In some embodiments of the present application, the final anti-shake intensity refers to the anti-shake intensity after the terminal device performs anti-shake processing on the current frame.

[0062] In some embodiments of the present application, the terminal device can update the determined initial anti-shake intensity based on the pixel offset, the cumulative pixel offset corresponding to at least two preset sliding windows, the average pixel offset, the cumulative vector offset, the acceleration / deceleration state value, the picture pixel offset, and the picture vector offset to obtain the intermediate anti-shake intensity of the current frame, and then continue to update the intermediate anti-shake intensity to determine the final anti-shake intensity of the current frame.

[0063] In some embodiments of the present application, the intermediate anti-shake intensity is obtained by updating the anti-shake intensity of the current frame based on the motion state of the current frame.

[0064] In some embodiments of the present application, the terminal device determines the motion state corresponding to the current frame based on the statistical value information; updates the initial anti-shake intensity based on the motion state of the current frame to determine the intermediate anti-shake intensity of the current frame; and updates the intermediate anti-shake intensity based on the motion state of the current frame and the anti-shake intensity of the previous frame to determine the final anti-shake intensity of the current frame.

[0065] In some embodiments of the present application, the motion state corresponding to the current frame is generally divided into a static holding state and a moving state. According to the magnitude of the motion amplitude of the moving state, the moving state is divided into a fine-tuning perspective and a large movement.

[0066] It should be noted that the static holding state, the fine-tuning perspective, and the large movement all have their corresponding default anti-shake intensities.

[0067] S104. Determine the final reference pose of the current frame based on the final anti-shake intensity and the angular velocity data.

[0068] In some embodiments of the present application, the pose of the current frame is divided into the actual pose of the current frame and the reference pose of the current frame. The final reference pose of the current frame is obtained by calculating the reference pose of the previous frame and the actual pose of the current frame through the final anti-shake intensity.

[0069] In some embodiments of the present application, based on the angular velocity data, determine the reference pose of the previous frame and the actual pose of the current frame; through the final anti-shake intensity, perform weighted processing on the reference pose of the previous frame and the actual pose of the current frame to determine the final reference pose of the current frame.

[0070] In some embodiments of the present application, the terminal device can integrate based on the acceleration data to obtain the pose information of each frame. Based on the pose information of each frame, determine the actual pose of the current frame and the reference pose of the previous frame. Then, through the determined final anti-shake intensity, perform weighted processing on the reference pose of the previous frame and the actual pose of the current frame to update the reference pose of the current frame and obtain the final reference pose of the current frame.

[0071] S105. Perform projection and mapping processing on the final reference pose to obtain the image after anti-shake of the current frame and display it.

[0072] In some embodiments of the present application, the terminal device can perform projection processing on the final reference pose through the camera model to obtain two-dimensional grid points, and perform mapping processing on the image through the two-dimensional grid points to obtain the image after anti-shake of the current frame. Display the image after anti-shake of the current frame.

[0073] Exemplarily, for the current frame and each frame, use the camera model to project the final reference pose into two-dimensional grid points, and then use the grid points to warp the image, and finally obtain the sequence of images after anti-shake.

[0074] It can be understood that, on the one hand, by combining the angular velocity data with different preset sliding windows to calculate the statistical value information, since the sizes of each preset sliding window are different, more abundant statistical value information can be determined. On the other hand, based on the statistical value information, update the initial anti-shake intensity to determine the final anti-shake intensity of the current frame, which can make the final anti-shake intensity more accurate. Thus, based on the final anti-shake intensity and the angular velocity data, determine the final reference pose of the current frame; perform projection and mapping processing on the final reference pose to obtain the image after anti-shake of the current frame, which can improve the stability and followability in different motion states, and thus achieve a better anti-shake effect.

[0075] In some embodiments of the present application, Figure 3An optional process schematic for a video anti-shake display method provided by an embodiment of the present application Figure 2 , as Figure 3 shown, S102 can be implemented through S201, S202, and S203, as follows:

[0076] S201. Integrate based on the angular velocity data to obtain the attitude information corresponding to each frame of data.

[0077] In some embodiments of the present application, the attitude information corresponding to each frame of data includes the reference attitude and the actual attitude of each frame.

[0078] In some embodiments of the present application, the terminal device can integrate based on the angular velocity data to obtain the reference attitude and the actual attitude of each frame respectively.

[0079] S202. Determine the attitude difference between the current frame and the previous frame based on the attitude information corresponding to each frame of data; and convert the attitude difference into a pixel offset; wherein, the pixel offset includes a first pixel offset and a second pixel offset.

[0080] In some embodiments of the present application, the pixel offset includes a first pixel offset and a second pixel offset. The directions of the first pixel offset and the second pixel offset are different.

[0081] Exemplarily, the first pixel offset can be the x-direction offset of the image pixels, and the second pixel offset can be the y-direction offset of the image pixels.

[0082] In some embodiments of the present application, the terminal device can determine the attitude difference between the current frame and the previous frame through the reference attitude and the actual attitude corresponding to each frame of data. Then convert the attitude difference between the current frame and the previous frame into a pixel offset.

[0083] S203. Determine the statistical value information based on the pixel offset and at least two preset sliding windows.

[0084] In some embodiments of the present application, the terminal device can perform operations based on the pixel offset through each preset sliding window among at least two preset sliding windows to determine the statistical value information.

[0085] In some embodiments of the present application, the terminal device performs operations on the pixel offset through at least two preset sliding windows to obtain the accumulated pixel offsets corresponding to each of the at least two preset sliding windows. Based on the accumulated pixel offsets, perform operations to determine the average pixel offset of the current frame and the cumulative vector offset of the current frame. Based on the cumulative vector offset of the current frame, the attitude information corresponding to each frame of data, and the pixel offset, determine the acceleration / deceleration state value.

[0086] In some embodiments of the present application, the terminal device uses the pixel offset, the accumulated pixel offset, the average pixel offset, the accumulated vector offset of the current frame, and the acceleration and deceleration state value as statistical value information.

[0087] It can be understood that the terminal device can perform integration based on the angular velocity data to obtain the attitude information corresponding to each frame of data. Based on the attitude information corresponding to each frame of data, operations and conversions are performed to determine the pixel offset. Based on the pixel offset, operations are performed through at least two preset sliding windows to determine the statistical value information. Since the sizes of each preset sliding window are different, more abundant statistical value information can be determined.

[0088] In some embodiments of the present application, Figure 4 is an optional process schematic diagram of a video anti-shake display method provided by an embodiment of the present application Figure 3 , as Figure 4 shown, S203 can be implemented through S301, S302, S303, and S304 as follows:

[0089] S301. Through at least two preset sliding windows, perform operations on the pixel offset to obtain the accumulated pixel offsets corresponding to each of the at least two preset sliding windows.

[0090] In some embodiments of the present application, the terminal device can perform operations on the pixel offset through each preset sliding window of the at least two preset sliding windows to obtain the accumulated pixel offset of each preset sliding window.

[0091] Exemplarily, the at least two preset sliding windows are respectively an image sequence of 10 frames, an image sequence of 15 frames, an image sequence of 20 frames, and an image sequence of 60 frames. The statistical information of different-sized sliding windows can assist each other to more accurately reflect the current handheld state. Record the accumulated pixel offsets of the historical 10 frames as shiftSum10.x and shiftSum10.y. Similarly, the accumulated pixel offsets corresponding to 15 frames, 20 frames, and 60 frames are shiftSum15.x, shiftSum15.y, shiftSum20.x, shiftSum20.y, shiftSum60.x, and shiftSum60.y.

[0092] S302. Based on the accumulated pixel offset, perform operations to determine the average pixel offset of the current frame and the accumulated vector offset of the current frame.

[0093] In some embodiments of the present application, the terminal device can perform operations based on the accumulated pixel offset of each preset sliding window to determine the average pixel offset of the current frame and the accumulated vector offset of the current frame.

[0094] Exemplarily, taking the third preset sliding window (image sequence of 15 frames) as an example, the average pixel offset shiftMean.x and shiftMean.y of the current frame can be implemented through the following formulas (1) and (2) as follows:

[0095] shiftMean.x = shiftSum20.x / 20 (1)

[0096] shiftMean.y = ShftSum20.y / 20 (2)

[0097] Taking the fourth preset sliding window (image sequence of 20 frames) as an example, the cumulative vector offset shiftSumAll of the current frame can be implemented through the following formula (3) as follows:

[0098]

[0099] For each frame, calculate the cumulative vector offset of the previous 15 frames, and maintain a queue sumQue with a maximum size of 10 to cache the shiftSumAll information of the adjacent 10 frames.

[0100] S303. Determine the acceleration and deceleration state values based on the cumulative vector offset of the current frame, the attitude information corresponding to each frame of data, and the pixel offset.

[0101] In some embodiments of the present application, the acceleration and deceleration state values include a first acceleration and deceleration state value, a second acceleration state value, a third acceleration state value, and a third deceleration state value. The first acceleration and deceleration state value includes a first acceleration state value and a first deceleration state value.

[0102] In some embodiments of the present application, the terminal device can use various methods to count the acceleration and deceleration state information.

[0103] It should be noted that the acceleration state information can be regarded as an acceleration state value; the deceleration state information can be regarded as a deceleration state value.

[0104] In some embodiments of the present application, the terminal device can perform operations based on the cumulative vector offset of the current frame to determine the first acceleration and deceleration state value. Determine the second acceleration state value and the third acceleration state value based on the attitude information corresponding to each frame of data. Determine the third deceleration state value based on the pixel offset.

[0105] In some embodiments of the present application, the terminal device can perform operations based on the cumulative vector offset of the current frame and the cumulative vector offset of the previous frame obtained to determine the first acceleration and deceleration state value.

[0106] In some embodiments of the present application, the terminal device determines the anti-shake offset of the current frame based on the attitude information corresponding to each frame of data; and performs calculations based on the anti-shake offset of the current frame to determine the second acceleration / deceleration state value.

[0107] In some embodiments of the present application, the terminal device obtains the picture pixel offset of the previous frame; determines the picture pixel offset of the current frame based on the picture pixel offset of the previous frame, the pixel offset, and the anti-shake offset of the current frame; and determines the third acceleration / deceleration state value based on the picture pixel offset of the current frame.

[0108] S304. Determine the statistical value information based on the pixel offset, the accumulated pixel offset, the average pixel offset, the accumulated vector offset of the current frame, and the acceleration / deceleration state value.

[0109] In some embodiments of the present application, the terminal device may determine the pixel offset, the accumulated pixel offset, the average pixel offset, the accumulated vector offset of the current frame, the first acceleration state value, the first deceleration state value, the second acceleration state value, the third acceleration state value, and the third deceleration state value as the statistical value information.

[0110] It can be understood that the terminal device performs calculations on the pixel offset through at least two preset sliding windows to obtain the accumulated pixel offset corresponding to each of the at least two preset sliding windows. Based on the accumulated pixel offset, calculations are performed to determine the average pixel offset of the current frame and the accumulated vector offset of the current frame. Based on the accumulated vector offset of the current frame, the attitude information corresponding to each frame of data, and the pixel offset, the acceleration / deceleration state value is determined; based on the pixel offset, the accumulated pixel offset, the average pixel offset, the accumulated vector offset of the current frame, and the acceleration / deceleration state value, the statistical value information is determined. Since the size of each preset sliding window is different, richer statistical value information can be determined, which is convenient for subsequently calculating and judging the motion state and determining the final anti-shake intensity according to the statistical value information.

[0111] In some embodiments of the present application, S303 may be implemented through S3031 - S3034 as follows:

[0112] S3031. Determine the first acceleration / deceleration state value based on the accumulated vector offset of the current frame and the accumulated vector offset of the previous frame obtained.

[0113] In some embodiments of the present application, the terminal device may obtain the accumulated vector offset of the previous frame from sumQue.

[0114] In some embodiments of the present application, the terminal device takes the absolute values of the cumulative vector offsets of the previous frame and the current frame obtained respectively, and then subtracts them to obtain a first difference. If the first difference is greater than a first preset threshold, the first deceleration state value is incremented by 1. The terminal device takes the absolute values of the cumulative vector offset of the current frame and the cumulative vector offset of the previous frame obtained respectively, and then subtracts them to obtain a second difference. If the second difference is greater than a second preset threshold, the first acceleration state value is incremented by 1.

[0115] Exemplarily, calculate the difference between two adjacent data in sumQue and compare it with a preset threshold (the first preset threshold or the second preset threshold). If the condition is met, the deceleration frame count descCnt is incremented by 1, or the acceleration frame count accCnt is incremented by 1, as shown in formulas (4) and (5):

[0116] if abs(sumQue[i])-abs(sumQue[i+1])>threshold1,then descCnt++ (4)

[0117] if abs(sumQue[i+1])-abs(sumQue[i])>threshold2,then accCnt++ (5)

[0118] It should be noted that the deceleration frame count descCnt is the first deceleration state value; the acceleration frame count accCnt is the first acceleration state value. threshold1 is the first preset threshold; threshold2 is the second preset threshold. sumQue[i] can be regarded as the cumulative vector offset of the previous frame, and sumQue[i+1] can be regarded as the cumulative vector offset of the current frame. The first preset threshold and the second preset threshold can be the same or different. In this regard, the embodiments of the present application do not make specific limitations.

[0119] S3032. Based on the attitude information corresponding to each frame of data, determine the anti-shake offset of the current frame; and based on the anti-shake offset of the current frame, perform an operation to determine the second acceleration state value and the third acceleration state value.

[0120] In some embodiments of the present application, the terminal device can determine the anti-shake offset of the current frame based on the attitude information corresponding to each frame of data. Based on the anti-shake offset of the current frame, perform an operation to determine the second acceleration state value and the third acceleration state value.

[0121] In some embodiments of the present application, based on the attitude information corresponding to each frame of data, the actual attitude of the current frame and the reference attitude of the previous frame are determined; through the initial anti-shake intensity, a weighted operation is performed on the actual attitude of the current frame and the reference attitude of the previous frame to determine the reference attitude of the current frame; based on the actual attitude of the current frame and the reference attitude of the current frame, the attitude difference of the current frame is determined; the attitude difference of the current frame is converted to obtain the anti-shake offset of the current frame; based on the obtained anti-shake offset of the previous frame and the anti-shake offset of the current frame, an operation is performed to determine the second acceleration state value and the third acceleration state value.

[0122] In some embodiments of the present application, the anti-shake offset includes a first anti-shake offset and a second anti-shake offset; the terminal device takes the absolute values of the first anti-shake offset of the current frame and the obtained first anti-shake offset of the previous frame respectively, and then performs a difference operation to obtain a third difference. If the third difference is greater than zero, the second acceleration state value is incremented by 1. The terminal device takes the absolute values of the second anti-shake offset of the current frame and the obtained second anti-shake offset of the previous frame respectively, and then performs a difference operation to obtain a fourth difference. If the fourth difference is greater than zero, the third acceleration state value is incremented by 1.

[0123] It should be noted that the second acceleration state value is the acceleration state value in the x direction; the third acceleration state value is the acceleration state value in the y direction.

[0124] Exemplarily, based on the angular velocity data, integration can be performed to obtain the actual attitude corresponding to each frame of the image, denoted as curQ. After anti-shake, the actual attitude curQ of the current frame will be rotated to the reference attitude refQ. Among them, the calculation of the initial refQ of the current frame is obtained by weighting the refQ_pre of the previous frame and the actual attitude curQ of the current frame using the initial anti-shake intensity targetStrength, as shown in formula (6) specifically:

[0125] The initial refQ of the current frame = targetStrength * refQ_pre of the previous frame + (1 - targetStrength) * curQ (6)

[0126] Therefore, for each frame of the image, the attitude difference between curQ and refQ can be calculated and converted to the image pixels to obtain the offsets diffShift.x and diffShift.y before and after anti-shake. The larger the absolute value of this value, the stronger the anti-shake stability and the worse the followability. The smaller the value, the stronger the followability of the anti-shake and the worse the stability. At the same time, by tracking the change of this value, it is also possible to detect whether there is an obvious subjective intention of camera movement, so as to balance the stability and followability of the anti-shake.

[0127] It should be noted that the offset diffShift.x and diffShift.y before and after anti-shake are the anti-shake offsets of the current frame. diffShift.x is the first anti-shake offset, and diffShift.y is the second anti-shake offset.

[0128] Maintain a queue diffQue with a maximum size of 5 to cache diffShift.x and diffShift.y of the adjacent 5 frames. Similarly, by calculating the difference between two adjacent data, the acceleration frame numbers xAccCnt and yAccCnt in the x direction or y direction can be obtained. xAccCnt is the second acceleration state value, and yAccCnt is the third acceleration state value. Calculating xAccCnt and yAccCnt can be achieved through formulas (7) and (8) as follows:

[0129] if abs(diffQue[i+1].x)-abs(diffQue[i].x)>0,then xAccCnt++,elsexAccCnt=0 (7)

[0130] if abs(diffQue[i+1].y)-abs(diffQue[i].y)>0,then yAccCnt++,elseyAccCnt=0 (8)

[0131] S3033. Obtain the screen pixel offset of the previous frame.

[0132] S3034. Determine the screen pixel offset of the current frame based on the screen pixel offset of the previous frame, the pixel offset, and the anti-shake offset of the current frame.

[0133] In some embodiments of the present application, the screen pixel offset includes: the first screen pixel offset and the second screen pixel offset; the anti-shake offset includes: the first anti-shake offset and the second anti-shake offset.

[0134] In some embodiments of the present application, if the screen pixel offset of the previous frame meets the first preset condition, then set the screen pixel offset of the previous frame to zero and determine the screen pixel offset of the current frame.

[0135] In some embodiments of the present application, if the screen pixel offset of the previous frame does not meet the first preset condition, and the screen pixel offset of the previous frame and the anti-shake offset of the current frame meet the second preset condition, or, the screen pixel offset of the previous frame does not meet the first preset condition, the screen pixel offset of the previous frame and the anti-shake offset of the current frame do not meet the second preset condition, and meet the third preset condition, then update the screen pixel offset of the previous frame through the pixel offset and determine the screen pixel offset of the current frame.

[0136] It can be understood that by setting different conditions, the pixel offset of the current frame is determined based on the pixel offset of the previous frame, the pixel offset, and the anti-shake offset of the current frame. Since the pixel offset of the picture belongs to a part of the statistical value information, the statistical value information is enriched.

[0137] In some embodiments of the present application, the first preset condition is that the first pixel offset of the previous frame or the second pixel offset of the previous frame is greater than the first threshold;

[0138] The second preset condition is that the pixel offset of the previous frame and the final anti-shake intensity of the previous frame satisfy the first sub-preset condition, the previous frame is in a static holding state, and the anti-shake offset of the current frame satisfies the second sub-preset condition; wherein,

[0139] The first sub-preset condition is that at least one of the following is satisfied: the final anti-shake intensity of the previous frame is greater than the second threshold, the first pixel offset of the previous frame is greater than zero, and the second pixel offset of the previous frame is greater than zero;

[0140] The second sub-preset condition is that at least one of the following is satisfied: the first anti-shake offset of the current frame is greater than the third threshold and the second anti-shake offset of the current frame is greater than the third threshold;

[0141] The third preset condition is that the previous frame is in a moving state.

[0142] Exemplarily, the pixel offset of the picture accumulated from the start of rest or small jitters is set as cumulativeOffset.x, cumulativeOffset.y, and the initial value is 0.

[0143] If condition 1 (i.e., the first preset condition) is satisfied: if either cumulativeOffset.x or cumulativeOffset.y statistically calculated in the previous frame is greater than threshold_3 (e.g., 1000), then reset cumulativeOffset.x and cumulativeOffset.y to 0. threshold_3 is the first threshold. cumulativeOffset.x is the first pixel offset of the picture, and cumulativeOffset.y is the second pixel offset of the picture.

[0144] If condition 1 is not satisfied, but condition 2 (i.e., the second preset condition) is satisfied: denote the anti-shake intensity of the previous frame as targetStrengt_pre. If condition 1 is not satisfied, but condition 2 is satisfied: denote the anti-shake intensity of the previous frame as targetStrengt_pre. Condition 2 is as follows:

[0145] if (targetStrength_pre > threshold_4 || cumulativeOffset.x > 0 || cumulativeOffset.y > 0) && the previous frame was in a static state && (diffShift.x > threnshold_5 || diffShift.y > threnshold_5),

[0146] then cumulativeOffset.x = cumulativeOffset.x + tshift.x

[0147] cumulativeOffset.y = cumulativeOffset.y + tshift.y

[0148] Among them, (targetStrength_pre > threshold_4 || cumulativeOffset.x > 0 || cumulativeOffset.y > 0) is the first sub - preset condition, and threshold_4 is the second threshold; (diffShift.x > threnshold_5 || diffShift.y > threnshold_5) is the second sub - preset condition, and threnshold_5 is the third threshold.

[0149] If conditions 1 and 2 are not met, but condition 3 (i.e., the third preset condition) is met: the previous frame was in a moving state, then update cumulativeOffset.x = cumulativeOffset.x + tshift.x, cumulativeOffset.y = cumulativeOffset.y + tshift.y to obtain the updated cumulativeOffset.x and cumulativeOffset.y of the current frame, and then convert them into the cumulative offset cumulativeShift in the vector direction, as shown in formula (9):

[0150]

[0151] Maintain a queue cumuShiftQue with a maximum size of 5 to cache the cumulativeShift information of the adjacent 5 frames.

[0152] S3035. Determine the third deceleration state value based on the pixel offset of the current frame.

[0153] In some embodiments of the present application, based on the pixel offset of the current frame, an operation is performed to determine the vector offset of the current frame; based on the obtained vector offset of the previous frame and the vector offset of the current frame, an operation is performed to determine the third deceleration state value.

[0154] Exemplarily, calculate the difference between two adjacent data in cumuShiftQue. If it is greater than 0, increment the deceleration frame count cumuDescCnt by 1. cumuDescCnt is the third deceleration state value. cumuDescCnt can be calculated by formula (10) as follows:

[0155] if cumuShiftQue[i]-cumuShiftQue[i+1]>0,then cumuDescCnt++ (10)

[0156] where cumuShiftQue[i+1] is the vector offset of the current frame; cumuShiftQue[i] is the vector offset of the previous frame.

[0157] It can be understood that based on the cumulative vector offset of the current frame and the obtained cumulative vector offset of the previous frame, an operation is performed to determine the first acceleration / deceleration state value; based on the attitude information corresponding to each frame of data, the anti-shake offset of the current frame is determined; and based on the anti-shake offset of the current frame, operations are performed to determine the second acceleration state value and the third acceleration state value; obtain the pixel offset of the previous frame. Based on the pixel offset of the previous frame, the pixel offset, and the anti-shake offset of the current frame, determine the pixel offset of the current frame; based on the pixel offset of the current frame, determine the third deceleration state value. By using a variety of different methods to calculate the acceleration / deceleration state value, the data of the acceleration / deceleration state value is enriched.

[0158] In some embodiments of the present application, Figure 5 is an optional process schematic diagram of a video anti-shake display method provided by an embodiment of the present application Figure 4 , as Figure 5 shown, S103 can be implemented through S401, S402, and S403 as follows:

[0159] S401. Based on the statistical value information, determine the motion state corresponding to the current frame.

[0160] In some embodiments of the present application, the motion state of the previous frame is determined; when the motion state of the previous frame is a moving state, if the statistical value information meets the fourth preset condition, it is determined that the motion state of the current frame is a static state; if the statistical value information does not meet the fourth preset condition, it is determined that the motion state of the current frame is a moving state; when the motion state of the previous frame is a static state, if the statistical value information meets the fifth preset condition, it is determined that the motion state of the current frame is a moving state; if the statistical value information does not meet the fifth preset condition, it is determined that the motion state of the current frame is a static state.

[0161] In some embodiments of the present application, the accumulated pixel offset includes the accumulated pixel offset of the first preset sliding window and the accumulated pixel offset of the second preset sliding window; each accumulated pixel offset includes a first accumulated pixel offset and a second accumulated pixel offset; the pixel offset includes a first pixel offset and a second pixel offset.

[0162] In some embodiments of the present application, the fourth preset condition is that the distances between the boundaries of the image output after anti-shake of the previous frame and the four boundaries of the original input image are all less than the fourth threshold, and the accumulated pixel offset, the third acceleration / deceleration state value, and the pixel offset meet the third sub-preset condition, or the accumulated vector offset of the current frame, the first acceleration / deceleration state value, and the accumulated pixel offset meet the fourth sub-preset condition; wherein, the first acceleration / deceleration state value includes a first acceleration state value and a first deceleration state value;

[0163] The third sub-preset condition is that the absolute value of the first accumulated pixel offset of the second preset sliding window is less than the fifth threshold, the absolute value of the second accumulated pixel offset of the second preset sliding window is less than the fifth threshold, the absolute value of the first pixel offset is less than the sixth threshold, the absolute value of the second pixel offset is less than the sixth threshold, and the third acceleration / deceleration state value is greater than zero;

[0164] The fourth sub-preset condition is that the accumulated vector offset of the current frame is less than the seventh threshold, the first acceleration state value is less than the eighth threshold, the first deceleration state value is greater than or equal to the eighth threshold, the first accumulated pixel offset of the first preset sliding window is less than the ninth threshold, and the second accumulated pixel offset of the first preset sliding window is less than the ninth threshold.

[0165] In some embodiments of the present application, the accumulated pixel offset further includes the accumulated pixel offset of the third preset sliding window; the accumulated pixel offset of the third preset sliding window includes: the first accumulated pixel offset of the third preset sliding window and the second accumulated pixel offset of the third preset sliding window;

[0166] In some embodiments of the present application, the fifth preset condition is to meet any one of the fifth sub-condition, the sixth sub-condition, the seventh sub-condition, and the eighth sub-condition; wherein,

[0167] The fifth sub - condition is that the screen vector offset is greater than the tenth threshold, the second acceleration state value is greater than or equal to the eighth threshold, and the third acceleration state value is less than the eighth threshold;

[0168] The sixth sub - condition is that the absolute value of the first cumulative pixel offset of the third preset sliding window is less than the eleventh threshold, the absolute value of the second cumulative pixel offset of the third preset sliding window is less than the eleventh threshold, and the average pixel offset is greater than the twelfth threshold;

[0169] The seventh sub - condition is that the average pixel offset is greater than the thirteenth threshold;

[0170] The eighth sub - condition is that the distance between the boundary of the image output after anti - shake in the previous frame and any one of the four boundaries of the original image is less than the fourteenth threshold.

[0171] Exemplarily, if the previous frame is in a moving state, it is necessary to determine whether the current frame can enter the static holding state according to the above - mentioned statistical information, as follows:

[0172] 1. Calculate the distances between the boundaries of the image output after anti - shake in the previous frame and the top, bottom, left, and right boundaries of the original input EIS image. The distances should all be less than threnshold_6, for example, taking 10% of the image size.

[0173] It should be noted that calculating the distances between the boundaries of the image output after anti - shake in the previous frame and the top, bottom, left, and right boundaries of the original input EIS image, and the distances should all be less than threnshold_6, can be regarded as the fourth preset condition; threnshold_6 is the fourth threshold, and the fourth threshold can be 10% of the image size.

[0174] 1.1. abs(shiftSum20.x)<threnshold_7&&abs(shiftSum20.y)<threnshold_7&&

[0175] abs(tshift.x)<threshold_8&&abs(tshift.y)<threshold_8&&cumuDescCnt>0

[0176] The above is the third sub - preset condition. threnshold_7 is the fifth threshold; threshold_8 is the sixth threshold. cumuDescCnt is the third deceleration state value. tshift.x is the first pixel offset; tshift.y is the second pixel offset.

[0177] 1.2. shiftSumAll<threshold_9&&descCnt>=5&&accCnt<5&&abs(shiftSum10.x)<

[0178] threshold_10 && abs(shiftSum10.y) < threshold_10

[0179] The above is the fourth preset condition, threshold_9 is the seventh threshold, 5 is the eighth threshold, and threshold_10 is the ninth threshold.

[0180] First, condition 1 is satisfied, and then 1.1 or 1.2 is satisfied, then it enters the static holding state.

[0181] If the previous frame is in the static holding state, it is necessary to determine whether the current frame can enter the moving state according to the above statistical information.

[0182] 1. cumulativeShift > threshold_11 && (xAccCnt >= 5 || yAccCnt >= 5), this is the fifth sub-condition. threshold_11 is the tenth threshold.

[0183] 2. abs(shiftSum60.x) < threshold_12 && abs(shiftSum60.y) < threshold_12 && shiftMean > threshold_13, this is the sixth sub-condition. threshold_12 is the eleventh threshold. threshold_13 is the twelfth threshold.

[0184] 3. shiftMean > threshold_14, this is the seventh sub-condition. threshold_14 is the thirteenth threshold.

[0185] 4. Calculate the distances between the boundaries of the image output after anti-shake in the previous frame and the upper, lower, left, and right four boundaries of the original input EIS image. As long as there is a distance less than the threshold threshold_15 from a certain boundary, such as taking 10% of the image size, this is the eighth sub-condition. threshold_15 is the fourteenth threshold.

[0186] The fifth preset condition is that if any of the above conditions is satisfied, it enters the moving state from the static holding state.

[0187] S402. Based on the motion state of the current frame, update the initial anti-shake intensity to determine the intermediate anti-shake intensity of the current frame.

[0188] In some embodiments of the present application, when the motion state of the current frame is a static holding state, the initial anti-shake intensity is updated by the first default anti-shake intensity to obtain an intermediate anti-shake intensity; when the motion state of the current frame is a moving state, the initial anti-shake intensity is updated by the second default anti-shake intensity or the third default anti-shake intensity to obtain an intermediate anti-shake intensity.

[0189] In some embodiments of the present application, the static holding state corresponds to the first default anti-shake intensity, and the moving state is divided into fine-tuning the viewing angle and large movement. Fine-tuning the viewing angle corresponds to the second default anti-shake intensity; large movement corresponds to the third default anti-shake intensity.

[0190] In some embodiments of the present application, S402 can be implemented by S4021 or S4022, as follows:

[0191] S4021. When the motion state of the current frame is a static holding state, the initial anti-shake intensity is updated by the first default anti-shake intensity to obtain an intermediate anti-shake intensity.

[0192] S4022. When the motion state of the current frame is a moving state, the initial anti-shake intensity is updated by the second default anti-shake intensity or the third default anti-shake intensity to obtain an intermediate anti-shake intensity.

[0193] In some embodiments of the present application, the moving state includes fine-tuning the viewing angle or large movement.

[0194] In some embodiments of the present application, when the motion state of the current frame is fine-tuning the viewing angle, the initial anti-shake intensity is updated by the second default anti-shake intensity to obtain an intermediate anti-shake intensity.

[0195] In some embodiments of the present application, when the motion state of the current frame is large movement, the initial anti-shake intensity is updated by the third default anti-shake intensity to obtain an intermediate anti-shake intensity.

[0196] Exemplarily, the anti-shake intensities for different hand-held states are set as follows:

[0197] Static holding state, the first default anti-shake intensity still_strength = 1.0.

[0198] Fine-tuning the viewing angle, the second default anti-shake intensity follow_strength = 0.85 (reference value).

[0199] Large movement, the third default anti-shake intensity move_strength = 0.5.

[0200] If the current frame is in a static holding state after the above hand-held state recognition, the anti-shake intensity is directly updated to obtain an intermediate anti-shake intensity, as shown in formula (11):

[0201] targetStrength 中间 = still_strenghth (11)

[0202] If the current frame is in a moving state, further, if the change value of the single-frame Euler angle displacement is greater than the threshold value threshold_16 (such as 0.001) (i.e., for fine-tuning the viewing angle), then update the anti-shake strength to obtain the intermediate anti-shake strength, as shown in formula (12):

[0203] targetStrength 中间 = min(follow_strength, targetStrength) (12)

[0204] Among them, targetStrength 中间 is the intermediate anti-shake strength; follow_strength is the second default anti-shake strength; targetStrength is the initial anti-shake strength.

[0205] Similarly, if the current frame is in a moving state, further, if the change value of the single-frame Euler angle displacement is greater than the threshold value threshold_17 (such as 0.004) (i.e., for large-scale movement), then further update the anti-shake strength to obtain the intermediate anti-shake strength, as shown in formula (13):

[0206] targetStrength 中间 = min(move_strength, targetStrength) (13)

[0207] Among them, targetStrength 中间 is the intermediate anti-shake strength; move_strength is the third default anti-shake strength.

[0208] It can be understood that in the case where the motion state of the current frame is a static holding state, the initial anti-shake strength is updated through the first default anti-shake strength to obtain the intermediate anti-shake strength; in the case where the motion state of the current frame is a moving state, the initial anti-shake strength is updated through the second default anti-shake strength or the third default anti-shake strength to obtain the intermediate anti-shake strength. By updating the initial anti-shake strength through different motion states to obtain the intermediate anti-shake strength, the stability can be enhanced during static holding, the stability and followability can be balanced during fine-tuning of the viewing angle, and the followability can be enhanced during large-scale movement, so as to achieve an adaptive adjustment of the anti-shake effect.

[0209] S403. Update the intermediate anti-shake strength based on the motion state of the current frame and the anti-shake strength of the previous frame, and determine the final anti-shake strength of the current frame.

[0210] In some embodiments of the present application, if the motion state of the previous frame is a moving state and the motion state of the current frame is a static state, then the anti-shake intensity of the previous frame is updated to obtain the updated anti-shake intensity of the previous frame; and based on the updated anti-shake intensity of the previous frame, the intermediate anti-shake intensity is updated to determine the final anti-shake intensity of the current frame;

[0211] Exemplarily, correct the anti-shake intensity of the previous frame, take the larger value of 0.95 and the actual anti-shake intensity of the previous frame. At the same time, update the cumulative offset of the picture vector cumulativeOffset.x and y to 0. The update of the anti-shake intensity of the previous frame can be achieved through formula (14) as follows:

[0212] targetStrength_pre 更新后 =max(0.95,targetStrength_pre) (14)

[0213] where targetStrength_pre 更新后 is the updated anti-shake intensity of the previous frame; targetStrength_pre is the anti-shake intensity of the previous frame.

[0214] Based on the updated anti-shake intensity of the previous frame, the intermediate anti-shake intensity is updated to determine the final anti-shake intensity of the current frame, which can be achieved through formula (15) as follows:

[0215] targetStrength 最终 =targetStrength_pre 更新后 *weight+(1-weight)*targetStrength 中间 (15)

[0216] where targetStrength_pre 更新后 is the updated anti-shake intensity of the previous frame; targetStrength 最终 is the final anti-shake intensity of the current frame; targetStrength 中间 is the intermediate anti-shake intensity.

[0217] The update of the cumulative offset of the picture vector can be achieved through formulas (16) and (17) as follows:

[0218] cumulativeOffset.x=0 (16)

[0219] cumulativeOffset.y=0 (17)

[0220] Among them, the cumulative offset of the first frame vector is cumulativeOffset.x; the cumulative offset of the second frame vector is cumulativeOffset.y.

[0221] In some embodiments of the present application, if the motion state of the current frame is a moving state, or the motion states of the previous frame and the current frame are both static holding states, then based on the anti-shake strength of the previous frame, the intermediate anti-shake strength is updated to determine the final anti-shake strength of the current frame.

[0222] Exemplarily, in order to make the anti-shake effect of the image sequence smooth, based on the anti-shake strength of the previous frame, the intermediate anti-shake strength is updated to determine the final anti-shake strength of the current frame, which can be achieved by formula (18) as follows:

[0223] targetStrength 最终 = targetStrength_pre * weight + (1 - weight) * targetStrength 中间 (18)

[0224] Among them, targetStrength_pre is the anti-shake strength of the previous frame; targetStrengts 最终 is the final anti-shake strength of the current frame; targetStrength 中间 is the intermediate anti-shake strength.

[0225] It can be understood that based on the motion state, the initial anti-shake strength is updated to determine the intermediate anti-shake strength of the current frame; based on the motion state of the current frame and the anti-shake strength of the previous frame, the intermediate anti-shake strength is updated to determine the final anti-shake strength of the current frame, making the final anti-shake strength more accurate.

[0226] In some embodiments of the present application, Figure 6 is an optional process schematic diagram of a video anti-shake display method provided by an embodiment of the present application Figure 5 , as Figure 6 shown, S101 can be implemented through S501 - S505 as follows:

[0227] S501. Integrate based on the angular velocity data to obtain the attitude information corresponding to each frame of data.

[0228] S502. Based on the attitude information corresponding to each frame of data, determine the attitude difference between the current frame and the previous frame; and convert the attitude difference into three-axis Euler angles in three-dimensional space; among them, the three-axis Euler angles include the first Euler angle, the second Euler angle, and the third Euler angle.

[0229] In some embodiments of the present application, the three-axis Euler angles include a first Euler angle, a second Euler angle, and a third Euler angle.

[0230] In some embodiments of the present application, the terminal device may perform calculations based on the attitude information corresponding to each frame of data to determine the attitude difference between the current frame and the previous frame; and convert the attitude difference into three-axis Euler angles in three-dimensional space.

[0231] Exemplarily, based on the angular velocity data, integrating can obtain the attitude information corresponding to each frame of data, then calculate the attitude difference between two frames, and convert it into the Euler angles corresponding to the X / Y / Z axes, which can represent the jitter situation between frames.

[0232] S503. Determine the main direction axis of the current frame based on the first Euler angle, the second Euler angle, and the third Euler angle.

[0233] In some embodiments of the present application, based on the first Euler angle, the second Euler angle, and the third Euler angle, perform an absolute value operation, and record the axis corresponding to the maximum value among the three-axis Euler angles (taking the absolute value) as the main direction axis of the current frame.

[0234] S504. Determine the cumulative Euler angle change value based on the main direction axis of the current frame and the main direction axis of the obtained previous frame.

[0235] In some embodiments of the present application, if the main direction axis of the current frame is the same as the main direction axis of the obtained previous frame, then accumulate the current displacement in the cumulative Euler angle change value; if the main direction axis of the current frame is different from the main direction axis of the obtained previous frame, then set the cumulative Euler angle change value to 0.

[0236] Exemplarily, determine whether the main direction axis of the current frame is the same as the direction axis of the previous frame. If different, then set the cumulative Euler angle change value of the main direction axis to 0 and update the direction axis information; if the same, then accumulate the current displacement in the cumulative Euler angle distance. In this way, the larger the distance, the more obvious the user's intention to pan around a certain axis or the larger the movement in a single frame, and the smaller it is, the more likely it is a small movement or in a static holding state, with random small jitters.

[0237] S505. Perform calculations based on the cumulative Euler angle change value, the preset cumulative Euler angle threshold, and the preset anti-shake intensity to determine the initial anti-shake intensity of the current frame.

[0238] In some embodiments of the present application, the preset cumulative Euler angle threshold includes a minimum cumulative Euler angle threshold and a maximum cumulative Euler angle threshold; the preset anti-shake intensity includes a minimum anti-shake intensity and a maximum anti-shake intensity.

[0239] In some embodiments of the present application, based on the minimum cumulative Euler angle threshold and the maximum cumulative Euler angle threshold, the cumulative Euler angle change value is updated to obtain the updated cumulative Euler angle change value; based on the minimum cumulative Euler angle threshold, the maximum cumulative Euler angle threshold, and the updated cumulative Euler angle change value, a weight value is determined; based on the weight value, a weighted operation is performed on the minimum anti-shake strength and the maximum anti-shake strength to determine the initial anti-shake strength.

[0240] Exemplarily, for the calculation of the initial anti-shake strength, two thresholds are set. The minimum cumulative Euler angle threshold is min_distance (which can be 0.1 in radians), the maximum cumulative Euler angle threshold is max_distance (which can be 0.5 in radians), the minimum anti-shake strength is min_strength (which can be 0.7), and the maximum anti-shake strength is max_strength (which can be 0.9). Then, the initially calculated anti-shake strength targetStrength can be calculated according to the following formulas (19) - (21) as follows:

[0241] update_distance = max(min_distanc, min(max_distanc, distanc)) (19)

[0242] α = (update_distanc - min_distanc) / (max_distanc - min_distanc) (20)

[0243] tar getstrength = α * min_strength + (1 - α) * max_strenghth (21)

[0244] Wherein, update_distance is the updated cumulative Euler angle transformation value; ɑ is the weight value; targetstrength is the initial anti-shake strength.

[0245] It can be understood that based on the angular velocity data, integration is performed to obtain the attitude information corresponding to each frame of data; based on the attitude information corresponding to each frame of data, the attitude difference between the current frame and the previous frame is determined; and the attitude difference is converted into three-axis Euler angles in three-dimensional space; based on the first Euler angle, the second Euler angle, and the third Euler angle, the main direction axis of the current frame is determined; based on the main direction axis of the current frame and the main direction axis of the obtained previous frame, the cumulative Euler angle change value is determined; based on the cumulative Euler angle change value, the preset cumulative Euler angle threshold, and the preset anti-shake intensity, calculations are performed to determine the initial anti-shake intensity of the current frame. Since the cumulative Euler angle change value is determined based on two consecutive frames, and thus calculations are performed based on the cumulative Euler angle change value, the preset cumulative Euler angle threshold, and the preset anti-shake intensity to determine the initial anti-shake intensity of the current frame, the accuracy of the initial anti-shake intensity calculation can be improved.

[0246] In some embodiments of the present application, the video anti-shake display method further includes:

[0247] In the case where the current frame is the first frame, the angular velocity data of the first frame is obtained; and based on the angular velocity data of the first frame, the actual attitude of the first frame is determined.

[0248] Determine that the initial anti-shake intensity of the first frame is the maximum anti-shake intensity;

[0249] Based on the maximum anti-shake intensity and the second default anti-shake intensity, determine the final anti-shake intensity of the first frame;

[0250] Based on the final anti-shake intensity and the actual attitude of the first frame, determine the final reference attitude of the first frame.

[0251] In some embodiments of the present application, the detailed determination method of the final reference attitude of the first frame refers to the above calculation method and will not be elaborated here.

[0252] It can be understood that through the data, the actual attitude of the first frame is determined, the initial anti-shake intensity of the first frame is determined to be the maximum anti-shake intensity; based on the maximum anti-shake intensity and the second default anti-shake intensity, the final anti-shake intensity of the first frame is determined; based on the final anti-shake intensity and the actual attitude of the first frame, the final reference attitude of the first frame is determined, and projection and mapping processing are performed on the final reference attitude of the first frame to obtain the image after anti-shake of the current frame and display it, thereby achieving adaptive adjustment of the anti-shake effect.

[0253] In some embodiments of the present application, Figure 7 is an optional flow schematic diagram of a video anti-shake display method provided by an embodiment of the present application Figure 6 , as Figure 7 shown, the video anti-shake display method includes the following steps:

[0254] S1. Initial anti-shake intensity calculation.

[0255] In some embodiments of the present application, the calculation of the initial anti-shake intensity may include the following steps:

[0256] (1) Based on the angular velocity data, integrating can obtain the attitude information corresponding to each frame of data. Then, calculate the attitude difference between two frames and convert it into the Euler angles corresponding to the X / Y / Z axes, which can represent the jitter situation between frames.

[0257] (2) Calculate the axis (such as the x-axis, or y-axis, or z-axis) corresponding to the maximum value among the absolute values of the three-axis Euler angles, and record it as the main direction axis of the current frame. And record this Euler angle as the single-frame Euler angle change value dispalcement.

[0258] (3) Determine whether the main direction axis of the current frame is the same as that of the previous frame. If it is different, set the cumulative Euler angle change value of the main direction axis to 0 and update the direction axis information; if it is the same, accumulate the current dispalcement in the cumulative Euler angle distance. The larger the distance, the more obvious the user's intention to move the camera around a certain axis, or the larger the single-frame movement. The smaller it is, the more likely it is a small movement or a static holding state, with random small jitters.

[0259] (4) Calculate the initial anti-shake intensity. Set two thresholds, the minimum cumulative Euler angle threshold min_distance (which can be taken as 0.1 in radians), the maximum cumulative Euler angle threshold max_distance (which can be taken as 0.5 in radians), the minimum anti-shake intensity min_strength (which can be taken as 0.7), and the maximum anti-shake intensity max_strength (which can be taken as 0.9), so as to calculate the initial anti-shake intensity targetStrength.

[0260] S2. Calculate based on the statistical value information of different sliding window sizes.

[0261] In some embodiments of the present application, the calculation based on the statistical value information of different sliding window sizes may include the following steps:

[0262] (1) Based on the angular velocity data, calculate the attitude difference between two frames, and then convert the attitude difference into the x-direction offset and y-direction offset based on image pixels, and record them as tshift.x and tshift.y.

[0263] (2) Set sliding windows of different sizes, such as 10 frames, 15 frames, 20 frames, and 60 frames. The statistical information of sliding windows of different sizes can assist each other to more accurately reflect the current hand-held state. Record the accumulated pixel offsets of the historical 10 frames as shiftSum10.x and shiftSum10.y. Similarly, the accumulated pixel offsets corresponding to 15 frames, 20 frames, and 60 frames are shiftSum15.x, shiftSum15.y, shiftSum20.x, shiftSum20.y, shiftSum60.x, and shiftSum60.y.

[0264] (3) Calculate the average pixel offsets shiftMean.x and shiftMean.y.

[0265] (4) Calculate the cumulative vector offset shiftSumAll of 15 frames.

[0266] (5) Calculate the acceleration and deceleration state values. Currently, multiple methods are used to statistically analyze the state information of acceleration and deceleration.

[0267] A. Calculate the difference between two adjacent data in sumQue and compare it with a preset threshold. If the condition is met, the deceleration frame count descCnt is incremented by 1, or the acceleration frame count accCnt is incremented by 1. The specific calculation method is shown in Formula (4) and Formula (5).

[0268] B. As described above, based on the angular velocity data, integrating it can obtain the actual attitude corresponding to each frame of the image, denoted as curQ. After anti-shake, the actual attitude curQ of the current frame will be rotated to the reference attitude refQ. The calculation of refQ for the current frame is obtained by weighting the refQ_pre of the previous frame and the actual attitude curQ of the current frame using the initial anti-shake strength targetStrength.

[0269] Therefore, for each frame of the image, the attitude difference between curQ and refQ can be calculated and converted to image pixels to obtain the offsets diffShift.x and diffShift.y before and after anti-shake. The larger the absolute value of this value, the stronger the anti-shake stability and the worse the followability. The smaller the value, the stronger the followability of the anti-shake and the worse the stability. At the same time, by tracking the change of this value, it is also possible to detect whether there is an obvious subjective intention to move the camera, so as to balance the stability and followability of the anti-shake.

[0270] Maintain a queue diffQue with a maximum size of 5 to cache diffShift.x and diffShift.y of the adjacent 5 frames. Similarly, by calculating the difference between two adjacent data, the acceleration frame counts xAccCnt and yAccCnt in the x or y direction can be obtained. The specific calculation method is shown in Formula (7) and Formula (8).

[0271] C. Set the cumulative pixel offsets of the screen starting from rest or slight jitter as cumulativeOffset.x and cumulativeOffset.y, with the initial values being 0.

[0272] If Condition 1 is met: if either cumulativeOffset.x or cumulativeOffset.y statistically calculated in the previous frame is greater than threshold_3 (such as 1000), then reset cumulativeOffset.x and cumulativeOffset.y to 0.

[0273] If Condition 1 is not met, but Condition 2 is met: Denote the anti-shake strength of the previous frame as targetStrengt_pre. The identification of "static holding state" and "moving state" in the following formula is mentioned in the third part [Handheld State Identification] later:

[0274] if(targetStrength_pre>threshold_4||cumulativeOffset.x>0||cumulativeOffset.y>0)&& the previous frame is in the static holding state &&(diffShift.x>threnshold_5||diffShift.y>threnshold_5),

[0275] then cumulativeOffset.x = cumulativeOffset.x + tshift.x

[0276] cumulativeOffset.y = cumulativeOffset.y + tshift.y

[0277] If conditions 1 and 2 are not met, but condition 3 is met: the previous frame is in a moving state, then update cumulativeOffset.x = cumulativeOffset.x + tshift.x, cumulativeOffset.y = cumulativeOffset.y + tshift.y, to obtain the updated cumulativeOffset.x and cumulativeOffset.y of the current frame, and then convert them into the cumulative offset cumulativeShift in the vector direction. The calculation of cumulativeShift refers to formula (9), and a queue cumuShiftQue with a maximum size of 5 is maintained to cache the cumulativeShift information of the adjacent 5 frames. Calculate the difference between two adjacent data in cumuShiftQue, and if it is greater than 0, increment the deceleration frame count cumuDescCnt by 1, specifically referring to formula (10).

[0278] S3. Judgment of the handheld state.

[0279] In some embodiments of the present application, this part mainly distinguishes the scene of the static holding state and the moving state through the above statistical values. It should be noted that the handheld state is the motion state.

[0280] (1) If the previous frame is in a moving state, it is necessary to judge whether the current frame can enter the static holding state according to the above statistical information.

[0281] 1. Calculate the distances between the upper, lower, left, and right boundaries of the image output after anti-shake in the previous frame and the original input EIS image. The distances should all be less than threnshold_6, such as taking 10% of the image size

[0282] 1.1 abs(shiftSum20.x) < threnshold_7 && abs(shiftSum20.y) < threnshold_7 &&

[0283] abs(tshift.x) < threshold_8 && abs(tshift.y) < threshold_8 && cumuDescCnt > 0

[0284] 1.2 shiftSumAll < threshold_9 && descCnt >= 5 && accCnt < 5 && abs(shiftSum10.x) <

[0285] threshold_10 && abs(shiftSum10.y) < threshold_10

[0286] First, condition 1 is satisfied. Then, if 1.1 or 1.2 is satisfied, it enters the static holding state. Further, correct the anti-shake intensity of the previous frame, taking the greater value of 0.95 and the actual anti-shake intensity of the previous frame. At the same time, update the cumulative offsets cumulativeOffset.x and y to 0.

[0287] (2) If the previous frame is in the static holding state, it is necessary to determine whether the current frame can enter the moving state based on the above statistical information

[0288] 1.cumulativeShift>threshold_11&&(xAccCnt>=5||yAccCnt>=5)

[0289] 2.abs(shiftSum60.x)<threshold_12&&abs(shiftSum60.y)<threshold_12&&shiftMean>

[0290] threshold_13

[0291] 3.shiftMean>threshold_14

[0292] 4. Calculate the distances between the boundaries of the image output after anti-shake in the previous frame and the top, bottom, left, and right four boundaries of the original input EIS image. As long as there is a distance less than the threshold threshold_15 from a certain boundary, for example, take 10% of the image size.

[0293] If any of the above conditions is satisfied, it enters the moving state from the static holding state.

[0294] S4. Anti-shake intensity update and filtering smoothing.

[0295] In some embodiments of the present application, set the anti-shake intensities for different handheld states:

[0296] In the static holding state, the first default anti-shake intensity still_strength = 1.0.

[0297] When fine-tuning the viewing angle, the second default anti-shake intensity follow_strength = 0.85 (reference value).

[0298] When moving significantly, the third default anti-shake intensity move_strength = 0.5.

[0299] Update the initial anti-shake intensity through the first default anti-shake intensity, the second default anti-shake intensity, and the third default anti-shake intensity to obtain the final anti-shake intensity. For details, see formulas (11) - (18).

[0300] S5. Anti-shake grid calculation and image mapping.

[0301] In some embodiments of the present application, based on the calculated reference posture, the reference posture is projected into two-dimensional grid points using a camera model, and the image is warped using the grid points to finally obtain a stabilized image sequence.

[0302] It is understandable that this application proposes a method for determining motion scenes for telephoto lens anti-shake based on data, and also provides a method for calculating anti-shake strength under different motion scenes, which is used to achieve real-time anti-shake for telephoto photo preview. Based on the specific user usage techniques of telephoto lenses, this application conducts more targeted mobile phone motion state recognition, and uses data combined with different sliding windows for statistics. It defines states and calculates anti-shake strength for static, fine-tuning of viewing angle, and large movements in mobile states. It can achieve enhanced stability when static, balanced stability and hand tracking when fine-tuning the viewing angle, and enhanced hand tracking when moving significantly, thereby achieving an adaptive anti-shake effect.

[0303] Based on the video stabilization display method of the above embodiment, the embodiment of the present application also provides a video stabilization display device, such as Figure 8 As shown, Figure 8 The structure diagram of a video stabilization display device provided in an embodiment of the present application is as follows. The video stabilization display device 8 includes: a determination unit 801, an update unit 802 and a display unit 803; wherein,

[0304] The determining unit 801 is configured to determine an initial anti-shake strength of a current frame based on the acquired angular velocity data of the sensor; wherein the current frame is any frame other than the first frame; and determine statistical value information based on the angular velocity data and at least two preset sliding windows; wherein the size of each of the at least two preset sliding windows is different;

[0305] The updating unit 802 is used to update the initial anti-shake strength based on the statistical value information to determine the final anti-shake strength of the current frame;

[0306] The determining unit 801 is further configured to determine a final reference posture of the current frame based on the final anti-shake strength and the angular velocity data;

[0307] The display unit 803 is used to perform projection and mapping processing on the final reference posture to obtain and display the image after the anti-shake of the current frame.

[0308] In some embodiments of the present application, the video stabilization display device 8 further includes: an acquisition unit 804; wherein,

[0309] The obtaining unit 804 is configured to perform integration based on the angular velocity data to obtain the attitude information corresponding to each frame of data;

[0310] The determining unit 801 is further configured to determine the attitude difference between the current frame and the previous frame based on the attitude information corresponding to each frame of data; and convert the attitude difference into a pixel offset; wherein the pixel offset includes a first pixel offset and a second pixel offset; and determine the statistical value information based on the pixel offset and the at least two preset sliding windows.

[0311] In some embodiments of the present application, the obtaining unit 804 is further configured to perform an operation on the pixel offset through the at least two preset sliding windows to obtain the accumulated pixel offset corresponding to each of the at least two preset sliding windows;

[0312] The determining unit 801 is further configured to perform an operation based on the accumulated pixel offset to determine the average pixel offset of the current frame and the accumulated vector offset of the current frame; determine the acceleration / deceleration state value based on the accumulated vector offset of the current frame, the attitude information corresponding to each frame of data, and the pixel offset; and determine the statistical value information based on the pixel offset, the accumulated pixel offset, the average pixel offset, the accumulated vector offset of the current frame, and the acceleration / deceleration state value.

[0313] In some embodiments of the present application, the acceleration / deceleration state value includes a first acceleration / deceleration state value, a second acceleration state value, a third acceleration state value, and a third deceleration state value;

[0314] The determining unit 801 is further configured to determine the first acceleration / deceleration state value based on the accumulated vector offset of the current frame and the accumulated vector offset of the previous frame obtained; determine the anti-shake offset of the current frame based on the attitude information corresponding to each frame of data; and perform an operation based on the anti-shake offset of the current frame to determine the second acceleration state value and the third acceleration state value;

[0315] The obtaining unit 804 is further configured to obtain the screen pixel offset of the previous frame;

[0316] The determining unit 801 is further configured to determine the screen pixel offset of the current frame based on the screen pixel offset of the previous frame, the pixel offset, and the anti-shake offset of the current frame; and determine the third deceleration state value based on the screen pixel offset of the current frame.

[0317] In some embodiments of the present application, the determining unit 801 is further configured to determine the actual pose of the current frame and the reference pose of the previous frame based on the pose information corresponding to each frame of data; perform a weighted operation on the actual pose of the current frame and the reference pose of the previous frame through the initial anti-shake intensity to determine the reference pose of the current frame; and determine the pose difference of the current frame based on the actual pose of the current frame and the reference pose of the current frame.

[0318] The obtaining unit 804 is further configured to convert the pose difference of the current frame to obtain the anti-shake offset of the current frame.

[0319] The determining unit 801 is further configured to perform an operation based on the obtained anti-shake offset of the previous frame and the anti-shake offset of the current frame to determine the second acceleration state value and the third acceleration state value.

[0320] In some embodiments of the present application, the determining unit 801 is further configured to perform an operation based on the pixel offset of the current frame to determine the vector offset of the current frame; and perform an operation based on the obtained vector offset of the previous frame and the vector offset of the current frame to determine the third deceleration state value.

[0321] In some embodiments of the present application, the determining unit 801 is further configured to, if the pixel offset of the previous frame meets the first preset condition, set the pixel offset of the previous frame to zero and determine the pixel offset of the current frame; if the pixel offset of the previous frame does not meet the first preset condition, and the pixel offset of the previous frame and the anti-shake offset of the current frame meet the second preset condition, or the pixel offset of the previous frame does not meet the first preset condition, the pixel offset of the previous frame and the anti-shake offset of the current frame do not meet the second preset condition, and meet the third preset condition, update the pixel offset of the previous frame through the pixel offset to determine the pixel offset of the current frame.

[0322] In some embodiments of the present application, the determining unit 801 is further configured to determine the motion state corresponding to the current frame based on the statistical value information; update the initial anti-shake intensity based on the motion state of the current frame to determine the intermediate anti-shake intensity of the current frame; and update the intermediate anti-shake intensity based on the motion state of the current frame and the anti-shake intensity of the previous frame to determine the final anti-shake intensity of the current frame.

[0323] In some embodiments of the present application, the determining unit 801 is further configured to determine the motion state of the previous frame; in the case where the motion state of the previous frame is a moving state, if the statistical value information meets the fourth preset condition, determine that the motion state of the current frame is a static state; if the statistical value information does not meet the fourth preset condition, determine that the motion state of the current frame is a moving state; in the case where the motion state of the previous frame is a static state, if the statistical value information meets the fifth preset condition, determine that the motion state of the current frame is a moving state; if the statistical value information does not meet the fifth preset condition, determine that the motion state of the current frame is a static state.

[0324] In some embodiments of the present application, the determining unit 801 is further configured to, in the case where the motion state of the current frame is a static state, update the initial anti-shake intensity through the first default anti-shake intensity to obtain the intermediate anti-shake intensity; in the case where the motion state of the current frame is a moving state, update the initial anti-shake intensity through the second default anti-shake intensity or the third default anti-shake intensity to obtain the intermediate anti-shake intensity.

[0325] In some embodiments of the present application, the obtaining unit 804 is further configured to, in the case where the motion state of the current frame is a fine-tuning perspective, update the initial anti-shake intensity through the second default anti-shake intensity to obtain the intermediate anti-shake intensity; in the case where the motion state of the current frame is a large-scale movement, update the initial anti-shake intensity through the third default anti-shake intensity to obtain the intermediate anti-shake intensity.

[0326] In some embodiments of the present application, the determining unit 801 is further configured to, if the motion state of the previous frame is a moving state and the motion state of the current frame is a static state, update the anti-shake intensity of the previous frame to obtain the updated anti-shake intensity of the previous frame; and based on the updated anti-shake intensity of the previous frame, update the intermediate anti-shake intensity to determine the final anti-shake intensity of the current frame; if the motion state of the current frame is a moving state, or the motion states of the previous frame and the current frame are both static states, update the intermediate anti-shake intensity based on the anti-shake intensity of the previous frame to determine the final anti-shake intensity of the current frame.

[0327] In some embodiments of the present application, the determining unit 801 is further configured to determine the reference attitude of the previous frame and the actual attitude of the current frame based on the angular velocity data; perform weighted processing on the reference attitude of the previous frame and the actual attitude of the current frame through the final anti-shake intensity to determine the final reference attitude of the current frame.

[0328] In some embodiments of the present application, the obtaining unit 804 is further configured to perform integration based on the angular velocity data to obtain the attitude information corresponding to each frame of data;

[0329] The determining unit 801 is further configured to determine the attitude difference between the current frame and the previous frame based on the attitude information corresponding to each frame of data; and convert the attitude difference into three-axis Euler angles in three-dimensional space; wherein the three-axis Euler angles include a first Euler angle, a second Euler angle, and a third Euler angle; determine the main direction axis of the current frame based on the first Euler angle, the second Euler angle, and the third Euler angle; determine the cumulative Euler angle change value based on the main direction axis of the current frame and the main direction axis of the obtained previous frame; perform calculations based on the cumulative Euler angle change value, a preset cumulative Euler angle threshold, and a preset anti-shake intensity to determine the initial anti-shake intensity of the current frame.

[0330] In some embodiments of the present application, the obtaining unit 804 is further configured to update the cumulative Euler angle change value based on the minimum cumulative Euler angle threshold and the maximum cumulative Euler angle threshold to obtain an updated cumulative Euler angle change value;

[0331] The determining unit 801 is further configured to determine a weight value based on the minimum cumulative Euler angle threshold, the maximum cumulative Euler angle threshold, and the updated cumulative Euler angle change value; perform a weighted operation on the minimum anti-shake intensity and the maximum anti-shake intensity based on the weight value to determine the initial anti-shake intensity.

[0332] In some embodiments of the present application, the obtaining unit 804 is further configured to obtain the angular velocity data of the first frame when the current frame is the first frame;

[0333] The determining unit 801 is further configured to determine the actual attitude of the first frame based on the angular velocity data of the first frame; determine the initial anti-shake intensity of the first frame as the maximum anti-shake intensity; determine the final anti-shake intensity of the first frame based on the maximum anti-shake intensity and the second default anti-shake intensity; determine the final reference attitude of the first frame based on the final anti-shake intensity and the actual attitude of the first frame.

[0334] Based on the video anti-shake display method of the above embodiments, an embodiment of the present application further provides a terminal device, such as Figure 9 shown, Figure 9 is a schematic structural diagram of a terminal device provided by an embodiment of the present application. The terminal device 9 includes: a processor 901 and a memory 902. The memory 902 is used to store a computer program; the processor 901 is used to call and run the computer program from the memory to execute the video anti-shake display method as described in the above embodiments.

[0335] In an embodiment of the present application, the above-mentioned processor 901 may be at least one of an Application Specific Integrated Circuit (ASIC), a Digital Signal Processor (DSP), a Digital Signal Processing Device (DSPD), a Programmable Logic Device (PLD), a Field Programmable Gate Array (FPGA), a Central Processing Unit (CPU), a controller, a microcontroller, and a microprocessor. It can be understood that for different devices, the electronic devices used to implement the functions of the above-mentioned processor may also be others, and the embodiments of the present application do not make specific limitations.

[0336] The embodiments of the present application provide a computer program product, including a computer program or instruction, which when executed by a processor, implements the display method for video anti-shake described in any of the above embodiments.

[0337] Exemplarily, the program instructions corresponding to a display method for video anti-shake in this embodiment may be stored on storage media such as an optical disc, a hard disk, and a USB flash drive. When the program instructions corresponding to a display method for video anti-shake in the storage media are read or executed by an electronic device, the display method for video anti-shake described in any of the above embodiments can be implemented.

[0338] In addition, in the embodiments of the present application, each functional module may be integrated in a processing unit, or each unit may exist physically alone, or two or more units may be integrated in one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of a software functional module.

[0339] When the integrated unit is implemented in the form of a software functional module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment, in essence, or the part that contributes to the prior art, or all or part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to execute all or part of the steps of the method of this embodiment. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROM), random access memories (RAM), magnetic disks, or optical discs.

[0340] It should be understood that the "one embodiment" or "an embodiment" or "some embodiments" mentioned throughout the specification means that the specific features, structures, or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, the "in one embodiment" or "in an embodiment" or "in some embodiments" that appear throughout the specification do not necessarily refer to the same embodiment. In addition, these specific features, structures, or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the magnitudes of the serial numbers of the above processes do not mean the order of execution is prior or subsequent. The order of execution of each process should be determined by its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present application. The serial numbers of the embodiments of the present application above are only for description and do not represent the advantages or disadvantages of the embodiments. The descriptions of the above embodiments tend to emphasize the differences between the embodiments, and their similarities or similarities can be referred to each other. For the sake of brevity, they will not be elaborated herein.

[0341] The modules described as separate components above may or may not be physically separated. The components shown as modules may or may not be physical modules; they can be located in one place or distributed to multiple network units; some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0342] In addition, in each embodiment of the present application, all the functional modules can be integrated in a processing unit, or each module can be separately used as a unit, or two or more modules can be integrated in a unit; the above integrated modules can be implemented in the form of hardware or in the form of hardware plus software functional units.

[0343] Those of ordinary skill in the art will understand that all or part of the steps to implement the above method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps including the above method embodiments; and the aforementioned storage medium includes: various media that can store program codes such as removable storage devices, read-only memory (ROM), magnetic disks, or optical discs.

[0344] The methods disclosed in several method embodiments provided by the embodiments of the present application can be arbitrarily combined without conflict to obtain new method embodiments.

[0345] The features disclosed in several product embodiments provided by the embodiments of the present application can be arbitrarily combined without conflict to obtain new product embodiments.

[0346] The features disclosed in several method or device embodiments provided by the embodiments of the present application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0347] As mentioned above, it is only the implementation manner of the embodiments of the present application, but the protection scope of the embodiments of the present application is not limited thereto. Any person skilled in the art within the technical scope disclosed in the present application can easily think of changes or substitutions, which should all be covered within the protection scope of the embodiments of the present application. Therefore, the protection scope of the embodiments of the present application should be subject to the protection scope of the claims.

Claims

1. A display method for video anti-shake, characterized in that, The method includes: Based on the acquired angular velocity data of the sensor, determining the initial anti-shake intensity of the current frame; wherein, the current frame is any frame other than the first frame; Based on the angular velocity data and at least two preset sliding windows, determining statistical value information; wherein, the size of each of the at least two preset sliding windows is different; Based on the statistical value information, updating the initial anti-shake intensity to determine the final anti-shake intensity of the current frame; Based on the final anti-shake intensity and the angular velocity data, determining the final reference pose of the current frame; Performing projection and mapping processing on the final reference pose to obtain the image after anti-shake of the current frame and displaying it.

2. The method according to claim 1, wherein The determining the statistical value information based on the angular velocity data and at least two preset sliding windows includes: Based on the angular velocity data, performing integration to obtain the pose information corresponding to each frame of data; Based on the pose information corresponding to each frame of data, determining the pose difference between the current frame and the previous frame; and converting the pose difference into a pixel offset; wherein, the pixel offset includes a first pixel offset and a second pixel offset; Based on the pixel offset and the at least two preset sliding windows, determining the statistical value information.

3. The method according to claim 1 or 2, characterized in that, The determining the statistical value information based on the pixel offset and the at least two preset sliding windows includes: Performing operations on the pixel offset through the at least two preset sliding windows to obtain the accumulated pixel offsets corresponding to the at least two preset sliding windows respectively; Based on the accumulated pixel offsets, performing operations to determine the average pixel offset of the current frame and the accumulated vector offset of the current frame; Based on the accumulated vector offset of the current frame, the pose information corresponding to each frame of data, and the pixel offset, determining an acceleration / deceleration state value; Based on the pixel offset, the accumulated pixel offsets, the average pixel offset, the accumulated vector offset of the current frame, and the acceleration / deceleration state value, determining the statistical value information.

4. The method according to claim 3, characterized in that The acceleration / deceleration state value includes a first acceleration / deceleration state value, a second acceleration state value, a third acceleration state value, and a third deceleration state value; The determining the acceleration / deceleration state value based on the accumulated vector offset of the current frame, the pose information corresponding to each frame of data, and the pixel offset includes: Based on the accumulated vector offset of the current frame and the accumulated vector offset of the acquired previous frame, determining the first acceleration / deceleration state value; Based on the pose information corresponding to each frame of data, determining the anti-shake offset of the current frame; and based on the anti-shake offset of the current frame, performing operations to determine the second acceleration state value and the third acceleration state value; Obtaining the pixel offset of the previous frame of the image; Based on the pixel offset of the previous frame of the image, the pixel offset, and the anti-shake offset of the current frame, determining the pixel offset of the current frame of the image; Based on the pixel offset of the current frame of the image, determining the third deceleration state value.

5. The method according to claim 4, wherein The determining the anti-shake offset of the current frame based on the pose information corresponding to each frame of data; And perform operations based on the anti-shake offset of the current frame to determine the second acceleration state value and the third acceleration state value, including: Determine the actual pose of the current frame and the reference pose of the previous frame based on the pose information corresponding to each frame of data; Perform a weighted operation on the actual pose of the current frame and the reference pose of the previous frame through the initial anti-shake intensity to determine the reference pose of the current frame; Determine the pose difference of the current frame based on the actual pose of the current frame and the reference pose of the current frame; Convert the pose difference of the current frame to obtain the anti-shake offset of the current frame; Perform operations based on the obtained anti-shake offset of the previous frame and the anti-shake offset of the current frame to determine the second acceleration state value and the third acceleration state value.

6. The method according to claim 4, characterized in that, The determining the third deceleration state value based on the picture pixel offset of the current frame includes: Perform operations based on the picture pixel offset of the current frame to determine the picture vector offset of the current frame; Perform operations based on the obtained picture vector offset of the previous frame and the picture vector offset of the current frame to determine the third deceleration state value.

7. The method according to claim 4, characterized in that, The determining the picture pixel offset of the current frame based on the picture pixel offset of the previous frame, the pixel offset, and the anti-shake offset of the current frame includes: If the picture pixel offset of the previous frame meets the first preset condition, set the picture pixel offset of the previous frame to zero and determine the picture pixel offset of the current frame; If the picture pixel offset of the previous frame does not meet the first preset condition, and the picture pixel offset of the previous frame and the anti-shake offset of the current frame meet the second preset condition, or the picture pixel offset of the previous frame does not meet the first preset condition, the picture pixel offset of the previous frame and the anti-shake offset of the current frame do not meet the second preset condition, and meet the third preset condition, then update the picture pixel offset of the previous frame through the pixel offset to determine the picture pixel offset of the current frame.

8. The method according to claim 7, wherein The picture pixel offset includes: a first picture pixel offset and a second picture pixel offset; the anti-shake offset includes: a first anti-shake offset and a second anti-shake offset; The first preset condition is that the first picture pixel offset or the second picture pixel offset of the previous frame is greater than a first threshold; The second preset condition is that the picture pixel offset of the previous frame and the final anti-shake intensity of the previous frame meet the first sub-preset condition, the previous frame is in a static holding state, and the anti-shake offset of the current frame meets the second sub-preset condition; where The first sub-preset condition is that at least one of the final anti-shake intensity of the previous frame being greater than a second threshold, the first picture pixel offset of the previous frame being greater than zero, and the second picture pixel offset of the previous frame being greater than zero is satisfied; The second sub-preset condition is that at least one of the first anti-shake offset of the current frame being greater than a third threshold and the second anti-shake offset of the current frame being greater than a third threshold is satisfied; The third preset condition is that the previous frame is in a moving state.

9. The method according to any one of claims 1-8, characterized in that, Updating the initial anti-shake intensity based on the statistical value information to determine the final anti-shake intensity of the current frame includes: Determining the motion state corresponding to the current frame based on the statistical value information; Updating the initial anti-shake intensity based on the motion state of the current frame to determine the intermediate anti-shake intensity of the current frame; Updating the intermediate anti-shake intensity based on the motion state of the current frame and the anti-shake intensity of the previous frame to determine the final anti-shake intensity of the current frame.

10. The method according to claim 9, characterized in that, Determining the motion state corresponding to the current frame based on the statistical value information includes: Determining the motion state of the previous frame; When the motion state of the previous frame is in a moving state, if the statistical value information meets the fourth preset condition, determining that the motion state of the current frame is in a static holding state; If the statistical value information does not meet the fourth preset condition, determining that the motion state of the current frame is in a moving state; When the motion state of the previous frame is in a static holding state, if the statistical value information meets the fifth preset condition, determining that the motion state of the current frame is in a moving state; If the statistical value information does not meet the fifth preset condition, determining that the motion state of the current frame is in a static holding state.

11. The method according to claim 10, wherein The accumulated pixel offset includes the accumulated pixel offset of the first preset sliding window and the accumulated pixel offset of the second preset sliding window; each accumulated pixel offset includes a first accumulated pixel offset and a second accumulated pixel offset; The fourth preset condition is that the distances between the boundaries of the image output after anti-shake of the previous frame and the four boundaries of the original input image are all less than the fourth threshold, and the accumulated pixel offset, the third acceleration / deceleration state value, and the pixel offset meet the third sub-preset condition, or the cumulative vector offset of the current frame, the first acceleration / deceleration state value, and the accumulated pixel offset meet the fourth sub-preset condition; wherein, the first acceleration / deceleration state value includes a first acceleration state value and a first deceleration state value; The third sub-preset condition is that the absolute value of the first accumulated pixel offset of the second preset sliding window is less than the fifth threshold, the absolute value of the second accumulated pixel offset of the second preset sliding window is less than the fifth threshold, the absolute value of the first pixel offset is less than the sixth threshold, the absolute value of the second pixel offset is less than the sixth threshold, and the third acceleration / deceleration state value is greater than zero; The fourth sub-preset condition is that the cumulative vector offset of the current frame is less than the seventh threshold, the first acceleration state value is less than the eighth threshold, the first deceleration state value is greater than or equal to the eighth threshold, the first accumulated pixel offset of the first preset sliding window is less than the ninth threshold, and the second accumulated pixel offset of the first preset sliding window is less than the ninth threshold.

12. The method according to claim 10, wherein The accumulated pixel offset further includes the accumulated pixel offset of the third preset sliding window; The accumulated pixel offset of the third preset sliding window includes: the first accumulated pixel offset of the third preset sliding window and the second accumulated pixel offset of the third preset sliding window; The fifth preset condition is to satisfy any one of a fifth sub - condition, a sixth sub - condition, a seventh sub - condition, and an eighth sub - condition; where The fifth sub - condition is that the screen vector offset is greater than a tenth threshold, the second acceleration state value is greater than or equal to an eighth threshold, and the third acceleration state value is less than the eighth threshold; The sixth sub - condition is that the absolute value of the first cumulative pixel offset of the third preset sliding window is less than an eleventh threshold, the absolute value of the second cumulative pixel offset of the third preset sliding window is less than the eleventh threshold, and the average pixel offset is greater than a twelfth threshold; The seventh sub - condition is that the average pixel offset is greater than a thirteenth threshold; The eighth sub - condition is that the distance between the boundary of the image output after anti - shake in the previous frame and any one of the four boundaries of the original image is less than a fourteenth threshold.

13. The method according to claim 9, wherein Updating the initial anti - shake intensity based on the motion state of the current frame to determine the intermediate anti - shake intensity of the current frame includes: When the motion state of the current frame is a static holding state, updating the initial anti - shake intensity through the first default anti - shake intensity to obtain the intermediate anti - shake intensity; When the motion state of the current frame is a moving state, updating the initial anti - shake intensity through the second default anti - shake intensity or the third default anti - shake intensity to obtain the intermediate anti - shake intensity.

14. The method according to claim 13, characterized in that The moving state includes fine - tuning the viewing angle or large - scale movement; When the motion state of the current frame is a moving state, updating the initial anti - shake intensity through the second default anti - shake intensity or the third default anti - shake intensity to obtain the intermediate anti - shake intensity includes: When the motion state of the current frame is fine - tuning the viewing angle, updating the initial anti - shake intensity through the second default anti - shake intensity to obtain the intermediate anti - shake intensity; When the motion state of the current frame is large - scale movement, updating the initial anti - shake intensity through the third default anti - shake intensity to obtain the intermediate anti - shake intensity.

15. The method according to claim 9, wherein Updating the intermediate anti - shake intensity based on the motion state of the current frame and the anti - shake intensity of the previous frame to determine the final anti - shake intensity of the current frame includes: If the motion state of the previous frame is a moving state and the motion state of the current frame is a static holding state, then updating the anti - shake intensity of the previous frame to obtain the updated anti - shake intensity of the previous frame; and updating the intermediate anti - shake intensity based on the updated anti - shake intensity of the previous frame to determine the final anti - shake intensity of the current frame; If the motion state of the current frame is a moving state, or the motion states of both the previous frame and the current frame are static holding states, then updating the intermediate anti - shake intensity based on the anti - shake intensity of the previous frame to determine the final anti - shake intensity of the current frame.

16. The method according to any one of claims 1 to 15, characterized in that, Determining the final reference pose of the current frame based on the final anti - shake intensity and the angular velocity data includes: Based on the angular velocity data, determining the reference pose of the previous frame and the actual pose of the current frame; Based on the final anti-shake intensity, perform weighted processing on the reference pose of the previous frame and the actual pose of the current frame to determine the final reference pose of the current frame.

17. The method according to any one of claims 1 to 15, characterized in that, The determining the initial anti-shake intensity of the current frame based on the obtained angular velocity data of the sensor includes: Based on the angular velocity data, perform integration to obtain the pose information corresponding to each frame of data; Based on the pose information corresponding to each frame of data, determine the pose difference between the current frame and the previous frame; and convert the pose difference into three-axis Euler angles in three-dimensional space; wherein, the three-axis Euler angles include a first Euler angle, a second Euler angle, and a third Euler angle; Based on the first Euler angle, the second Euler angle, and the third Euler angle, determine the main direction axis of the current frame; Based on the main direction axis of the current frame and the main direction axis of the obtained previous frame, determine the cumulative Euler angle change value; Based on the cumulative Euler angle change value, a preset cumulative Euler angle threshold, and a preset anti-shake intensity, perform calculations to determine the initial anti-shake intensity of the current frame.

18. The method according to claim 17, wherein The preset cumulative Euler angle threshold includes a minimum cumulative Euler angle threshold and a maximum cumulative Euler angle threshold; the preset anti-shake intensity includes a minimum anti-shake intensity and a maximum anti-shake intensity; The performing calculations based on the cumulative Euler angle change value, the preset cumulative Euler angle threshold, and the preset anti-shake intensity to determine the initial anti-shake intensity of the current frame includes: Based on the minimum cumulative Euler angle threshold and the maximum cumulative Euler angle threshold, update the cumulative Euler angle change value to obtain an updated cumulative Euler angle change value; Based on the minimum cumulative Euler angle threshold, the maximum cumulative Euler angle threshold, and the updated cumulative Euler angle change value, determine a weight value; Based on the weight value, perform a weighted operation on the minimum anti-shake intensity and the maximum anti-shake intensity to determine the initial anti-shake intensity.

19. The method according to claim 1, characterized in that, The method further includes: In the case where the current frame is the first frame, obtain the angular velocity data of the first frame; and based on the angular velocity data of the first frame, determine the actual pose of the first frame; Determine that the initial anti-shake intensity of the first frame is the maximum anti-shake intensity; Based on the maximum anti-shake intensity and a second default anti-shake intensity, determine the final anti-shake intensity of the first frame; Based on the final anti-shake intensity and the actual pose of the first frame, determine the final reference pose of the first frame.

20. A display device with video anti-shake, characterized in that The display device for video anti-shake includes: a determination unit, an update unit, and a display unit; wherein, The determination unit is configured to determine the initial anti-shake intensity of the current frame based on the obtained angular velocity data of the sensor; wherein, the current frame is any frame other than the first frame; based on the angular velocity data and at least two preset sliding windows, determine statistical value information; wherein, the size of each of the at least two preset sliding windows is different; The update unit is configured to update the initial anti-shake intensity based on the statistical value information to determine the final anti-shake intensity of the current frame; The determination unit is further configured to determine the final reference pose of the current frame based on the final anti-shake intensity and the angular velocity data; The display unit is configured to perform projection and mapping processing on the final reference pose, obtain the image after anti-shake processing of the current frame, and display the image.

21. A terminal device, characterized in that, It includes a processor and a memory. The memory is used to store a computer program, and the processor is configured to call and run the computer program stored in the memory to execute the method according to any one of claims 1 to 19.

22. A computer program product, characterized in that, It includes a computer program or instruction, and when the computer program or instruction is executed by a processor, the method according to any one of claims 1 to 19 is implemented.

Images