Video anti-shake method and device, storage medium and video acquisition equipment

Abstract

A video anti-shake method and device, a storage medium and a video acquisition device. The video anti-shake method is applied to a video acquisition device including a gyroscope, and includes: when ambient brightness meets a preset low-illumination condition, performing video anti-shake based on electronic anti-shake, wherein the electronic anti-shake includes: determining a first relative displacement of an image collected by the video acquisition device according to data of the gyroscope, and correcting the image collected by the video acquisition device according to the first relative displacement. The scheme provided in the embodiment uses electronic anti-shake in a low-illumination environment. In a low-illumination environment, the accuracy of feature point recognition and matching is relatively low, and there is a large error. If digital anti-shake is used, the anti-shake effect is often not ideal. The use of electronic anti-shake can improve the anti-shake effect in a low-illumination scene.

Description

Technical Field

[0001] This article relates to video processing technology, and in particular to a video stabilization method and device, storage medium, and video acquisition equipment. Background Technology

[0002] In the field of video surveillance, cameras are often installed in outdoor natural environments, such as on poles or towers. These cameras are prone to mechanical vibration under wind or other conditions, resulting in shaky video footage. Video shakiness can distort or blur images, causing dizziness and making it difficult to observe useful information. It can even affect the identification and tracking of structured elements in the image, impacting intelligent monitoring operations. Therefore, video shakiness poses a significant threat in the surveillance field. Summary of the Invention

[0003] This application provides a video stabilization method and apparatus, storage medium, and video acquisition device, which can achieve video stabilization.

[0004] This application provides a video stabilization method, applied to a video acquisition device including a gyroscope, comprising:

[0005] When the ambient brightness meets the preset low-light conditions, video stabilization is performed based on electronic image stabilization. The electronic image stabilization includes: determining the first relative displacement of the image captured by the video acquisition device based on the data from the gyroscope, and correcting the image captured by the video acquisition device based on the first relative displacement.

[0006] In an exemplary embodiment, the method further includes: determining the jitter intensity of the image based on the data from the gyroscope; and performing video stabilization based on digital stabilization when the ambient brightness does not meet a preset low-light condition and the jitter intensity is less than or equal to a preset jitter threshold; the digital stabilization includes: determining a second relative displacement based on images of different video frames captured by the video acquisition device, and correcting the images captured by the video acquisition device based on the second relative displacement.

[0007] In one exemplary embodiment, the method further includes performing video stabilization based on the electronic image stabilization when the ambient brightness does not meet the preset low-light conditions and the jitter intensity is greater than the preset jitter threshold.

[0008] In an exemplary embodiment, determining the jitter intensity of the image based on the data from the gyroscope includes: acquiring data from the gyroscope within a first preset time period, the data including angular velocities of multiple coordinate axes, calculating the average of the absolute values ​​of the angular velocities of each coordinate axis, and taking the maximum value among the average values ​​as the jitter intensity.

[0009] In one exemplary embodiment, the ambient brightness meeting the preset low-light condition includes: when the brightness value of the image acquired by the video acquisition device is less than a preset brightness threshold and the gain value is greater than a preset gain threshold, the preset low-light condition is met.

[0010] In an exemplary embodiment, determining the second relative displacement based on images of different video frames acquired by the video acquisition device includes:

[0011] Obtain multiple feature points from adjacent video frames and determine the relative displacement of the multiple feature points;

[0012] The relative displacement threshold range is determined based on the gyroscope data between adjacent video frames, or the relative displacement estimate is determined.

[0013] Determine qualified feature points among the plurality of feature points, and determine the second relative displacement based on the relative displacement of the qualified feature points, wherein the relative displacement of the qualified feature points is within the range of the relative displacement threshold, or the absolute value of the difference between the relative displacement of the qualified feature points and the estimated relative displacement is less than or equal to a preset threshold.

[0014] In one exemplary embodiment, determining the relative displacement threshold range based on gyroscope data between adjacent video frames includes:

[0015] Acquire gyroscope data within a second preset time period when the video acquisition device is stationary, and determine the root mean square ω of the gyroscope data within the second preset time period. rms ;

[0016] Obtain the gyroscope data ω based on the adjacent video frames, and based on ω-ω rms Determine the minimum relative displacement based on ω+ω rms The maximum relative displacement is determined, and the range from the minimum relative displacement to the maximum relative displacement is the relative displacement threshold range.

[0017] In an exemplary embodiment, the step of performing video stabilization based on digital stabilization when the ambient brightness does not meet the preset low light conditions and the jitter intensity is less than or equal to the preset jitter threshold includes: when the video acquisition device is in a non-autofocus process, and the ambient brightness does not meet the preset low light conditions and the jitter intensity is less than or equal to the preset jitter threshold, performing video stabilization based on digital stabilization.

[0018] The method further includes, when the video acquisition device is in the autofocus process, performing image stabilization based on the electronic image stabilization, and correcting the image of the video acquisition device according to the data of the gyroscope, so that adjacent video frames are in the same field of view, calculating the image sharpness evaluation value, and performing autofocus according to the image sharpness evaluation value.

[0019] In one exemplary embodiment, the method further includes controlling the exposure time of the video acquisition device to satisfy at least one of the following:

[0020]

[0021]

[0022] The h T W is the threshold value in the first direction. h Let θ be the length of the image in the first direction. h Let ω be the current field of view of the video acquisition device in the first direction, and v be the average angular velocity of the video acquisition device jitter during the exposure time. T θ is the threshold in the second direction. v W represents the current field of view of the video acquisition device in the second direction, and W represents the length of the image in the second direction.

[0023] This disclosure provides a video stabilization device, including a memory and a processor. The memory stores a program, which, when read and executed by the processor, implements the video stabilization method described in any of the above embodiments.

[0024] This disclosure provides a computer-readable storage medium storing one or more programs that can be executed by one or more processors to implement the video stabilization method described in any of the above embodiments.

[0025] This disclosure provides a video capture device, including the aforementioned video stabilization device.

[0026] This application includes a video stabilization method and apparatus, a storage medium, and a video acquisition device. The video stabilization method is applied to a video acquisition device including a gyroscope and includes: when the ambient brightness meets preset low-light conditions, performing video stabilization based on electronic image stabilization. The electronic image stabilization includes: determining a first relative displacement of the image acquired by the video acquisition device based on data from the gyroscope, and correcting the image acquired by the video acquisition device based on the first relative displacement. The solution provided in this embodiment uses electronic image stabilization in low-light environments. In low-light environments, the accuracy of feature point recognition and matching is low, resulting in large errors. If digital image stabilization is used, the stabilization effect is often unsatisfactory. Using electronic image stabilization can improve the stabilization effect in low-light scenes.

[0027] Other features and advantages of this application will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing the application. Other advantages of this application can be realized and obtained by means of the solutions described in the description and the accompanying drawings. Attached Figure Description

[0028] The accompanying drawings are used to provide an understanding of the technical solutions of this application and constitute a part of the specification. They are used together with the embodiments of this application to explain the technical solutions of this application and do not constitute a limitation on the technical solutions of this application.

[0029] Figure 1 A schematic diagram of the coordinate system of the video acquisition device and the coordinate system of the gyroscope.

[0030] Figure 2 This is a flowchart of a video stabilization method provided in an embodiment of this application;

[0031] Figure 3 This is a schematic diagram of a video stabilization device provided in an embodiment of this application. Detailed Implementation

[0032] This application describes several embodiments, but these descriptions are exemplary and not restrictive, and it will be apparent to those skilled in the art that many more embodiments and implementations are possible within the scope of the embodiments described herein. Although many possible combinations of features are shown in the drawings and discussed in the detailed description, many other combinations of the disclosed features are also possible. Unless specifically limited, any feature or element of any embodiment may be used in combination with, or may replace, any feature or element of any other embodiment.

[0033] This application includes and contemplates combinations of features and elements known to those skilled in the art. The embodiments, features, and elements disclosed in this application may also be combined with any conventional features or elements to form a unique inventive scheme as defined by the claims. Any feature or element of any embodiment may also be combined with features or elements from other inventive schemes to form another unique inventive scheme as defined by the claims. Therefore, it should be understood that any feature shown and / or discussed in this application may be implemented individually or in any suitable combination. Therefore, the embodiments are not limited except by the limitations imposed by the appended claims and their equivalents. Furthermore, various modifications and changes may be made within the scope of the appended claims.

[0034] Furthermore, in describing representative embodiments, the specification may have presented methods and / or processes as a specific sequence of steps. However, the method or process should not be limited to the specific order of steps described herein, to the extent that it does not depend on such a specific order. As will be understood by those skilled in the art, other sequences of steps are also possible. Therefore, the specific order of steps set forth in the specification should not be construed as a limitation of the claims. Moreover, the claims concerning the method and / or process should not be limited to the steps performed in the written order, and those skilled in the art will readily understand that these orders can be varied and still remain within the spirit and scope of the embodiments of this application.

[0035] There are three solutions to video shakiness. The first is optical image stabilization (OIS). OIS uses a gyroscope mounted on the camera to detect camera shake, and then uses an optical correction mechanism on the lens or image sensor to change the angle of light entering the lens, thereby counteracting the camera's own shake and stabilizing continuously shot video. The second is digital image stabilization (DIS). DIS uses feature point recognition and matching between adjacent video frames to calculate the relative displacement of feature points in adjacent frames. Then, it selects one frame as the base frame and uses radial changes to correct the pixels in other video frames, achieving video stabilization. The third is electronic image stabilization (EIS). EIS uses an electronic gyroscope to detect camera shake and then uses radial changes to correct the relative displacement of the camera image based on the shake. Each of these methods has its advantages and disadvantages. Optical image stabilization is more effective, but it requires special lenses or sensors, is more expensive, and currently suffers from technical problems such as insufficient correction displacement and system instability. In low-light environments, digital image stabilization often results in less than ideal stabilization due to lower accuracy in feature point recognition and matching. Electronic image stabilization technology has relatively simple hardware requirements; it only requires adding a gyroscope to a regular camera.

[0036] This disclosure provides a video acquisition device, which may include a gyroscope and a video acquisition unit (such as a camera). A gyroscope is a microelectromechanical system (MEMS) sensor that can measure three-dimensional rotational angular velocity. The gyroscope and the video acquisition unit can be rigidly connected; for example, the gyroscope can be soldered onto the main control circuit board or the image sensor circuit board of the video acquisition unit. The main control chip of the video acquisition unit and the gyroscope sensor can be connected via interfaces such as a Serial Peripheral Interface (SPI) or an Inter-Integrated Circuit Bus (I2C), but are not limited to these.

[0037] This disclosure discloses a video stabilization method applied to a video acquisition device including a gyroscope, comprising:

[0038] When the ambient brightness meets the preset low-light conditions, video stabilization is performed based on electronic image stabilization. The electronic image stabilization includes: determining the first relative displacement of the image captured by the video acquisition device based on the data from the gyroscope, and correcting the image captured by the video acquisition device based on the first relative displacement.

[0039] The solution provided in this embodiment uses electronic image stabilization in low-light environments. In low-light environments, the accuracy of feature point recognition and matching is low, resulting in large errors. If digital image stabilization is used, the stabilization effect is often unsatisfactory. Using electronic image stabilization can improve the stabilization effect in low-light scenarios.

[0040] In an exemplary embodiment, electronic image stabilization is achieved by integrating the real-time rotation angle of the video acquisition device based on the real-time angular velocity of the gyroscope, obtaining the first relative displacement of the same spatial point on different video frames based on the camera imaging model, and then cropping, translating and magnifying the video frames based on the first relative displacement, so that the relative displacement of the same spatial point on different video frames is zero, thereby achieving video stabilization.

[0041] In an exemplary embodiment, before determining the first relative displacement of the image captured by the video acquisition device based on the gyroscope data, the method further includes: determining the zero bias value of the gyroscope, and performing zero bias correction on the gyroscope data based on the zero bias value. The zero bias value can be determined before the video acquisition device operates. That is, the working process of the video acquisition device can be divided into a preparation stage and an operating stage, with the zero bias value determined during the preparation stage. During the preparation stage, the correspondence between the coordinate systems of the gyroscope and the video acquisition device can also be determined, and the gyroscope error can be calculated. The main control chip of the video acquisition device can read the gyroscope data in real time, which generally includes the angular velocity ω of the gyroscope in three dimensions. g =(ω x ω y ω z The coordinate system of the video acquisition device and the coordinate system of the gyroscope can be transformed by a matrix, i.e., ω = ω g *M, where ω is the angular velocity data in the coordinate system of the video acquisition device obtained after transformation. The transformation matrix M can be determined based on the installation position of the gyroscope. In this embodiment, for example... Figure 1 As shown,

[0042]

[0043] Theoretically, the angular velocity output by a gyroscope should be zero when stationary. However, in reality, the angular velocity at rest is often not zero and exhibits random fluctuations. This random fluctuation is the system's random error, and the average value at rest is the gyroscope's zero bias value. Therefore, zero bias correction must be performed on the gyroscope before use. One method for zero bias correction is to place the video acquisition device on a stationary workbench and continuously test to obtain gyroscope data ω(i) (i = 0, 1, 2…) over a second preset time period, and calculate the average value ω of this data sequence. drift and root mean square ω rms , will ω drift As the zero bias value, the average error of the three coordinate axes is denoted as ω. e (i.e. ω) drift (The average of the three coordinate values). Gyroscope data read during normal operation is corrected for zero bias before use: ω = ω - ω drift The length of the second preset duration can be set as needed. Root mean square ω rms For subsequent use.

[0044] In one exemplary embodiment, the ambient brightness satisfying the preset low-light condition includes: when the brightness value of the image captured by the video acquisition device is less than a preset brightness threshold and the gain value is greater than a preset gain threshold, the preset low-light condition is satisfied. That is:

[0045]

[0046] Wherein, luma is the brightness value of the image captured by the video capture device, lumaT is the preset brightness threshold, Gain is the gain value of the image captured by the video capture device, and GainT is the preset gain threshold. lumaT and GainT can be determined by testing.

[0047] The method for determining ambient brightness provided in this embodiment is only an example. Other methods can be used to determine whether the ambient brightness meets the preset low-light conditions. For example, the ambient brightness can be determined directly based on the light brightness collected by the light sensor of the video acquisition device.

[0048] In an exemplary embodiment, the method further includes: determining the jitter intensity of the image based on data from the gyroscope; and performing video stabilization based on digital stabilization when the ambient brightness does not meet a preset low-light condition and the jitter intensity is less than or equal to a preset jitter threshold. The digital stabilization includes: determining a second relative displacement based on images of different video frames captured by the video acquisition device, and correcting the images captured by the video acquisition device based on the second relative displacement. The solution provided in this embodiment performs digital stabilization in non-low-light environments (where the preset low-light condition is not met) and when the jitter is relatively small. Since electronic stabilization is less effective in low-speed jitter scenarios, the solution provided in this embodiment can improve the stabilization effect in low-speed jitter scenes. The solution provided in this embodiment avoids the shortcomings of both digital and electronic stabilization algorithms, selecting the appropriate stabilization algorithm based on different ambient brightness and jitter intensity, thus improving the stabilization effect in low-light and low-speed jitter scenarios.

[0049] Because gyroscopes inherently have systematic errors ω rms If the video capture device is stationary, calculating the relative displacement using the gyroscope's angular velocity will reveal a certain displacement between two frames. Using this displacement to correct the video will cause random jitter in the originally static video, consistent with the gyroscope's random error. Therefore, only when the camera's actual jitter is significant (e.g., greater than the average error) will the gyroscope's signal-to-noise ratio be high, and using gyroscope data to correct the video will produce a more ideal effect. For example, ω can be taken as... T =k*ω e As a preset jitter threshold, k is an empirical coefficient, for example, k is greater than 1. If ω s >ω T If the gyroscope exhibits inherent noise, it will perform electronic image stabilization (EIS); otherwise, it will perform digital image stabilization (DIS). Because gyroscopes inherently contain noise, random fluctuations occur when stationary or experiencing minor shaking, resulting in less effective EIS at low speeds. The solution provided in this embodiment employs DIS to improve image stabilization in low-speed shaking scenarios.

[0050] In an exemplary embodiment, the method further includes performing video stabilization based on electronic image stabilization when the ambient brightness does not meet preset low-light conditions and the jitter intensity is greater than a preset jitter threshold. The solution provided in this embodiment performs electronic image stabilization in non-low-light environments with significant jitter. When the jitter is significant, the stabilization effect of electronic image stabilization is superior to that of digital image stabilization. Therefore, using electronic image stabilization can improve the stabilization effect when the jitter is significant.

[0051] In an exemplary embodiment, determining the image jitter intensity based on the gyroscope data includes: acquiring multiple data ω = (ω_0.05) within a first preset time period t1 of the gyroscope.x ω y ω z ), calculate the average of the absolute values ​​of the angular velocities of each coordinate axis (ω). sx ω sy ω sz ), take the average value ω sx ω sy ω sz The maximum value in the range is taken as the jitter intensity. The method for determining jitter intensity provided in this embodiment is only an example, and jitter intensity can be determined by other methods.

[0052] In an exemplary embodiment, determining the second relative displacement based on images of different video frames acquired by the video acquisition device includes:

[0053] Obtain multiple feature points from adjacent video frames and determine the relative displacement of the multiple feature points;

[0054] The relative displacement threshold range is determined based on gyroscope data between adjacent video frames;

[0055] The second relative displacement is determined based on feature points whose relative displacement falls within the relative displacement threshold range. The solution provided in this embodiment eliminates feature points outside the relative displacement threshold range and determines the second relative displacement only based on feature points within the relative displacement threshold range. This improves the anti-interference capability of digital image stabilization and also solves the background dragging problem caused by feature point matching errors and interference from moving objects.

[0056] In an exemplary embodiment, determining the relative displacement threshold range based on gyroscope data between adjacent video frames includes:

[0057] Acquire gyroscope data within a second preset time period when the video acquisition device is stationary, and determine the root mean square of the gyroscope data;

[0058] The relative displacement threshold range is determined based on the gyroscope data between adjacent video frames and the root mean square.

[0059] The following is a specific example to illustrate this.

[0060] Digital image stabilization can include calculating the displacement between adjacent video frames by identifying and matching feature points. Typically, multiple feature points are matched, and multiple relative displacements V(x(k), y(k)) are calculated based on these feature points, where k = 0, 1, 2, ..., N. Here, x(k) and y(k) represent the horizontal and vertical displacements of the image, respectively, in pixels, and N represents the number of relative displacements in the set. One digital image stabilization method directly calculates the average of these N data points as the global average relative displacement between the two frames, and uses the first frame as a reference to correct the images of other frames. These N data points may contain abnormal data due to matching errors or interference from moving objects; in this embodiment, gyroscope data can be used to remove abnormal data. The specific procedure is as follows: During the transition from frame i to frame j, the main control chip acquires M angular velocity data ω(t) from the gyroscope. Integrating the angular velocities of the gyroscope yields the three-dimensional rotation angles (α(t), β(t), γ(t)). Based on these three-dimensional rotation angles (α(t), β(t), γ(t)), the rotation matrix R(t) of the video acquisition device can be obtained as follows:

[0061]

[0062] According to the camera mapping model, if a point A in 3D space has imaging coordinates a on the image, then...

[0063] a=KR(t)A

[0064]

[0065] Where K and f are intrinsic parameters of the video capture device, (o x o y () is the origin of the image coordinate system.

[0066] The images of A in frame i and frame j are respectively,

[0067] a i =KR(t) i A,a j =KR(t) j A

[0068] Wherein, R(t) i ) is the integral obtained by integrating the gyroscope data from the reference time point to the time of the i-th frame. i ),β(t i ),γ(t i The rotation matrix R(t) is calculated. j (α(t)) is obtained by integrating the gyroscope data from the reference time point to the time frame j. j ),β(t j),γ(t j The calculated rotation matrix. The reference time point can be set as needed, for example, it could be the time of the i-th frame.

[0069] This allows us to determine the relative position of 'a' in frame i and frame j.

[0070] a j =KR(t) j )RT(t i )K -1 a i

[0071] Furthermore, the relative displacement of the imaging points on the i-th and j-th frames can be obtained as follows:

[0072] a ji =a j -a i

[0073] Since these M gyroscope data ω(t) include the average error ω rms Therefore, it is impossible to accurately obtain the true jitter, but the maximum deviation ω(t)+ω can be estimated. rms and minimum deviation ω(t)-ω rms According to ω(t)+ω rms The maximum displacement 'a' of the two video frames can be obtained. jimax According to ω(t)-ω rms The minimum displacement 'a' between two video frames can be obtained. jimin The minimum displacement a jimin To the maximum displacement a jimax The range of values ​​formed is used as the relative displacement threshold range. Where, a jimax =(x max y max ), a jimin =(x min y min ).

[0074] Frame i and frame j can be adjacent video frames, or they can be non-adjacent video frames. We will explain this using the case where frames i and j are adjacent video frames. Based on frames i and j, N feature points are matched. N relative displacements V(x(k), y(k)) are calculated from these N feature points, where k = 0, 1, 2, ..., N. These N relative displacements V(x(k), y(k)) are then evaluated. If the relative displacement of a feature point exceeds the minimum displacement x... jimin To the maximum displacement x jimax If the relative displacement threshold range is defined, the feature point is considered an outlier, i.e., x(k) is located within x. min To x maxOutside the range, or y(k) is located in y min To y max Points outside the defined range are considered outliers. The set after removing outliers is, for example, V(x(k),y(k)), where k = 0, 1, 2, ..., N1. The average value of the N1 data points is calculated and used as the global average relative displacement between the two frames, i.e., the second relative displacement. The i-th frame can be used as a reference for video correction processing.

[0075] In another embodiment, a can be obtained ji =a j -a i Then, a ji =(x es y es As the relative displacement estimate, a preset threshold α is used. T =(x T y T The relative displacements of the feature points satisfy |x(k)-x es |Greater than x T Or |y(k)-y es |greater than y T When this characteristic point is an anomalous characteristic point, the relative displacement of the characteristic point satisfies |x(k)-x es | Less than or equal to x T And |y(k)-y es | Less than or equal to y T At that time, the feature point is a qualified feature point; or, it can be determined by a ji The distance between the feature point and V(x(k), y(k)) is used to determine whether the feature point is a qualified feature point, and so on.

[0076] In an exemplary embodiment, the step of performing video stabilization based on digital stabilization when the ambient brightness does not meet the preset low light conditions and the jitter intensity is less than or equal to the preset jitter threshold includes: when the video acquisition device is in a non-autofocus process, and the ambient brightness does not meet the preset low light conditions and the jitter intensity is less than or equal to the preset jitter threshold, performing video stabilization based on digital stabilization.

[0077] The method further includes, when the video acquisition device is in the autofocus process, performing image stabilization based on the electronic image stabilization, and correcting the image of the video acquisition device according to the data from the gyroscope, calculating the image sharpness evaluation value after adjacent video frames are in the same field of view, and performing autofocus based on the image sharpness evaluation value. For high-magnification zoom cameras, shaking usually leads to images that cannot be focused clearly, and the image blurriness, in turn, prevents digital image stabilization from finding effective feature points and thus cannot solve the shaking problem. The solution provided in this embodiment uses electronic image stabilization during the autofocus process, solving the cycle problem of images that cannot be focused clearly due to shaking and images that cannot be stabilized due to blurriness. During the autofocus process, electronic image stabilization maintains a stable field of view, achieving clear focus. If digital image stabilization is used subsequently, the accuracy of feature point recognition can be guaranteed. The solution provided in this embodiment enables zoom cameras to achieve ideal focusing and image stabilization effects.

[0078] Autofocus controls motor movement based on the image sharpness rating (FV) to obtain the motor position corresponding to the sharpest image. When the camera shakes significantly, the difference in the field of view between adjacent frames leads to significant differences in information within that field of view. The FV trend reflects the combined effects of motor movement and image shake, failing to accurately reflect the changes in image sharpness caused by motor movement. In this case, electronic image stabilization (EIS) can be applied first, cropping the differing fields of view between adjacent frames so that the FV values ​​are calculated within the same field of view. This allows the FV trend to more accurately reflect the changes in image sharpness caused by motor movement, achieving accurate autofocus. If, based on ambient brightness and shake intensity, digital image stabilization should be used, it should be executed after focusing is complete and the image is clear. After focusing, the stabilization method (digital or electronic) can be re-evaluated and selected, or the stabilization method determined before focusing can be used.

[0079] During video recording, camera shake can cause motion blur, resulting in ghosting of the captured target and thus image blur. Ghosting blur is evaluated using the displacement (h, v) of the ghosting within the exposure time; that is, a displacement of h in the first direction and a displacement of v in the second direction. The unit of displacement can be pixels, and the first and second directions can be perpendicular. Taking the first direction as an example, if the target moves a distance h relative to the image within the exposure time due to camera movement, the displacement h is required to be... <h T Only in this way can we ensure that the image is not blurry, where h T This is the threshold value in the first direction. Based on the relationship between rotational displacement and angle,

[0080] h=ω*t*D

[0081] Where ω is the average angular velocity of the video capture device during the exposure time, which can be determined based on gyroscope data, and D is the distance between the target and the video capture device, which can be in pixels. According to the definition of field of view...

[0082] tan(θ h / 2)=(W h / 2) / D

[0083] Among them W h θ represents the length of the image in the first direction, in pixels. h Let be the current field of view of the video acquisition device in the first direction. Based on the three formulas above, we can obtain:

[0084] ω*t*W h / (2*tan(θ h / 2))<h T

[0085] so,

[0086]

[0087] That is, when running the video stabilization algorithm, the exposure time is controlled to meet the following requirements. This can prevent noticeable motion blur in the first direction, resulting in a more ideal video stabilization effect.

[0088] Similarly, in the second direction, control

[0089]

[0090] This ensures that the image does not exhibit noticeable ghosting in the second direction, where v T θ is the threshold in the second direction. v W represents the current field of view of the video acquisition device in the second direction. v This represents the length of the image in the second direction, and the unit can be pixels.

[0091] In one exemplary embodiment, control can be implemented. or or and

[0092] Figure 2 This is a flowchart illustrating a video stabilization method as an exemplary embodiment. In this embodiment, the stabilization algorithm can be selected periodically, but this embodiment is not limited to this; it can be non-periodic, for example, the stabilization algorithm can be selected based on user instructions, or it can be selected when triggered by an event, etc. Figure 2 As shown, the video stabilization method provided in this embodiment includes:

[0093] Step 201: When the decision period arrives, determine whether the ambient brightness meets the preset low illumination conditions. If the ambient brightness meets the preset low illumination conditions, proceed to step 205. If the ambient brightness does not meet the preset low illumination conditions, proceed to step 202.

[0094] Step 202: Determine the jitter intensity based on the gyroscope data;

[0095] Step 203: Determine whether the jitter intensity is greater than a preset jitter threshold. If the jitter intensity is greater than the preset jitter threshold, proceed to step 205; if the jitter intensity is less than or equal to the preset jitter threshold, proceed to step 204.

[0096] Step 204: Use digital image stabilization to stabilize the video, then proceed to step 206;

[0097] In the process of digital image stabilization, gyroscope data can be used to filter the matched feature points, but the embodiments of this disclosure are not limited to this and feature point filtering may not be required.

[0098] Step 205: Use electronic image stabilization to stabilize the video, then proceed to step 206;

[0099] Step 206: Determine if the focusing process is in progress. If yes, proceed to step 207. If not, wait for the decision period to arrive and proceed to step 201.

[0100] Step 207: If digital image stabilization is currently in use, switch to electronic image stabilization. If electronic image stabilization is in use, keep it unchanged. Correct the image based on gyroscope data so that adjacent video frames are in the same field of view. Calculate the image sharpness evaluation value and perform automatic focusing based on the image sharpness evaluation value.

[0101] like Figure 3 As shown, this embodiment of the present disclosure provides a video stabilization device 70, including a memory 710 and a processor 720. The memory 710 stores a program, which, when read and executed by the processor 720, implements the video stabilization method described in any embodiment.

[0102] This disclosure provides a computer-readable storage medium storing one or more programs that can be executed by one or more processors to implement the video stabilization method described in any embodiment.

[0103] This disclosure provides a video capture device, including the aforementioned video stabilization device.

[0104] It will be understood by those skilled in the art that all or some of the steps, systems, or apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all components may be implemented as software executed by a processor, such as a digital signal processor or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software may be distributed on a computer-readable medium, which may include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

Claims

1. A video stabilization method, characterized in that, Applied to video capture devices including gyroscopes, including: When the ambient brightness meets the preset low-light conditions, video stabilization is performed based on electronic image stabilization. The electronic image stabilization includes: determining the first relative displacement of the image captured by the video acquisition device based on the data of the gyroscope, and correcting the image captured by the video acquisition device based on the first relative displacement. The method further includes: determining the jitter intensity of the image based on the data from the gyroscope; when the ambient brightness does not meet the preset low-light conditions and the jitter intensity is less than or equal to the preset jitter threshold, performing video stabilization based on digital stabilization; the digital stabilization includes: determining a second relative displacement based on the images of different video frames captured by the video acquisition device, and correcting the images captured by the video acquisition device based on the second relative displacement; When the ambient brightness does not meet the preset low-light conditions and the jitter intensity is greater than the preset jitter threshold, video stabilization is performed based on the electronic image stabilization.

2. The video stabilization method according to claim 1, characterized in that, The step of determining the image jitter intensity based on the gyroscope data includes: acquiring data from the gyroscope within a first preset time period, the data including angular velocities of multiple coordinate axes, calculating the average of the absolute values ​​of the angular velocities of each coordinate axis, and taking the maximum value among the average values ​​as the jitter intensity.

3. The video stabilization method according to claim 1, characterized in that, The ambient brightness meeting the preset low-light condition includes: when the brightness value of the image acquired by the video acquisition device is less than a preset brightness threshold and the gain value is greater than a preset gain threshold, the preset low-light condition is met.

4. The video stabilization method according to claim 1, characterized in that, Determining the second relative displacement based on images of different video frames acquired by the video acquisition device includes: Obtain multiple feature points from adjacent video frames and determine the relative displacement of the multiple feature points; The relative displacement threshold range is determined based on the gyroscope data between adjacent video frames, or the relative displacement estimate is determined. Determine qualified feature points among the plurality of feature points, and determine the second relative displacement based on the relative displacement of the qualified feature points, wherein the relative displacement of the qualified feature points is within the range of the relative displacement threshold, or the absolute value of the difference between the relative displacement of the qualified feature points and the estimated relative displacement is less than or equal to a preset threshold.

5. The video stabilization method according to claim 4, characterized in that, The relative displacement threshold range is determined based on gyroscope data between adjacent video frames, including: Acquire gyroscope data within a second preset time period when the video acquisition device is stationary, and determine the root mean square ω of the gyroscope data within the second preset time period. rms ; Obtain the gyroscope data ω based on the adjacent video frames, and based on ω-ω rms Determine the minimum relative displacement based on ω+ω rms The maximum relative displacement is determined, and the range from the minimum relative displacement to the maximum relative displacement is the relative displacement threshold range.

6. The video stabilization method according to claim 1, characterized in that, The method of performing video stabilization based on digital stabilization when the ambient brightness does not meet the preset low light conditions and the jitter intensity is less than or equal to the preset jitter threshold includes: when the video acquisition device is in a non-autofocus process, and the ambient brightness does not meet the preset low light conditions and the jitter intensity is less than or equal to the preset jitter threshold, performing video stabilization based on digital stabilization. The method further includes, when the video acquisition device is in the autofocus process, performing image stabilization based on the electronic image stabilization, and correcting the image of the video acquisition device according to the data of the gyroscope, so that adjacent video frames are in the same field of view, calculating the image sharpness evaluation value, and performing autofocus according to the image sharpness evaluation value.

7. The video stabilization method according to any one of claims 1 to 6, characterized in that, The method further includes controlling the exposure time of the video acquisition device to satisfy at least one of the following: The h T W is the threshold value in the first direction. h Let θ be the length of the image in the first direction. h Let ω be the current field of view of the video acquisition device in the first direction, and v be the average angular velocity of the video acquisition device jitter during the exposure time. T θ is the threshold in the second direction. v W represents the current field of view of the video acquisition device in the second direction. v The length of the image in the second direction.

8. A video stabilization device, characterized in that, The device includes a memory and a processor, wherein the memory stores a program that, when read and executed by the processor, implements the video stabilization method as described in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores one or more programs, which can be executed by one or more processors to implement the video stabilization method as described in any one of claims 1 to 7.

10. A video acquisition device, characterized in that, Includes the video stabilization device as described in claim 8.

Images