Video stabilization method, device and electronic equipment

Abstract

The application provides a video stabilization method, device and electronic equipment, wherein the video stabilization method comprises the following steps: acquiring a first imaging device pose corresponding to a current frame image, and a second imaging device pose corresponding to a previous frame image or a previous key frame image, and determining a pose change amount; acquiring a first optical flow used for compensating the pose change amount; fusing the pose change amount and the first optical flow to acquire a second optical flow; acquiring an imaging device global motion trajectory; performing smoothing processing on the imaging device global motion trajectory to acquire a smoothed trajectory; and acquiring a current frame image subjected to the video stabilization processing. In the implementation process of the above scheme, the pose change amount between frames is first determined, and then the compensation optical flow is used to compensate the pose change amount between frames, so that the characteristics that the pose change amount represents large-scale shaking and the compensation optical flow compensates small-scale shaking are fully utilized, the accuracy of motion estimation is improved, and the video stabilization effect of the above video stabilization method is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a video stabilization method and device and electronic equipment. BACKGROUND

[0002] Electronic image stabilization is a technology for smoothing compensation of a dynamic image sequence acquired by an imaging device with random motion or jitter to obtain a smooth video output.

[0003] The electronic image stabilization technology in the prior art generally only uses pose or only uses optical flow estimation to obtain inter-frame motion, and then performs image stabilization optimization. The error between the estimation of inter-frame motion and the real motion is large, resulting in poor image stabilization effect. SUMMARY

[0004] The purpose of the embodiments of the present application is to provide a video stabilization method, device and electronic equipment to improve the image stabilization effect.

[0005] In a first aspect, the embodiments of the present application provide a video stabilization method, comprising: acquiring a first imaging device pose corresponding to a current frame image in a video to be stabilized, and a second imaging device pose corresponding to a previous frame image or a previous key frame image, and determining a pose change amount; acquiring a first optical flow used for compensating the pose change amount according to the pose change amount; fusing the pose change amount and the first optical flow to acquire a second optical flow; acquiring an imaging device global motion trajectory according to the second optical flow; performing smoothing processing on the imaging device global motion trajectory to acquire a smoothed trajectory; and performing image transformation on the current frame image according to the smoothed trajectory to acquire a current frame image after image stabilization processing.

[0006] In the implementation process of the above scheme, the pose change amount between frames is first determined, and then the compensation optical flow is used to compensate the pose change amount between frames. The characteristics of using the pose change amount to represent large jitter and using the compensation optical flow to compensate small jitter are fully utilized, the accuracy of motion estimation is improved, and the image stabilization effect of the above video stabilization method is improved.

[0007] In an implementation manner of the first aspect, the acquiring of the first optical flow used for compensating the pose change amount according to the pose change amount comprises: performing alignment processing on the previous frame image or the previous key frame image according to the pose change amount to acquire an intermediate image; and acquiring a first optical flow between the intermediate image and the current frame image.

[0008] In the implementation process of the scheme, the intermediate image can be obtained by aligning the previous frame image and the previous key frame image according to the pose change amount, and then the optical flow between the intermediate image and the current frame image is calculated, that is, the first optical flow used for compensating the pose change amount is obtained. The compensation effect of the first optical flow on the pose change amount is good, so that the video stabilization method can realize more accurate inter-frame motion estimation, and further improve the video stabilization effect of the video stabilization method.

[0009] In an implementation form of the first aspect, the intermediate image is obtained by aligning the previous frame image or the previous key frame image according to the pose change amount, comprising: obtaining a homography matrix according to the pose change amount; and projecting the previous frame image or the previous key frame image according to the homography matrix to obtain the intermediate image.

[0010] In the implementation process of the scheme, the intermediate image is obtained by projecting the previous frame image or the previous key frame image to the intermediate image by using the homography matrix, and the efficiency of obtaining the intermediate image is high. At the same time, the first optical flow used for compensating the pose change amount can be obtained through the optical flow between the intermediate image and the current frame image, so that the video stabilization method can realize more accurate inter-frame motion estimation, and further improve the video stabilization effect of the video stabilization method.

[0011] In an implementation form of the first aspect, the second optical flow is obtained by fusing the pose change amount and the first optical flow, comprising: obtaining a third optical flow used for representing the pose change amount according to the pose change amount; and fusing the first optical flow and the third optical flow to obtain the second optical flow.

[0012] In the implementation process of the scheme, the second optical flow capable of accurately representing the inter-frame motion information is obtained by fusing the third optical flow used for representing the inter-frame pose change amount and the first optical flow used for compensating the pose change amount, and then the path smoothing and image transformation processing of the current frame image are performed through the second optical flow, thereby improving the video stabilization effect of the video stabilization method.

[0013] In an implementation form of the first aspect, the second optical flow is obtained by fusing the first optical flow and the third optical flow, comprising: summing or weighted summing the first optical flow and the third optical flow to obtain the second optical flow.

[0014] In the implementation process of the scheme, the first optical flow and the third optical flow can be fused by using the direct summing method or the weighted method, so that the video stabilization method can be applied to more scenes, and the applicability of the video stabilization method is improved.

[0015] In one implementation of the first aspect, the step of smoothing the global motion trajectory of the imaging device to obtain a smooth trajectory includes: using a filtering method or a trajectory smoothing algorithm to smooth the global motion trajectory of the imaging device to obtain a smooth trajectory.

[0016] In the implementation of the above scheme, two trajectory smoothing methods are provided, which makes the above video stabilization method applicable to more application scenarios and improves the applicability of the above video stabilization method.

[0017] In one implementation of the first aspect, obtaining the first imaging device pose and the second imaging device pose corresponding to the current frame image and the previous frame image in the video to be stabilized includes: obtaining imaging device pose data corresponding to the video to be stabilized; synchronizing the video to be stabilized with the imaging device pose data to obtain the first imaging device pose corresponding to the current frame image and the second imaging device pose corresponding to the previous frame image or the previous keyframe image.

[0018] In the implementation of the above scheme, by synchronizing the video to be stabilized with the pose data of the imaging device, the accurate poses of the first and second imaging devices can be obtained, thereby obtaining the accurate pose change amount. This enables the above video stabilization method to achieve more accurate inter-frame motion estimation, further improving the video stabilization effect of the above video stabilization method.

[0019] Secondly, embodiments of this application provide a video stabilization device, comprising:

[0020] The imaging device pose acquisition module is used to acquire the first imaging device pose corresponding to the current frame image in the video to be stabilized, and the second imaging device pose corresponding to the previous frame image or the previous key frame image, and to determine the pose change amount.

[0021] An optical flow calculation module is used to obtain a first optical flow for compensating the pose change based on the pose change amount.

[0022] The fusion module is used to fuse the pose change and the first optical flow to obtain the second optical flow;

[0023] The motion trajectory acquisition module is used to acquire the global motion trajectory of the imaging device based on the second optical flow;

[0024] The trajectory smoothing module is used to smooth the global motion trajectory of the imaging device and obtain a smooth trajectory.

[0025] The image transformation module is used to perform image transformation on the current frame image according to the smoothing trajectory to obtain the current frame image after image stabilization processing.

[0026] Thirdly, embodiments of this application provide a computer-readable storage medium storing computer program instructions, which, when read and executed by a processor, perform the method provided in the first aspect or any possible implementation thereof.

[0027] Fourthly, embodiments of this application provide an electronic device, including: a memory and a processor, wherein the memory stores computer program instructions, and the computer program instructions are read and executed by the processor to perform the method provided in the first aspect or any possible implementation of the first aspect.

[0028] Other features and advantages of this application will be set forth in the following description and will be apparent in part from the description or may be learned by practicing embodiments of this application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0029] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 A flowchart illustrating the video stabilization method provided in this application embodiment;

[0031] Figure 2 This is a schematic diagram of the video stabilization device provided in the embodiments of this application;

[0032] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0033] The technical solutions of the embodiments of this application will now be described with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solutions of this application, and are therefore merely examples and should not be used to limit the scope of protection of this application.

[0034] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the application; the terms “comprising” and “having”, and any variations thereof, in the specification, claims, and foregoing description of the drawings are intended to cover non-exclusive inclusion.

[0035] In the description of the embodiments of this application, technical terms such as "first" and "second" are used only to distinguish different objects and should not be construed as indicating or implying relative importance or implicitly specifying the number, specific order, or primary and secondary relationship of the indicated technical features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly defined.

[0036] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0037] In the description of the embodiments in this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship.

[0038] Please see Figure 1 This application provides a video stabilization method, including:

[0039] Step S110: Obtain the first imaging device pose corresponding to the current frame image in the video to be stabilized, and the second imaging device pose corresponding to the previous frame image or the previous key frame image, and determine the pose change amount.

[0040] Step S120: Based on the pose change, obtain the first optical flow used to compensate for the pose change;

[0041] Step S130: Fuse the pose change and the first optical flow to obtain the second optical flow;

[0042] Step S140: Obtain the global motion trajectory of the imaging device based on the second optical flow;

[0043] Step S150: Smooth the global motion trajectory of the imaging device to obtain a smooth trajectory;

[0044] Step S160: Based on the smoothing trajectory, perform image transformation on the current frame image to obtain the current frame image after image stabilization.

[0045] In the implementation of the above scheme, the method of first determining the inter-frame pose change amount and then using compensating optical flow to compensate for the inter-frame pose change amount is adopted. This fully utilizes the characteristics of pose change amount representing large-amplitude jitter and compensating optical flow compensating for small-amplitude jitter, thereby improving the accuracy of motion estimation and thus improving the stabilization effect of the above video stabilization method.

[0046] Steps S110 to S160 are described in detail below:

[0047] The video stabilization method described above, comprising steps S110 to S160, can be summarized into three steps: motion estimation, path smoothing, and image transformation.

[0048] The motion estimation step, corresponding to steps S110 to S140 in the video stabilization method described above, refers to calculating the motion information between the imaging device pose corresponding to the current frame image in the video and the imaging device corresponding to the previous frame image or the previous key frame image.

[0049] The path smoothing step, corresponding to step S150 in the video stabilization method above, refers to using a smoothing algorithm to smooth the motion trajectory of the current frame after motion estimation.

[0050] The image transformation step, corresponding to step S160 in the video stabilization method described above, refers to transforming the image based on the difference between the smoothed path and the motion estimation path after path smoothing is completed, to obtain the video after image stabilization.

[0051] The following details step S110: The video to be stabilized in step S110 refers to the video acquired by the imaging device. Since the above video stabilization method is real-time stabilization, the video to be stabilized is essentially an image sequence composed of single-frame images. Step S110 is to process the currently acquired frame image.

[0052] As an optional implementation of the above video stabilization method, step S110, acquiring the first and second imaging device poses corresponding to the current frame and the previous frame in the video to be stabilized, respectively, includes: acquiring the imaging device pose data corresponding to the video to be stabilized; synchronizing the video to be stabilized with the imaging device pose data to acquire the first imaging device pose corresponding to the current frame and the second imaging device pose corresponding to the previous frame or the previous keyframe. For example, when using a pose measurement device such as an inertial measurement unit (IMU) to acquire the imaging device pose, the imaging device pose at the current frame cannot be directly acquired. Therefore, it is necessary to synchronize the video to be stabilized with the imaging device pose data to acquire the first imaging device pose corresponding to the current frame; the second imaging device pose corresponding to the previous frame or the previous keyframe can also be acquired through the synchronized pose data; after acquiring the first and second imaging device poses, the pose change between the first and second imaging device poses can be obtained.

[0053] It is understandable that since steps S110 to S160 process the current frame image, the previous frame image or the previous keyframe image may also have been processed by the above steps S110 to S160. Therefore, the pose of the second imaging device corresponding to the previous frame image or the previous keyframe image can also be queried in the historical data.

[0054] In addition, the above pose changes include both position information changes (Δx, Δy, Δz) and attitude information changes (Δi, Δj, Δk), where Δx, Δy, and Δz represent displacement changes along the x-axis, y-axis, and z-axis, respectively, and Δi, Δj, and Δk represent rotation changes along the x-axis, y-axis, and z-axis, respectively.

[0055] It should be noted that the previous frame image can be used directly for inter-frame motion estimation, or the previous keyframe image can be used for inter-frame motion estimation. The inter-frame motion estimation method can be set according to the actual situation.

[0056] The following details step S120: Step S120 uses the pose change amount to obtain the first optical flow that compensates for the pose change amount.

[0057] As an optional implementation of the above-mentioned video stabilization method, step S120 obtains a first optical flow for compensating for the pose change based on the pose change amount, including: aligning the previous frame image or the previous keyframe image according to the pose change amount to obtain an intermediate image; and obtaining the first optical flow between the intermediate image and the current frame image. For example, when using a pose measurement device such as an inertial measurement unit (IMU) to obtain the pose of the imaging device, there is a certain measurement error, and the pose change amount of the imaging device cannot be accurately obtained. Therefore, this embodiment uses a first optical flow to compensate for the pose change amount. The specific calculation method of the first optical flow is as follows: aligning the previous frame image or the previous keyframe image according to the pose change amount to obtain an intermediate image, and then obtaining the first optical flow between the intermediate image and the current frame image. This optical flow is the inter-frame motion compensation information used to compensate for the pose change amount.

[0058] It should be noted that the optical flow mentioned above refers to optical flow, which in physical terms is the instantaneous velocity of pixels moving on the imaging plane of a moving object in space. A proper explanation of optical flow is as follows: When a human eye observes the movement of an object, the image of the object forms a series of continuously changing images on the retina of the human eye. This series of continuously changing information flows through the retina (i.e., the image plane) like a flow of light, hence the name optical flow. Optical flow expresses the changing information of the image.

[0059] It is understandable that the aforementioned optical flow is the instantaneous velocity of pixel motion. This instantaneous velocity is a vector velocity, possessing not only magnitude but also direction. Therefore, optical flow can characterize the motion information of pixel motion. Pose change and optical flow are two different forms used to represent inter-frame motion information, and they can be converted into each other. Specific conversion methods can refer to existing technologies.

[0060] As an optional implementation of the above video stabilization method, step S120 aligns the previous frame image or the previous keyframe image according to the pose change amount to obtain an intermediate image, including: obtaining a homography matrix according to the pose change amount; and projecting the previous frame image or the previous keyframe image according to the homography matrix to obtain an intermediate image. Homography (transmission transformation) is a concept in projective geometry, also known as projective transformation. The homography matrix can fuse the intrinsic parameters of the imaging device and the relative pose of the imaging device during pose changes; this relative pose is the aforementioned pose change amount. Step S120 obtains the homography matrix M based on the pose change amount or by combining the pose change amount and the intrinsic parameters of the imaging device, and then projects the previous frame image or the previous keyframe image onto the intermediate image using the homography matrix M to obtain the intermediate image.

[0061] Step S130 is described in detail below: Step S130 obtains inter-frame motion information that accurately characterizes the current frame image and the previous frame image or the previous keyframe image by fusing the pose change amount and the first optical flow used to compensate for the pose change amount.

[0062] As an optional implementation of the above video stabilization method, step S130, fusing pose change and first optical flow to obtain second optical flow, includes: obtaining a third optical flow characterizing pose change based on pose change; and fusing the first and third optical flows to obtain the second optical flow. An example of this implementation is:

[0063] Based on the pose change, generate the motion matrix M1 of the imaging device:

[0064] M1 = F1(pose)

[0065] Where M1 is the motion matrix of the imaging device; pose is the pose change; F1 is the process of converting the pose change into a motion matrix, and the specific processing method can be found in the prior art.

[0066] By combining the imaging device intrinsic parameter matrix M2, the camera motion matrix M1 is mapped to the projection transformation matrix M3 in the pixel coordinate system:

[0067] M3 = F2(M1, M2)

[0068] Wherein, M2 is the projection transformation matrix in the pixel coordinate system; M2 is the intrinsic parameter matrix of the imaging device; F2 is the process of mapping the camera motion matrix M1 to the projection transformation matrix M3 in the pixel coordinate system by combining the intrinsic parameter matrix M2 of the imaging device. For the specific processing process, please refer to the prior art.

[0069] The third optical flow used to characterize the pose change is obtained based on the projection transformation matrix M3:

[0070] flow3 = F3(M3)

[0071] Wherein, F3 represents the process of converting the projection transformation matrix M3 into the third optical flow flow3, and the specific process can be found in the prior art;

[0072] The first optical flow (flow1) and the third optical flow (flow3) are fused to obtain the second optical flow (flow2):

[0073] flow2 = F4(flow1, flow3)

[0074] F4 represents the fusion process of fusing the first optical flow flow1 and the third optical flow flow3.

[0075] As an optional implementation of the above video stabilization method, step S130, fusing the first optical flow and the third optical flow to obtain the second optical flow, includes: summing or weighted summing the first optical flow and the third optical flow to obtain the second optical flow. These two implementation methods are described in detail below:

[0076] In the first implementation, the first optical flow and the third optical flow are directly summed to obtain the second optical flow:

[0077] flow2 = flow1 + flow3

[0078] This implementation method has high operating efficiency and is suitable for scenarios that require rapid image stabilization of video.

[0079] In the second implementation, the first optical flow and the third optical flow are weighted and summed to obtain the second optical flow:

[0080] flow2 = δ1flow1 + δ3flow3

[0081] Where δ1 and δ3 are the weights of the first optical flow and the third optical flow, respectively;

[0082] It is understandable that during the calculation of inter-frame motion information, the pose information measured by the inertial measurement unit (IMU) and the compensated optical flow information calculated by the optical flow algorithm have different confidence levels. Therefore, weights can be set for the first and third optical flows according to the data confidence level and the actual application, so that the second optical flow obtained by fusion can more accurately represent the inter-frame motion information.

[0083] Step S140 is described in detail below: Step S140 obtains the global motion trajectory of the imaging device based on the fused second optical flow. It should be noted that when obtaining the global motion trajectory of the imaging device based on the second optical flow, the epipolar geometry method, the PnP (Perspective-n-Point) method, or the ICP (Iterative Closest Point) method can be used. For the specific processing flow of the epipolar geometry method, the PnP method, and the ICP method, please refer to the prior art.

[0084] In addition, the pose change and optical flow mentioned above are both vectors. The first and third optical flows obtained are also vectors. When fusing the first and third optical flows, it is necessary to ensure that the calculation direction of the optical flows is consistent. For example, if the first optical flow represents the optical flow from the intermediate image to the current frame image, then the third optical flow should represent the optical flow from the previous frame image or the previous keyframe image to the intermediate image; if the first optical flow represents the optical flow from the current frame image to the intermediate image, then the third optical flow should represent the optical flow from the intermediate image to the previous frame image or the previous keyframe image.

[0085] The following is a detailed description of step S150: Step S150 performs global trajectory smoothing on the imaging device to obtain a smooth trajectory.

[0086] As an optional implementation of the above-mentioned video stabilization method, step S150 smooths the global motion trajectory of the imaging device to obtain a smoothed trajectory, including: smoothing the global motion trajectory of the imaging device using a filtering method or a trajectory smoothing algorithm to obtain a smoothed trajectory. For example, when smoothing the global motion trajectory of the imaging device, a filtering method can be used to filter the changes in the motion parameters of the imaging device, or a trajectory smoothing algorithm can be used to directly smooth the global motion trajectory of the imaging device. Examples of filtering methods include Gaussian filtering and Kalman filtering, while examples of trajectory smoothing algorithms include moving average smoothing and spline curve smoothing.

[0087] The following details step S160: Step S160 performs image transformation on the current frame image based on the smooth trajectory. The purpose is to transform the current frame image from the global motion trajectory of the imaging device obtained in step S140 to the smooth trajectory obtained in step S150. The specific transformation method is as follows: the trajectory offset can be determined by the global motion trajectory of the imaging device and the smooth trajectory, and then the current frame image is transformed to the smooth trajectory based on the trajectory offset to obtain a stable current frame image, thereby achieving anti-shake smoothing processing of the video.

[0088] It should be noted that the methods for obtaining the first and second optical flows mentioned above can be achieved using optical flow estimation algorithms such as the LK (Lucas–Kanade) optical flow method and deep learning optical flow method. The first and second optical flows can be obtained using either sparse or dense optical flow; the specific method chosen depends on the actual application.

[0089] Based on the same inventive concept, please refer to Figure 2 This application embodiment also provides a video stabilization device 200, including:

[0090] The imaging device pose acquisition module 210 is used to acquire the first imaging device pose corresponding to the current frame image in the video to be stabilized, and the second imaging device pose corresponding to the previous frame image or the previous key frame image, and to determine the pose change amount.

[0091] The optical flow calculation module 220 is used to obtain a first optical flow for compensating the pose change based on the pose change amount.

[0092] Fusion module 230 is used to fuse the pose change and the first optical flow to obtain the second optical flow;

[0093] The motion trajectory acquisition module 240 is used to acquire the global motion trajectory of the imaging device based on the second optical flow;

[0094] The trajectory smoothing module 250 is used to smooth the global motion trajectory of the imaging device and obtain a smooth trajectory.

[0095] The image transformation module 260 is used to perform image transformation on the current frame image according to the smoothing trajectory to obtain the current frame image after image stabilization processing.

[0096] As an optional implementation of the above-mentioned video stabilization device, the optical flow calculation module 220 obtains a first optical flow for compensating the pose change based on the pose change amount, including: aligning the previous frame image or the previous keyframe image based on the pose change amount to obtain an intermediate image; and obtaining the first optical flow between the intermediate image and the current frame image.

[0097] As an optional implementation of the aforementioned video stabilization device, the optical flow calculation module 220 performs alignment processing on the previous frame image or the previous keyframe image based on the pose change amount to obtain an intermediate image, including: obtaining a homography matrix based on the pose change amount; and projecting the previous frame image or the previous keyframe image based on the homography matrix to obtain an intermediate image.

[0098] As an optional implementation of the above-mentioned video stabilization device, the fusion module 230 fuses the pose change amount and the first optical flow to obtain a second optical flow, including: obtaining a third optical flow to characterize the pose change amount based on the pose change amount; and fusing the first optical flow and the third optical flow to obtain the second optical flow.

[0099] As an optional implementation of the above-mentioned video stabilization device, the fusion module 230 fuses the first optical flow and the third optical flow to obtain the second optical flow, including: summing or weighted summing the first optical flow and the third optical flow to obtain the second optical flow.

[0100] As an optional implementation of the above-mentioned video stabilization device, the trajectory smoothing module 250 smooths the global motion trajectory of the imaging device to obtain a smooth trajectory, including: using a filtering method or a trajectory smoothing algorithm to smooth the global motion trajectory of the imaging device to obtain a smooth trajectory.

[0101] As an optional implementation of the above-mentioned video stabilization device, the imaging device pose acquisition module 210 acquires the first imaging device pose and the second imaging device pose corresponding to the current frame image and the previous frame image in the video to be stabilized, respectively, including: acquiring the imaging device pose data corresponding to the video to be stabilized; synchronizing the video to be stabilized with the imaging device pose data to acquire the first imaging device pose corresponding to the current frame image, and the second imaging device pose corresponding to the previous frame image or the previous keyframe image.

[0102] Figure 3 This is a schematic diagram of an electronic device provided in an embodiment of this application. (Refer to...) Figure 3 The electronic device 300 includes a processor 310, a memory 320, and a communication interface 330. These components are interconnected and communicate with each other via a communication bus 340 and / or other forms of connection mechanism (not shown).

[0103] The memory 320 includes one or more (only one is shown in the figure), which may be, but is not limited to, Random Access Memory (RAM), Read Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), etc. The processor 310 and other possible components may access the memory 320 to read and / or write data therein.

[0104] Processor 310 includes one or more (only one is shown in the figure), which can be an integrated circuit chip with signal processing capabilities. The processor 310 described above can be a general-purpose processor, including a central processing unit (CPU), a microcontroller unit (MCU), a network processor (NP), or other conventional processors; it can also be a special-purpose processor, including a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0105] Communication interface 330 includes one or more (only one is shown in the figure) and can be used to communicate directly or indirectly with other devices to exchange data. For example, communication interface 330 can be an Ethernet interface; it can be a mobile communication network interface, such as an interface for 3G, 4G, or 5G networks; or it can be other types of interfaces with data transmission and reception functions.

[0106] One or more computer program instructions may be stored in the memory 320, and the processor 310 may read and run these computer program instructions to implement the video stabilization method provided in the embodiments of this application and other desired functions.

[0107] Understandable. Figure 3 The structure shown is for illustrative purposes only; the electronic device 300 may also include components that are more advanced than those shown. Figure 3 The more or fewer components shown, or having the same Figure 3 The different configurations shown. Figure 3 The components shown can be implemented using hardware, software, or a combination thereof. For example, electronic device 300 can be a single server (or other device with computing power), a combination of multiple servers, a cluster of a large number of servers, etc., and can be either a physical device or a virtual device.

[0108] This application also provides a computer-readable storage medium storing computer program instructions. These computer program instructions are read and executed by a computer's processor to perform the video stabilization method provided in this application. For example, the computer-readable storage medium can be implemented as follows: Figure 3 The memory 320 in the electronic device 300.

[0109] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0110] Furthermore, the units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0111] Furthermore, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0112] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A video stabilization method, characterized by, The method comprises the following steps: obtaining a first imaging device pose corresponding to a current frame image in a video to be stabilized and a second imaging device pose corresponding to a previous frame image or a previous key frame image, and determining a pose change amount; obtaining a first optical flow for compensating the pose change amount according to the pose change amount; fusing the pose change amount and the first optical flow to obtain a second optical flow; obtaining an imaging device global motion trajectory according to the second optical flow; performing smoothing processing on the imaging device global motion trajectory to obtain a smoothed trajectory; performing image transformation on the current frame image according to the smoothed trajectory to obtain a current frame image after stabilization processing; The method comprises the following steps: obtaining a first optical flow for compensating the pose change amount according to the pose change amount, which comprises the following steps:

2. The video stabilization method of claim 1, wherein, aligning the previous frame image or the previous key frame image according to the pose change amount to obtain an intermediate image; and obtaining a first optical flow between the intermediate image and the current frame image. The method comprises the following steps:

3. The video stabilization method of claim 1, wherein, obtaining imaging device pose data corresponding to the video to be stabilized; and synchronizing the video to be stabilized and the imaging device pose data to obtain the first imaging device pose corresponding to the current frame image and the second imaging device pose corresponding to the previous frame image or the previous key frame image. The method comprises the following steps:

4. The video stabilization method of claim 3, wherein, obtaining a homography matrix according to the pose change amount; projecting the previous frame image or the previous key frame image according to the homography matrix to obtain the intermediate image.

5. The video stabilization method of claim 1, wherein, The method comprises the following steps: obtaining a third optical flow for representing the pose change amount according to the pose change amount; 6. A video stabilizing apparatus characterized by comprising: fusing the first optical flow and the third optical flow to obtain the second optical flow. The method comprises the following steps: performing summation or weighted summation on the first optical flow and the third optical flow to obtain the second optical flow. The method comprises the following steps: performing smoothing processing on the imaging device global motion trajectory by using a filtering method or a trajectory smoothing algorithm to obtain the smoothed trajectory. The method comprises the following steps: an imaging device pose obtaining module, configured to obtain a first imaging device pose corresponding to a current frame image in a video to be stabilized and a second imaging device pose corresponding to a previous frame image or a previous key frame image, and determine a pose change amount; an optical flow calculating module, configured to obtain a first optical flow for compensating the pose change amount according to the pose change amount; a fusing module, configured to fuse the pose change amount and the first optical flow to obtain a second optical flow; a motion trajectory obtaining module, configured to obtain an imaging device global motion trajectory according to the second optical flow; a trajectory smoothing module, configured to perform smoothing processing on the imaging device global motion trajectory to obtain a smoothed trajectory; and a stabilizing module, configured to perform image transformation on the current frame image according to the smoothed trajectory to obtain a current frame image after stabilization processing. An image transformation module is configured to perform image transformation on the current frame image according to the smooth trajectory to obtain a current frame image after image stabilization processing. The optical flow calculation module obtains a first optical flow for compensating the pose change amount according to the pose change amount, including: performing alignment processing on the previous frame image or the previous key frame image according to the pose change amount to obtain an intermediate image; and obtaining a first optical flow between the intermediate image and the current frame image. The imaging device pose obtaining module obtains a first imaging device pose and a second imaging device pose corresponding to the current frame image and the previous frame image in the video to be stabilized, including: obtaining imaging device pose data corresponding to the video to be stabilized; and synchronizing the video to be stabilized with the imaging device pose data to obtain the first imaging device pose corresponding to the current frame image and the second imaging device pose corresponding to the previous frame image or the previous key frame image.

7. An electronic device, comprising: comprising: a processor, a memory and a bus, wherein the processor and the memory complete mutual communication through the bus; The memory stores program instructions executable by the processor, and the processor calling the program instructions can execute the method of any one of claims 1-5.

8. A non-transitory computer-readable storage medium, comprising: The non-transitory computer readable storage medium stores computer instructions, and the computer instructions make the computer execute the method of any one of claims 1-5 when the computer runs.

Images