Video stabilization display methods and electronic devices

Abstract

A video stabilization display method and electronic device, relating to the field of terminal technology, can improve video stabilization efficiency, retain more video frame content, and obtain multiple stabilized videos. The method includes: acquiring a portion of video frame packets through each of multiple asynchronous threads; processing the portion of video frame packets to obtain jitter paths of reference pixels between video frames within the video frame packets; determining the video jitter path based on the jitter paths of reference pixels in the portion of video frames processed by each asynchronous thread; smoothing jitter in the jitter path where the jitter amplitude is less than a first threshold and the jitter frequency is greater than a second threshold to determine a first smoothing path; determining at least one second smoothing path based on at least two first smoothing paths combined with an interpolation algorithm; determining a reconstruction transformation matrix group corresponding to the candidate smoothing paths through a second asynchronous thread; and determining the stabilized video based on the reconstruction transformation matrix group.

Description

Technical Field

[0001] This application relates to the field of terminal technology, and in particular to a video stabilization display method and electronic device. Background Technology

[0002] Before playing a video, electronic devices can use video stabilization (shake removal) technology to perform post-processing calculations on the video, automatically removing the additional shaking introduced during video shooting, thereby stabilizing the video content and improving video quality.

[0003] In related technologies, electronic devices perform video decluttering frame-by-frame based on a sequential decoding framework. This method is limited by the input speed of video frames, resulting in low decluttering efficiency. Alternatively, decluttering can be performed using pre-trained convolutional neural networks, which involves high computational load, high latency, and low efficiency. Furthermore, users have limited choices regarding the decluttering results, making it difficult to meet the needs of different scenarios. Summary of the Invention

[0004] This application provides a video stabilization display method and electronic device, which can improve video stabilization efficiency, retain more video image content, and obtain multiple stabilized videos.

[0005] To achieve the above objectives, the embodiments of this application provide the following technical solutions:

[0006] In a first aspect, this application provides a video stabilization display method, comprising: acquiring a portion of video frame packets in a video through each of a plurality of first asynchronous threads, and processing the portion of video frame packets to obtain a jitter path of a reference pixel between video frames in the video frame packets; the video frame packets processed by different first asynchronous threads are different; the video frame packets include at least two adjacent video frames; determining a jitter path of the video based on the jitter path of the reference pixel in the portion of video frames processed by each first asynchronous thread; smoothing jitter in the jitter path where the jitter amplitude is less than a first threshold and the jitter frequency is greater than a second threshold to determine a first smoothing path; determining at least one second smoothing path based on at least two first smoothing paths and an interpolation algorithm; determining a reconstruction transformation matrix group corresponding to a candidate smoothing path through a second asynchronous thread; the candidate smoothing path includes at least two first smoothing paths and at least one second smoothing path; the reconstruction transformation matrix group includes at least one reconstruction transformation matrix corresponding to a video frame packet; the reconstruction transformation matrix is ​​obtained by performing image transformation on the video frames in the video frame packets; the reconstruction transformation matrix represents the correspondence between the corresponding pixels in the video frames and the image-transformed video frames; and determining a stabilized video by stabilizing the video through the reconstruction transformation matrix group.

[0007] Thus, electronic devices employ multi-threaded asynchronous parallel processing to dedop video frames, improving dedop efficiency and reducing latency and congestion. Shaking amplitudes within the shaking path that are less than a first threshold and have a frequency greater than a second threshold are smoothed. Shaking amplitudes that are too large or too small are not processed, reducing the cropped area and preserving more video image content. Multiple smoothing paths are generated using interpolation algorithms, resulting in multiple dedop videos to meet the diverse dedop results needs of different users in different scenarios.

[0008] For example, such as Figure 2 As shown, the mobile phone decodes the video frame by frame, obtaining a video frame queue. Two consecutive video frames in the video frame queue are packaged into multiple image pairs (video frame packets), and these image pairs form a relay queue. The mobile phone can use multiple first asynchronous threads to perform asynchronous parallel feature matching and asynchronous parallel path estimation on the multiple image pairs to obtain jitter paths. Then, a mixed-order adaptive path smoothing (MAPS) algorithm is used to smooth jitter in the jitter path where the jitter amplitude is less than a first threshold and the jitter frequency is greater than a second threshold, obtaining smoothed paths (first smoothed paths). Based on at least two first smoothed paths, at least one second smoothed path is determined using an interpolation algorithm. Then, for each image pair in the first and second smoothed paths (candidate smoothed paths), multiple second asynchronous threads are used to perform asynchronous parallel motion compensation and asynchronous parallel image transformation, obtaining unordered reconstruction transformation matrices corresponding to the image pairs. The mobile phone can then perform streaming sorting on multiple unordered reconstruction transformation matrices to obtain ordered reconstruction transformation matrix groups. Afterwards, the mobile phone can render and reconstruct the jittery video frames according to the ordered reconstruction transformation matrix group to obtain the jitter-free video frames. The jitter-free video frames are then encoded to obtain the final jitter-free stable video (jitter-free video).

[0009] In one possible implementation, each of multiple asynchronous threads acquires a portion of video frame packets from the video and processes these packets to obtain the jitter path of a reference pixel within each video frame. This includes: each asynchronous thread determining the position of the reference pixel on each video frame within at least one video frame packet; the jitter path is obtained by curve fitting to the position of the reference pixel determined by each asynchronous thread. This allows for the rapid acquisition of the video's jitter path.

[0010] For example, such as Figure 7A and Figure 7BAs shown, the coordinates of the four vertices (reference pixels) of the video frame are determined. The jitter paths of the four vertices of the video frame are calculated based on the global homography transformation matrix gloH.

[0011] In one possible implementation, jitter in the jitter path where the jitter amplitude is less than a first threshold and the jitter frequency is greater than a second threshold is smoothed to determine a first smoothing path. This includes: determining smoothing parameters, which at least include a maximum amplitude, a minimum amplitude, and a preset sliding window; and smoothing jitter in the jitter path where the jitter amplitude is less than the first threshold and the jitter frequency is greater than the second threshold based on the smoothing parameters, the jitter path, and the number of video frames to determine the first smoothing path. This removes jitter with "small amplitude" and "high frequency" values ​​from the jitter path. It does not remove jitter with larger amplitude or lower frequency values, thus avoiding excessive déjà vu in video frames, which could lead to excessively large black borders. It also reduces the degree of video frame cropping, preserving more edge information of the video frames.

[0012] For example, "small amplitude" jitter refers to jitter with a small amplitude within the jitter path. For instance, jitter where the global motion vector offset in a video frame is less than 5% of the shorter side of the image. For example, a video frame with a horizontal length of 100 pixels has a jitter distance of less than 5 pixels between its left and right sides. "High frequency" jitter refers to jitter with a high frequency within the jitter path. For example, jitter where the global vector direction changes more than 3 times within 1 second. For example, a video frame jitters more than 3 times from left to right and more than 3 times from right to left within 1 second.

[0013] In one possible implementation, the method further includes: determining a first sliding window based on the maximum amplitude and the minimum amplitude, the first sliding window being used to remove jitter in the jitter path whose jitter amplitude is less than a first threshold; determining a second sliding window based on the maximum amplitude, the minimum amplitude, and a preset sliding window, the second sliding window being used to remove jitter in the jitter path whose jitter frequency is greater than a second threshold; determining a third sliding window based on the first sliding window and the second sliding window, the third sliding window being used to remove jitter in the jitter path whose jitter amplitude is less than the first threshold and whose jitter frequency is greater than the second threshold; and determining a first smoothed path by smoothing the jitter path based on the third sliding window and the jitter path, combined with Gaussian filtering.

[0014] For example, a zero-order penalty term (zero-order velocity modeling) is introduced to determine the sliding window M_spd (first sliding window) used to remove "small amplitude" jitter in the jitter path, and a first-order penalty term (first-order acceleration modeling) is introduced to determine the sliding window M_acc (second sliding window) used to remove "high frequency" jitter in the jitter path. Then, a mixed-order model is constructed, and the size of the Gaussian filter sliding window M (third sliding window) used to remove "small amplitude" and "high frequency" jitter is determined based on M_spd and M_acc. Finally, the mobile phone can perform Gaussian filtering based on the jitter path S and sliding window M input at each time step (each frame) to determine the smooth path S' at each time step (each frame).

[0015] In one possible implementation, determining at least one second smoothing path based on at least two first smoothing paths using an interpolation algorithm includes: determining at least one second smoothing path based on at least two first smoothing paths using a linear interpolation algorithm. This allows for the rapid acquisition of multiple smoothing paths, resulting in multiple stabilized videos and meeting the diverse stabilization needs of different users in different scenarios.

[0016] For example, such as Figure 9 As shown, an asynchronous parallel approach is used to interpolate the smooth path marked as 1 and the smooth path marked as 10. That is, multiple threads simultaneously perform interpolation on the smooth path marked as 1 and the smooth path marked as 10, which can quickly obtain n smooth paths, where n is an integer greater than or equal to 3.

[0017] In one possible implementation, multiple second asynchronous threads are used to determine the reconstruction transformation matrix group corresponding to a candidate smoothing path. This includes: for a candidate smoothing path, determining the reconstruction transformation matrix of each video frame corresponding to the candidate smoothing path using multiple second asynchronous threads; and sorting the reconstruction transformation matrices of each video frame using a stream sorting algorithm to determine the reconstruction transformation matrix group corresponding to a candidate smoothing path. In this way, multiple reconstruction transformation matrices are obtained in an asynchronous and parallel manner, improving the efficiency of video stabilization.

[0018] The multiple reconstruction transformation matrices obtained by asynchronous parallelism are unordered reconstruction transformation matrices. A stream sorting algorithm is needed to sort the reconstruction transformation matrices of each video frame, and the sorted multiple reconstruction transformation matrices are determined as a reconstruction transformation matrix group.

[0019] In one possible implementation, the method further includes storing a set of reconstructed transformation matrices. In this way, the mobile phone can perform shading processing on the video based on the stored set of reconstructed transformation matrices, which is memory-efficient and highly effective.

[0020] For example, a mobile phone can decode a video to obtain multiple video frames, and then render and reconstruct these multiple video frames using the corresponding reconstruction transformation matrices in the reconstruction transformation matrix group to obtain decluttered video frames. The mobile phone can then encode these decluttered video frames to obtain a decluttered video. This is displayed as shown in the example. Figure 4 In the video window 408 shown in (d).

[0021] In one possible implementation, the method further includes: displaying a first interface, the first interface including a stabilization control; receiving a first operation from a user on the stabilization control, the first operation being used to input a first stabilization level; and stabilizing the video by reconstructing a transformation matrix group, including: in response to receiving the first operation, stabilizing the video according to the reconstruction transformation matrix group corresponding to the first stabilization level; the reconstruction transformation matrix group corresponding to the first stabilization level represents the reconstruction transformation matrix group corresponding to the candidate smoothing path corresponding to the first stabilization level; wherein, the stabilization level and the candidate smoothing path correspond one-to-one. In this way, video stabilization can be performed quickly, resulting in multiple stabilized videos, improving the user experience.

[0022] For example, a user can slide (or tap) the intensity bar to determine the de-shake level. For example, such as... Figure 5A As shown in (b), the user can slide the intensity bar to stunting level 3 (first operation). The phone responds to the user's action of sliding the intensity bar to stunting level 3 by performing stunting on the video. Stunting levels 1-10 correspond to smoothing paths 1-10.

[0023] In another embodiment, such as Figure 5B As shown in (a), the user can click the de-shake control 413 (first operation) on the toolbar 411. In response to the user clicking the de-shake control 413, the mobile phone can first perform de-shake processing on the video.

[0024] Optionally, the above is only an example of the first operation. The first operation can also be other operations, and the embodiments of this application do not limit this.

[0025] In a second aspect, this application provides an electronic device comprising: a processor and a memory coupled to the processor, the memory being used to store computer program code, the computer program code including computer instructions, wherein when the processor reads the computer instructions from the memory, the electronic device causes the electronic device to perform the method described in the first aspect.

[0026] Thirdly, this application provides a computer-readable storage medium storing instructions that, when executed on an electronic device, cause the electronic device to perform the method described in the first aspect.

[0027] Fourthly, this application provides a chip system including at least one processor and at least one interface circuit, wherein the at least one interface circuit is used to perform transceiver functions and send instructions to the at least one processor, and when the at least one processor executes the instructions, the at least one processor performs the method as described in any one of the first aspects above.

[0028] Fifthly, this application also provides a computer program product, including instructions that, when run on a computer, cause the computer to perform the method as described in any one of the first aspects.

[0029] In a sixth aspect, embodiments of this application provide a circuit system including a processing circuit configured to perform the method of the first aspect or any of the embodiments of the first aspect.

[0030] The technical effects corresponding to aspects two through six, and any one of their implementations, can be found in the first aspect and any one of its implementations, and will not be repeated here. Attached Figure Description

[0031] Figure 1 A flowchart of a video stabilization display method provided in this application embodiment;

[0032] Figure 2 Flowchart of another video stabilization display method provided in this application embodiment;

[0033] Figure 3A This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0034] Figure 3B This is a schematic diagram of the structure of another electronic device provided in an embodiment of this application;

[0035] Figure 3C A schematic diagram of a module of an electronic device provided in an embodiment of this application;

[0036] Figure 4 A schematic diagram of the interface of an electronic device provided in an embodiment of this application;

[0037] Figure 5A A schematic diagram of the interface of another electronic device provided in an embodiment of this application;

[0038] Figure 5B A schematic diagram of the interface of another electronic device provided in an embodiment of this application;

[0039] Figure 6 A schematic diagram of the interface of another electronic device provided in an embodiment of this application;

[0040] Figure 7A A schematic diagram comparing interfaces provided for embodiments of this application;

[0041] Figure 7B Another interface comparison diagram provided for embodiments of this application;

[0042] Figure 8 A schematic diagram of a jitter curve provided for an embodiment of this application;

[0043] Figure 9 A flowchart of an interpolation method provided in an embodiment of this application;

[0044] Figure 10 This is another schematic diagram of a jitter curve provided in an embodiment of this application;

[0045] Figure 11 A schematic diagram of the interface of another electronic device provided in an embodiment of this application;

[0046] Figure 12 Flowchart of another video stabilization display method provided in this application embodiment;

[0047] Figure 13 Flowchart of another video stabilization display method provided in this application embodiment;

[0048] Figure 14 Flowchart of another video stabilization display method provided in this application embodiment;

[0049] Figure 15 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0050] Figure 16 This is a schematic diagram of a chip system provided in an embodiment of this application. Detailed Implementation

[0051] The following description, in conjunction with the accompanying drawings, details an operating method and electronic device provided in an embodiment of this application.

[0052] The terms "comprising" and "having," and any variations thereof, used in the description of the embodiments of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the steps or units listed, but may optionally include other steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices.

[0053] It should be noted that in the embodiments of this application, the words "exemplary" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design scheme described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of the words "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0054] In the description of the embodiments in this application, unless otherwise stated, "multiple" means two or more. "And / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone.

[0055] When shooting video, unstable shooting equipment, unstable lighting, or unstable shooting environment (such as shooting in windy or rainy weather) can all cause shaky video footage. Before playing the video on an electronic device, video stabilization (anti-shake) technology can be used to reduce the shakiness of the shot, allowing the electronic device to play stable video content.

[0056] In related technologies, electronic devices use a frame-by-frame de-jittering framework based on sequential decoding, such as... Figure 1 As shown, the electronic device performs feature matching, path estimation, jitter path calculation, path smoothing, motion compensation, and image transformation on each video frame, finally outputting a stabilized frame after jitter reduction. The entire process is performed synchronously; if one step is slow, it may block the entire process, reducing the efficiency of jitter reduction.

[0057] Alternatively, the electronic device performs corner detection on the current frame of video image to be displayed and the previous frame of video image that has already been displayed, calculates the affine transformation matrix of the corresponding corner points, and determines the video frame after stabilization based on the affine transformation matrix. This achieves video stabilization display.

[0058] However, this method is limited by the speed of video frame input, resulting in low shaking efficiency. Furthermore, the stability of the current frame is based on the result of the previous frame. If there are errors after shaking the previous frame, then there will also be errors after shaking the current frame, resulting in an error accumulation effect and a decrease in the overall stability of the shaking-free video.

[0059] Alternatively, electronic devices can extract key feature points from video frames, input these key feature points into a video stabilization model, and output stabilization displacement data corresponding to each key feature point. Based on the stabilization displacement data, the video frames are re-rendered to generate stabilized video frames.

[0060] This method requires training a video stabilization model using deep learning technology, resulting in high computational load, high latency, and demanding hardware requirements for electronic devices. It also requires a large amount of data to train the neural network, leading to high costs and long training cycles.

[0061] To address the aforementioned problems, embodiments of this application provide a video stabilization display method. In this method, an electronic device can utilize a streaming asynchronous stabilization framework to perform stabilization processing on the video. Specifically, as shown... Figure 2 As shown, the electronic device decodes the video frame by frame, obtaining a video frame queue. Two consecutive video frames in the queue are packaged into multiple image pairs (video frame packets), which form a relay queue. The phone can use multiple first asynchronous threads to perform asynchronous parallel feature matching and asynchronous parallel path estimation on the multiple image pairs to obtain jitter paths. The jitter paths are then smoothed to obtain smoothed paths. Subsequently, multiple second asynchronous threads are used to perform asynchronous parallel motion compensation and asynchronous parallel image transformation on the image pairs in the smoothed paths, obtaining unordered reconstruction transformation matrices for the corresponding image pairs. The phone can then perform streaming sorting on the unordered reconstruction transformation matrices to obtain an ordered reconstruction transformation matrix group. Finally, the phone can render and reconstruct the jittered video frames based on the ordered reconstruction transformation matrix group to obtain de-jittered video frames. These de-jittered video frames are then encoded to obtain the final stable de-jittered video. The entire de-jittering process adopts a streaming architecture. After decoding the video, the electronic device immediately processes the decoded video frames, avoiding the need to read the entire video, effectively reducing memory overhead. Therefore, it can support the electronic device in processing videos of arbitrarily long durations without being limited by memory size. Multiple processes are processed asynchronously in parallel, allowing for the simultaneous processing of multiple video frames, resulting in high efficiency.

[0062] The electronic device can utilize the mixed-order adaptive path amoothing (MAPS) method to smooth the jittery path of the video, obtaining a smooth path (stable path). Alternatively, the electronic device can use a stepless stabilization method to interpolate and generate multiple smooth paths from two smooth paths, thus obtaining multiple stable videos. The stepless stabilization method involves processing two smooth paths using interpolation to obtain multiple smooth paths.

[0063] The technical solution of this application adopts a streaming stabilization framework to perform stabilization processing on video, which improves the stabilization efficiency of electronic devices and reduces latency and congestion. The MAPS method is used to smooth the jitter path, reducing the cropped area and retaining more image content. A stepless stabilization method is used to generate multiple smooth paths, thereby obtaining multiple stabilized videos to meet the diverse stabilization results required by different users in different scenarios.

[0064] In one possible implementation, the embodiments of this application can be applied to an electronic device that can implement the above-described video stabilization display method. This electronic device can be a personal computer (PC), mobile phone, tablet computer, laptop computer, desktop computer, computer with transceiver capabilities, virtual reality (VR) terminal device, augmented reality (AR) terminal device, wireless terminal in industrial control, wireless terminal in self-driving, wireless terminal in remote medical care, wireless terminal in smart grid, wireless terminal in transportation safety, wireless terminal in smart city, wireless terminal in smart home, wearable device, in-vehicle equipment, and other terminal devices. The embodiments of this application do not impose any special limitations on the specific form of this electronic device.

[0065] Figure 3A The diagram shows a hardware structure of an electronic device provided in an embodiment of this application. This electronic device can be the electronic device described above. The electronic device includes at least one processor 201, a communication line 202, a memory 203, and at least one communication interface 204. The memory 203 may also be included within the processor 201.

[0066] Figure 3A The diagram shows a hardware structure of an electronic device provided in an embodiment of this application. This electronic device can be the first electronic device and / or the second electronic device described above. The electronic device includes at least one processor 201, a communication line 202, a memory 203, and at least one communication interface 204. The memory 203 may also be included within the processor 201.

[0067] The processor 201 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0068] Communication line 202 may include a path for transmitting information between the aforementioned components.

[0069] Communication interface 204 is used for communication with other devices. In this embodiment, the communication interface can be a module, circuit, bus, interface, transceiver, or other device capable of communication functions, used for communication with other devices. Optionally, when the communication interface is a transceiver, the transceiver can be a separately configured transmitter used to send information to other devices, or it can be a separately configured receiver used to receive information from other devices. The transceiver can also be a component that integrates sending and receiving information functions; this embodiment does not limit the specific implementation of the transceiver.

[0070] Memory 203 can be volatile memory or non-volatile memory, or may include both. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM), or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures, accessible by a computer, but not limited to these. The memory may exist independently and be connected to the processor 201 via communication line 202. The memory 203 may also be integrated with the processor 201.

[0071] The memory 203 stores computer execution instructions for implementing the scheme of this application, and its execution is controlled by the processor 201. The processor 201 executes the computer execution instructions stored in the memory 203, thereby implementing the carrier transmission method provided in the following embodiments of this application.

[0072] Optionally, the computer execution instructions in the embodiments of this application may also be referred to as application code, instructions, computer program or other names, and the embodiments of this application do not specifically limit them.

[0073] In a specific implementation, as one embodiment, the processor 201 may include one or more CPUs, for example... Figure 3A CPU0 and CPU1 in the CPU.

[0074] In a specific implementation, as one example, an electronic device may include multiple processors, for example... Figure 3A Processors 201 and 205 are described herein. Each of these processors may be a single-core (single-CPU) processor or a multi-core (multi-CPU) processor. A processor here may refer to one or more devices, circuits, and / or processing cores used to process data (e.g., computer program instructions).

[0075] The aforementioned electronic device can be a general-purpose device or a special-purpose device. The embodiments of this application do not limit the type of electronic device.

[0076] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the first electronic device 100 and the second electronic device 200. In other embodiments of this application, the first electronic device 100 and the second electronic device 200 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0077] In this application's embodiments, a mobile phone is used as an example of an electronic device for description, but this does not constitute a limitation on the form, function, or other aspects of the electronic device. (See references) Figure 3B This is a schematic diagram of a mobile phone structure provided in an embodiment of this application. The methods in the following embodiments can be implemented in a mobile phone having the above-described hardware structure.

[0078] like Figure 3B As shown, a mobile phone may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, antenna 1, antenna 2, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a camera 193, and a display screen 194, etc. Optionally, the mobile phone may also include a mobile communication module 150, etc.

[0079] It is understood that the structure illustrated in this embodiment does not constitute a specific limitation on the mobile phone. In other embodiments, the mobile phone may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0080] Processor 110 may include one or more processing units, such as: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, memory, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0081] The controller can be the nerve center and command center of a mobile phone. Based on the instruction opcode and timing signals, the controller generates operation control signals to control the fetching and execution of instructions.

[0082] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0083] In some embodiments, the processor 110 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a USB interface, etc.

[0084] The charging management module 140 receives charging input from the charger. While charging the battery 142, the charging management module 140 can also power the mobile phone via the power management module 141. The power management module 141 connects the battery 142, the charging management module 140, and the processor 110. The power management module 141 can also receive input from the battery 142 to power the mobile phone.

[0085] The wireless communication function of a mobile phone can be implemented through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor, and baseband processor.

[0086] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the mobile phone can be used to cover one or more communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with a tuning switch.

[0087] When a mobile phone includes a mobile communication module 150, the mobile communication module 150 can provide solutions for wireless communication applications such as 2G / 3G / 4G / 5G on the mobile phone. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation.

[0088] A modem processor may include a modulator and a demodulator. The modulator is used to modulate the low-frequency baseband signal to be transmitted into a medium- to high-frequency signal.

[0089] The wireless communication module 160 can provide solutions for wireless communication applications in mobile phones, including wireless local area networks (WLAN) (such as Wi-Fi), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), NFC, and infrared (IR) technologies. The wireless communication module 160 can be one or more devices integrating at least one communication processing module.

[0090] In some embodiments, the mobile phone's antenna 1 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, enabling the mobile phone to communicate with networks and other devices via wireless communication technology.

[0091] The mobile phone implements its display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0092] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Minied, MicroLED, Micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the mobile phone may include one or N displays 194, where N is a positive integer greater than 1. In some embodiments of this application, during the image stabilization process, the display screen 194 displays images such as... Figure 4 , Figure 5A , Figure 5B , Figure 6 The debouncing interface is shown below.

[0093] A mobile phone can perform shooting functions through an ISP, camera 193, video codec, GPU, display 194, and application processor. In some embodiments, the mobile phone may include one or N cameras 193, where N is a positive integer greater than 1. In this embodiment, the mobile phone can remove shake from shaky videos captured by the camera 193.

[0094] The external storage interface 120 can be used to connect an external memory card, thereby expanding the phone's storage capacity. The external memory card communicates with the processor 110 through the external storage interface 120 to perform data storage functions. For example, music, video, and other files can be saved on the external memory card.

[0095] The internal memory 121 can be used to store computer executable program code, which includes instructions. The processor 110 executes various mobile phone functions and data processing by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback, image playback, etc.), etc. The data storage area may store data created during mobile phone use (such as audio data, phonebook, etc.). Furthermore, the internal memory 121 may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc. In this embodiment, the internal memory 121 may store a reconstruction transformation matrix.

[0096] Mobile phones can perform audio functions, such as music playback and recording, through components like the audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and application processor.

[0097] Figure 3C The diagram shows a module schematic of an electronic device provided in an embodiment of this application. The electronic device includes an application module 301 and an algorithm module 302.

[0098] Application module 301 is used to acquire jittery video, decode the jittery video into jittery video frames, and pack two consecutive jittery video frames into image pairs. Application module 301 is also used to send the jittery video frames to algorithm module 302. Application module 301 is also used to receive multiple reconstruction transformation matrix groups from algorithm module 302 and store them on the local disk. The reconstruction transformation matrix H can represent the corresponding pixel transformation relationship between the transformed video frame and the original video frame. The reconstruction transformation matrix H can also represent the corresponding pixel transformation relationship between the shattered video frame and the original video frame. Application module 301 is also used to render and reconstruct the video frames sequentially according to the reconstruction transformation matrix group corresponding to the user-selected shatter level, to obtain the shattered video.

[0099] Algorithm module 302 is used to receive image pairs from the application module. Algorithm module 302 is also used to perform asynchronous feature matching and asynchronous path estimation on the image pairs to obtain jittered paths within the streaming asynchronous denoising framework. Algorithm module 302 is also used to smooth the jittered paths according to a mixed-order adaptive path smoothing optimization algorithm to obtain smoothed paths. Algorithm module 302 is also used to perform differential processing on the smoothed paths according to a stepless denoising method to obtain at least three smoothed paths. Algorithm module 302 is also used to perform asynchronous motion compensation and asynchronous image transformation processing on the smoothed paths to obtain multiple sets of reconstructed transformation matrices.

[0100] Figure 3C The illustrations are merely illustrative of the functional modules in an electronic device and the connections between them. It should be understood that an electronic device may include more or fewer modules than illustrated, or combine or separate certain modules, or have different module arrangements and connections. The illustrated modules may be implemented in hardware, software, or a combination of both.

[0101] Users typically edit videos before sharing or publishing them. This includes actions such as trimming, de-shaking, and adding effects to achieve a higher quality video.

[0102] Optionally, the "video" involved in the embodiments of this application can be any type or any length of video, such as a short video or a long video. The video can be referred to as a source video, source video file, video to be edited, video file to be edited, video file, or other similar names.

[0103] After receiving a user's input command for image stabilization, the mobile phone can use relevant algorithms to perform image stabilization processing on the video. The video stabilization method provided in this application can be embedded into multimedia applications (APPs) such as photo galleries and cameras, and also into applications such as video editing, without relying on underlying hardware devices. Cameras provide the ability to capture photos or videos, and photo galleries store various images and videos. Video editing applications can import videos. When editing videos, photo galleries can serve as a suitable entry point, cameras can serve as an entry point, and video editing applications can also serve as entry points.

[0104] This application embodiment uses the example of implementing a video editing entry point in a mobile phone video editing application to illustrate a scenario of video decluttering.

[0105] like Figure 4As shown in (a), this is a schematic diagram of the main interface 401 of a mobile phone provided in an embodiment of this application. The main interface 401 includes icons for clock, calendar, gallery, memo, video clip 402, settings, music, calculator, camera, contacts, phone, and messages. The user can click on the video clip 402 in the main interface 40. In response to the user's click on the video clip 402, the mobile phone displays the following... Figure 4 The interface 403 is shown in (b). Interface 403 includes an import window 404, a draft, etc. The user can click on the import window 404 of interface 403. In response to the user's click on the import window 404, the mobile phone displays as shown in (b). Figure 4 Interface 405 is shown in (c). Interface 405 includes multiple videos stored in the phone. Optionally, the user can select the video to be processed and click the import control 406. In response to the user clicking the import control 406, the phone displays as shown in (c). Figure 4 Interface 407 is shown in (d).

[0106] Interface 407 includes a video window 408, playback controls 409, a video progress bar area 410, a toolbar 411, and an export control 412. The toolbar 411 includes function options such as a split control, a speed control, a de-jitter control 413, and a delete control. The toolbar 411 may also include more or fewer other function options; this embodiment does not limit this.

[0107] When a user clicks the playback control 409 on interface 407, the phone responds to the user's click and plays the video in video window 408. The phone can also display the playback status of the video at the corresponding moment in the video progress bar area 410.

[0108] In one embodiment, such as Figure 5A As shown in (a), the user can click the de-shake control 413 in the toolbar 411. In response to the user clicking the de-shake control 413, the phone can display the following: Figure 5A The image stabilization frame 4130 is shown in (b). The image stabilization frame 4130 includes a stabilization intensity bar. The leftmost end of the intensity bar represents stabilization level 1, with the lowest stabilization intensity; the leftmost end of the intensity bar represents stabilization level 10, with the highest stabilization intensity. Users can then slide (or click) the intensity bar to determine the stabilization level. For example, as shown... Figure 5A As shown in (b), the user can slide the intensity bar to the shading level 3. In response to the user's action of sliding the intensity bar to shading level 3, the phone performs shading processing on the video and can display the following: Figure 5A The shading progress shown in (c) is 414, for example, 5%. When the shading progress 414 changes from 0% to 100%, it indicates that the video shading is complete. Afterwards, once the shading process is complete, the phone can display as shown... Figure 5AThe "Anti-shake processing complete" message box shown in (d) indicates to the user that video anti-shake processing is complete. Simultaneously, video window 408 can display the anti-shake video corresponding to anti-shake level 3.

[0109] In another embodiment, such as Figure 5B As shown in (a), the user can click the de-shake control 413 in the toolbar 411. In response to the user clicking the de-shake control 413, the phone can first perform de-shake processing on the video and display the result as shown in (a). Figure 5B The image stabilization progress bar 414 and the image stabilization frame 4130 are shown in (b). After the image stabilization process is complete, the phone can display the image as shown in (b). Figure 5B The "Anti-shake processing complete" message box shown in (c) indicates to the user that the video anti-shake processing is complete. Afterwards, the user can slide (or click) the intensity bar to determine the anti-shake level. For example, as shown... Figure 5B As shown in (d), the user can slide the intensity bar to the shading level 3. In response to the user's action of sliding the intensity bar to shading level 3, the video window 408 can display the shading video corresponding to shading level 3.

[0110] In another embodiment, the developer can pre-set the initial shading level. For example, the user clicks the shading control 413 on the toolbar 411. In response to the user's click on the shading control 413, the mobile phone can first perform shading processing on the video and display, as shown... Figure 5B The image stabilization progress bar 414 and the image stabilization frame 4130 are shown in (b). After the image stabilization process is complete, the phone can display the image as shown in (b). Figure 5B The "Anti-shake processing complete" message box shown in (c) indicates to the user that the video anti-shake processing is complete. It defaults to anti-shake level 3 at this point. Figure 5B As shown in (d), video window 408 can display the shaking-reduced video corresponding to shaking reduction level 3.

[0111] Optionally, the user can click the playback control 409 on interface 407 again. In response to the user's click on the playback control 409, the phone can play the shaky-free video in the video window 408. The phone can also display the playback screen of the shaky-free video in the video playback progress bar area 410.

[0112] Optionally, the "Anti-shake processing complete" prompt box can be displayed for 3 seconds and then disappear. In this embodiment of the application, the display time of the prompt box is not specifically limited in this respect.

[0113] Optionally, users can slide the intensity bar again to select a new stabilization level. For example... Figure 6 As shown in (a), the user can slide the intensity bar to the de-shake level 7. The phone responds to the user's action of sliding the intensity bar to the de-shake level 7. Figure 6As shown in (b), the mobile phone can display the shading video corresponding to shading level 7 in video window 408. In this way, users can select multiple shading levels to shade videos, covering a variety of shading scenarios and meeting the needs of various users.

[0114] Optionally, the mobile phone image stabilization process is as follows:

[0115] The mobile phone can decode user-uploaded videos frame by frame, obtaining a video frame queue and corresponding interface parameter information. The mobile phone inputs the video frame queue and corresponding interface parameter information (such as video information (video info) and process information (proc info)) into the algorithm module, which then uses a de-jitter algorithm to process the video frames. This embodiment supports expanding the interface parameter information, offering high flexibility.

[0116] Interface parameters can include video information parameters (videoinfo) and processing information parameters (procinfo). Video information parameters include the number of video frames, which are sent to the algorithm module so that it can create a corresponding memory space to store the video information. Processing information parameters include the number of pyramid levels (int pryrlv), representing the number of image pyramids needed for video frame feature matching; the feature point response threshold (int kpres), representing the feature point response threshold for the accelerated segmentation test (FAST) method during feature point detection; the number of feature points (int kpnum), representing the maximum number of effective feature points during feature point detection; and the maximum jitter pixel ratio (float shkmax), representing the maximum jitter range that the phone can handle.

[0117] Optionally, the algorithm module can be a collection of software code with debouncing capabilities, such as components or services. Components can be independent entities logically divided in C / C++, and their constituent files can be .h files, .so files, or .dll files, etc. Services refer to modules that can be deployed and run independently in a service-oriented environment. Services coordinate and cooperate with each other to provide a logical business function for the user. Each service runs in its own independent process, and services can collaborate with each other using lightweight communication mechanisms (typically RESTful APIs based on the Hypertext Transfer Protocol (HTTP)). For example, in this embodiment, video frames and corresponding interface parameter information can be input into the .so file of the algorithm module.

[0118] Optionally, the phone can preprocess each video frame in the video frame queue, converting the color RGB (Red, Green, Blue) video frames into grayscale video frames. Then, the phone can package the current video frame and the next video frame into image pairs using a data packetization operation, and add multiple image pairs to a relay queue. The image pairs in the relay queue are then distributed to multiple asynchronous threads for processing to obtain the video jitter path.

[0119] Optionally, the number of image pairs is one less than the number of video frames.

[0120] For example, the mobile phone can pack the first and second frames into an image pair 1, the second and third frames into an image pair 2, the third and fourth frames into an image pair 3, and the fourth and fifth frames into an image pair 4. The mobile phone can sequentially pack the five decoded video frames into four image pairs and add the four image pairs to the relay queue.

[0121] Optionally, multiple asynchronous threads can process multiple image pairs simultaneously. This reduces latency and improves denoising efficiency.

[0122] For example, in this embodiment, four threads process image pairs in the relay queue. The first asynchronous thread processes image pair 1, the second asynchronous thread processes image pair 2, the third asynchronous thread processes image pair 3, and the fourth asynchronous thread processes image pair 4. After processing image pair 1, the first asynchronous thread continues processing image pair 5; after processing image pair 1, the second asynchronous thread continues processing image pair 6; after processing image pair 1, the third asynchronous thread continues processing image pair 7; and after processing image pair 1, the fourth asynchronous thread continues processing image pair 8. Similarly, multiple asynchronous threads process multiple image pairs simultaneously in parallel to obtain the video jitter path.

[0123] Optionally, each first asynchronous thread can perform feature matching and motion estimation on video frames in the image pair to determine the jitter path of the video.

[0124] Feature matching is based on matching feature points in video frames within an image-to-video pair. By performing feature matching on video frames within an image-to-video pair using a mobile phone, the correspondence between feature points in the video frames can be obtained.

[0125] The methods for matching features between two video frames using mobile phones include, but are not limited to, scale-invariant feature transform (SIFT), features from accelerated segment test (FAST), oriented FAST and rotated BRIEF (ORB), speeded up robust features (SURF), and super point methods. BRIEF stands for binary robust independent elementary features.

[0126] This application embodiment takes feature matching using the FAST method as an example. In this method, the mobile phone can first perform feature detection on two video frames in an image pair to determine multiple feature points, and then perform feature matching. The FAST method can quickly detect feature points and perform real-time feature matching on video frames in an image pair, reducing latency. For example, the mobile phone can identify pixels that differ significantly from a sufficient number of pixels in its surrounding neighborhood as feature points, and then perform feature matching on the feature points of the two video frames.

[0127] For example, such as Figure 7A The image shows two video frames, including one containing a water glass. Figure 7A (a) is the current video frame. Figure 7A (b) is the previous video frame. The first asynchronous thread of the mobile phone can determine four feature points using the FAST method. For example... Figure 7A The feature points A1, B1, C1, D1, E1, and F1 of the current video frame shown in (a) are as follows: Figure 7A The feature points A2, B2, C2, D2, E2, and F2 of the previous video frame shown in (b) are as follows. Among them, feature point A1 matches feature point A2; feature point B1 matches feature point B2; feature point C1 matches feature point C2; feature point D1 matches feature point D2; feature point E1 matches feature point E2; and feature point F1 matches feature point F2.

[0128] Optionally, the phone can use the FAST method to determine six feature points for each frame in each image pair.

[0129] It should be understood that the above-described method of determining six feature points of the image-aligned video frame using the FAST method and performing feature matching on the six feature points of the image-aligned video frame is only one example. Multiple feature points of the image-aligned video frame can also be determined using other methods, and feature matching can be performed on the multiple feature points of the image-aligned video frame. The embodiments in this application do not limit this approach.

[0130] Optionally, the mobile phone can use the Lucas-Kanade optical flow method to track multiple feature points, and based on the tracking results, perform path estimation on the four vertices of the video frame in the image pair to obtain the jitter path of the four vertices of the video frame in the image pair. Optical flow is a two-frame difference estimation algorithm that can calculate the motion information of the object corresponding to the pixel using the difference between the previous frame and the current frame. Optionally, other methods can also be used to process the feature points to obtain the jitter path of the four vertices of the video frame in the image pair; this embodiment is not limited in this respect.

[0131] Specifically, the mobile phone can perform path estimation based on multiple feature points of video frames in an image pair to obtain the global homography transformation matrix gloH between the video frames in the image pair. The global homography transformation matrix gloH is used to represent the corresponding transformation relationship of corresponding pixels in two video frames in the image pair. For example, the transformation relationship between the six feature points of the current video frame and the six feature points of the previous video frame satisfies the global homography transformation matrix gloH. Optionally, the global homography transformation matrix gloH is a 3*3 matrix.

[0132] Then, the mobile phone can calculate the jitter path of the four vertices of the video frame based on the global homography transformation matrix gloH. The four vertices represent the top-left, top-right, bottom-left, and bottom-right vertices of the video frame.

[0133] For example, taking the second frame of a video as an example. The first asynchronous thread in the phone can determine the coordinates of the top-left vertex of the second frame after transformation based on the coordinates of the top-left vertex of the first frame and the global homography transformation matrix gloH. It can also determine the coordinates of the top-right vertex of the second frame after transformation based on the coordinates of the top-right vertex of the first frame and the global homography transformation matrix gloH. Finally, it can determine the coordinates of the bottom-left vertex of the second frame after transformation based on the coordinates of the bottom-right vertex of the first frame and the global homography transformation matrix gloH. Thus, the phone can obtain the transformed coordinates of the four vertices of the second frame, which is the jitter path of the four vertices of the second frame.

[0134] Similarly, while the first asynchronous thread performs feature matching and path estimation to determine the transformed coordinates of the four vertices of the second frame of the video, at least one other asynchronous thread in the phone can determine the transformed coordinates of the four vertices of at least one other video frame. In this way, the transformed coordinates of the four vertices of each video frame, i.e., the jitter path of the four vertices of each video frame, can be obtained asynchronously and in parallel.

[0135] Then, the phone can determine the jitter path of the top left corner vertex based on the jitter path of the top left corner vertex in each frame of the video.

[0136] Similarly, a mobile phone can determine the jitter path of other vertices based on the coordinates of other vertices in each frame of a video. For example, such as... Figure 8 Assume the path marked as 0 is the jitter path corresponding to the top-left vertex. At time t1, the displacement of the transformed top-left vertex in the video frame compared to the top-left vertex in the previous frame is y1. At time t2, the displacement of the transformed top-left vertex in the video frame compared to the top-left vertex in the previous frame is y2.

[0137] Afterward, the phone can obtain the jitter paths of each of the four vertices of the entire video. The jitter path represents the motion trajectory of that vertex, that is, the pixel within the video frame. The jitter paths of each of the four vertices of the entire video can be considered the video's jitter path.

[0138] In some embodiments, the above process determines the jitter path using four vertices. The phone can also use other numbers of pixels to determine the jitter path, as long as the number of pixels is greater than or equal to four. That is, the phone can use at least four pixels (or referred to as reference pixels) to determine the jitter path.

[0139] Afterwards, the mobile phone can smooth the jitter paths of each of the four vertices of the video frame, and obtain the smoothed smooth paths of each of the four vertices.

[0140] Optionally, the mobile phone can use the MAPS algorithm to smooth the jitter path in the video, resulting in a smoothed path.

[0141] For example, such as Figure 7B As shown in (a), this is the image of the first frame of the video, which includes a water glass. Figure 7B Image (b) is the second frame of the video, which also includes a water glass. Due to camera shake during filming, the position of the water glass differs between the second and first frames. Figure 7BAs shown in (c), the phone can perform translation and other transformations on the second frame image based on the results obtained from the MAPS algorithm to align the water cups in the two video frames. However, after alignment, uneven parts will appear between the edges of the two video frames, which are the black borders of the video frames. Subsequently, the phone needs to crop these black borders to obtain the cropped video frames and enlarge them to a suitable size for display on the phone screen.

[0142] Specifically, such as Figure 7B The coordinates of the top-left corner (a1, b1), top-right corner (a2, b2), bottom-left corner (a3, b3), and bottom-right corner (a4, b4) of the first frame a shown in (a) are given. Figure 7B The coordinates of the top left corner of the second frame b shown in (b) are (a1', b1'), the coordinates of the top right corner are (a2', b2'), the coordinates of the bottom left corner are (a3', b3'), and the coordinates of the bottom right corner are (a4', b4').

[0143] The coordinates of the four vertices in the smoothed second frame are determined using the MAPS algorithm. For example, ... Figure 7B As shown in (c), the coordinates of the top left corner of the smoothed second frame c are (c1, d1), the coordinates of the top right corner are (a2', b2'), the coordinates of the bottom left corner are (a3, b3), and the coordinates of the bottom right corner are (c4, d4).

[0144] For the first frame a, the area d outside the smoothed second frame c is a black border region. For the first frame b, the area e outside the smoothed second frame c is a black border region.

[0145] Similarly, the coordinates of the top-left vertex (m1, n1), the top-right vertex (m2, n2), the bottom-left vertex (m3, n3), and the bottom-right vertex (m4, n4) of the third frame are determined using the MAPS algorithm (not shown in the figure). The black border area of ​​the smoothed third frame is also determined.

[0146] Optionally, the black border area of ​​a video frame can be the misaligned area between at least two video frames, and this application embodiment does not impose specific limitations on this.

[0147] The MAPS algorithm can introduce a zero-order penalty term (zero-order velocity modeling) to determine the sliding window M_spd corresponding to the "small amplitude" jitter in the jitter path, and introduce a first-order penalty term (first-order acceleration modeling) to determine the sliding window M_acc corresponding to the "high frequency" jitter in the jitter path.

[0148] "Small amplitude" jitter refers to jitter with a small amplitude within the jitter path. For example, jitter in a video frame where the global motion vector offset is less than 5% of the shorter side of the image. For instance, if a video frame is 100 pixels wide horizontally, the jitter distance between the left and right sides is less than 5 pixels. "High frequency" jitter refers to jitter with a high frequency within the jitter path. For example, jitter where the global vector direction changes more than 3 times within 1 second. For instance, if a video frame jitters more than 3 times from left to right and more than 3 times from right to left within 1 second. The sliding window M is determined based on sliding window M_spd and sliding window M_acc to remove "small amplitude" and "high frequency" jitter from the jitter path. Then, the phone can use a Gaussian filter with sliding window M to remove the "small amplitude" and "high frequency" jitter from the jitter path.

[0149] The MAPS algorithm takes a jittery path S as input and a path length N, where the path length is the number of frames in the input video. The output is an optimized, smooth path S' with a path length N.

[0150] Optionally, developers can preset smoothing parameters: maximum amplitude (maxamp), minimum amplitude (minamp), and preset sliding window (winsize).

[0151] The specific process of the MAPS algorithm is as follows:

[0152] Step 1. Modeling the zeroth-order velocity model

[0153] 1.1) Calculate velocity weights

[0154] For the jitter path S of the input at each time step (each frame):

[0155] M_spd[t]=2*(1 / (1+np.exp(-S[t]*0.01))-0.5);

[0156] Where M_spd[t] represents the motion speed weight of the jitter path at time t, and S[t] represents the value of the jitter path S at each time (each frame). The value of the jitter path S can be understood as the displacement amplitude of the pixel point of the video frame at time t relative to the previous frame (also known as the jitter amplitude), and np.exp() represents the power function of e (also known as the exponential function).

[0157] This step allows us to calculate the speed weight value of the jitter path at each moment.

[0158] 1.2) Mapping

[0159] M_spd[t]=np.cos(M_spd[t]*np.pi / 2)*(maxamp-minamp)+minamp;

[0160] Where np.cos represents the cosine function, np.pi represents pi, maxamp represents the maximum amplitude of the jitter path, and minamp represents the minimum amplitude of the jitter path.

[0161] This step allows the velocity weight M_spd[t] from step 1.1 to be nonlinearly mapped to the corresponding Gaussian sliding window size.

[0162] 1.3) Smoothing

[0163] M_spd=gaussian_filter1d(M_spd,sigma=1)

[0164] Where M_spd represents the size of the Gaussian filter sliding window in the jitter path to remove "small amplitude" jitter, and sigma represents the standard deviation of the Gaussian kernel.

[0165] Through this step, the phone can determine the size of the Gaussian filter sliding window, M_spd, based on the preset maximum and minimum amplitude values ​​to remove "small amplitude" jitter in the jitter removal path. The larger the difference between the maximum and minimum amplitude values, the larger M_spd, and the better the removal of "small amplitude" jitter.

[0166] Step 2. First-order acceleration model modeling

[0167] 2.1) Zero-crossing numbers

[0168] grad = gradient(S)

[0169] zp_arr = grad / abs(grad)

[0170] zp_arr[t]=(zp_arr[t]–zp_arr[t-1]) / 2

[0171] Here, grad represents the gradient of the jittering path S, zp_arr(zero-pass array) represents the array storing the number of zero-crossings, abs(grad) represents the calculation of the absolute value of grad, and zp_arr[t] represents the number of zero-crossings at time t. The number of zero-crossings indicates the number of times the jittering path crosses the horizontal axis, which is the number of times the gradient changes from positive to negative.

[0172] This step allows us to calculate the number of zero-crossings zp_arr of the jitter path S.

[0173] 2.2) Zero-crossing rate

[0174] Get zp_pad using the padding function.

[0175] For each time step (each frame), the path length N is:

[0176] zp_val=sum(zp_pad[t:t+winsize])

[0177] if zp_val < 2: zp_val = 0

[0178] zp_ratio[t] = zp_val / winsize

[0179] Where zp_pad represents the number of zero crossings after padding, zp_val represents the sum of all zero crossings within the time period from t to t+winsize, zp_ratio[t] represents the zero crossing rate of the jitter path S, and winsize represents the preset threshold.

[0180] This step allows us to calculate the zero-crossing rate zp_ratio of the jitter path S.

[0181] 2.3) Mapping

[0182] M_acc=np.sin(zp_ratio*np.pi / 2)*(maxamp-minamp)+minamp

[0183] Where np.sin represents the sine function, np.pi represents pi, maxamp represents the maximum amplitude of the jitter path, and minamp represents the minimum amplitude of the jitter path.

[0184] This step allows the zero-crossing rate zp_ratio from step 2.2 above to be mapped to the Gaussian sliding window size.

[0185] 2.4) Smoothing

[0186] M_acc=gaussian_filter1d(M_acc,sigma=1)

[0187] Where M_acc represents the size of the Gaussian filter sliding window for removing "high-frequency" jitter in the jitter path, and sigma represents the standard deviation of the Gaussian kernel.

[0188] Through this step, the phone can determine the Gaussian filter sliding window size M_acc based on the preset maximum amplitude, minimum amplitude, and sliding window size to remove "high-frequency" jitter in the jitter removal path. Among them, the larger the difference between the maximum and minimum amplitude, the minimum amplitude, and the preset sliding window, the larger M_acc is, and the better the effect of removing "high-frequency" jitter.

[0189] Step 3. Construct a mixed-order model

[0190] M = (M_spd + M_acc) / 2

[0191] Where M represents the size of the Gaussian filter sliding window for removing "small amplitude" and "high frequency" jitter in the jitter path, M_spd represents the size of the Gaussian filter sliding window for removing "small amplitude" jitter in the jitter path, and M_acc represents the size of the Gaussian filter sliding window for removing "high frequency" jitter in the jitter path.

[0192] Through this step, the phone can determine the size of the Gaussian filter sliding window, M, to remove "small amplitude" and "high frequency" jitter based on M_spd and M_acc.

[0193] Step 4. Path Smoothing

[0194] For each time step (each frame), the path length N is:

[0195] S'[t]=Gaussian_filter1d(S[t], winsize=M[t])

[0196] Where S'[t] represents the smooth path S' at each time step (each frame).

[0197] Through this step, the mobile phone can perform Gaussian filtering based on the jitter path S and sliding window M input at each moment (each frame) to determine the smooth path S' at each moment (each frame). The smooth path S' at each moment (each frame) corresponds to the smoothed video frame. In this way, "small amplitude" and "high frequency" jitter in the jitter path can be removed to obtain the smooth path. Furthermore, this MAPS algorithm only removes "small amplitude" and "high frequency" jitter in the jitter path, without removing jitter with larger amplitude or lower frequency, thus avoiding excessive jitter in the video frame, resulting in excessively large black borders. This reduces the degree of video frame cropping and retains more edge information of the video frames.

[0198] For example, developers can set the first set of smoothing parameters (the weakest parameter set): for the zero-order velocity model, the parameters are minamp = 0.8, maxamp = 1.0, and winsize = 31; for the first-order acceleration model, the parameters are minamp = 0.8, maxamp = 1.0, and winsize = 31. The first set of smoothing parameters can be represented as (0.8, 1.0; 0.8, 1.0). The mobile phone can perform Gaussian filtering on the jitter path of the video based on the first set of smoothing parameters to obtain the smoothed path corresponding to each of the four vertices of the video.

[0199] Optionally, the mobile phone can determine a set of smooth paths for the video based on the smooth paths corresponding to the four vertices of the video.

[0200] When the parameters of the MAPS algorithm are set to the first set of smoothing parameters, the smoothing strength of the jitter path is low because the maximum and minimum amplitude values ​​of the first set of smoothing parameters are small, resulting in a poor jitter removal effect on the smoothed path.

[0201] As another example, developers can set a second set of smoothing parameters (the strongest parameter set): the zero-order velocity model parameters are minamp = 5.3, maxamp = 23.5, winsize = 31; the first-order acceleration model parameters are minamp = 5.3, maxamp = 23.5, winsize = 31. The second set of smoothing parameters can be represented as (5.3, 23.5; 5.3, 23.5).

[0202] The mobile phone can smooth the path based on the second set of smoothing parameters to obtain a smooth path with higher stability. When the parameters set for the MAPS algorithm are the second set of smoothing parameters, the smoothing strength of the jittery path is higher because the maximum and minimum amplitude values ​​of the second set of smoothing parameters are larger, resulting in a better anti-jitter effect.

[0203] The parameters of the zero-order velocity model and the parameters of the first-order acceleration model can be the same or different, and this application does not limit this.

[0204] For example, Figure 8 The diagram shows a smoothed path obtained based on smoothing parameters. The horizontal axis represents time (or frame count), and the vertical axis represents the displacement of a pixel. The vertical axis can represent the pixel's displacement in the x-direction (e.g., along the long side of the phone), the pixel's displacement in the y-direction (e.g., along the short side of the phone), or the magnitude of the vector representing the pixel's displacement in the x and y directions. This embodiment of the application does not limit this. Figure 8 In the diagram, the jittered path is marked as 0; the smoothed path obtained based on the first set of smoothing parameters (0.8, 1.0; 0.8, 1.0) is marked as 1; and the smoothed path obtained based on the second set of smoothing parameters (5.3, 23.5; 5.3, 23.5) is marked as 10. Figure 8 The smooth path marked as 1 and the smooth path marked as 10 shown can be the average of the smooth paths corresponding to the four vertices of the video. As another example, the smooth path marked as 1 and the smooth path marked as 10 can be the smooth path of one of the four vertices of the video. For example, the smooth path marked as 1 and the smooth path marked as 10 can be the smooth path of the top-left vertex of the video.

[0205] In this case, the winsize of both the first and second sets of smoothing parameters is 31, which means that the winsize of the smoothing parameter corresponding to the smoothing path marked as 1 and the smoothing path marked as 10 has not changed. Figure 8 The value of winsize is not shown in the text.

[0206] Optionally, the mobile phone can use a stepless de-shake method to interpolate the smooth path marked as 1 and the smooth path marked as 10 to obtain multiple sets of smooth paths.

[0207] like Figure 9 As shown, path smoothing is performed on the jittery path based on the weakest parameter set, resulting in a smooth path marked as 1. Path smoothing is performed on the jittery path based on the strongest parameter set, resulting in a smooth path marked as 10. An asynchronous parallel interpolation method is used to perform interpolation on the smooth paths marked as 1 and 10 simultaneously, that is, multiple threads simultaneously perform interpolation on the smooth paths marked as 1 and 10, which can quickly obtain n smooth paths, where n is an integer greater than or equal to 3.

[0208] Optionally, interpolation can be performed on the smooth path marked as 1 and the smooth path marked as 10 to obtain the calculation formula for multiple smooth paths, as shown in the following formula (1):

[0209] S2=S1*a+S10*b Formula (1)

[0210] Where S2 represents the smooth path obtained by interpolation, S1 represents the smooth path marked as 1, S10 represents the smooth path marked as 10, a and b are positive numbers less than 1, and a+b=1.

[0211] For example, such as Figure 10 The diagram shows eight smooth paths obtained by interpolating the smooth path marked as 1 and the smooth path marked as 10. The horizontal axis represents time (or frame number), and the vertical axis represents the displacement of a pixel. The vertical axis can be either the displacement of a pixel in the x-direction or the displacement of a pixel in the y-direction; this embodiment is not limited in this respect.

[0212] In this diagram, jittered paths are marked as 0; smoothed paths marked as 1 are obtained based on the first set of smoothing parameters; smoothed paths marked as 10 are obtained based on the second set of smoothing parameters; and smoothed paths marked as 2-9 are eight smoothed paths obtained by interpolating the smoothed paths marked as 1 and 10. The de-jitter intensity increases sequentially from smoothed paths marked as 1 to smoothed paths marked as 10.

[0213] Optionally, each smoothing path can correspond to a de-shake level. For example, a smoothing path marked as 1 corresponds to de-shake level 1; a smoothing path marked as 2 corresponds to de-shake level 2... and a smoothing path marked as 10 corresponds to de-shake level 10.

[0214] Optionally, the mobile phone can determine the smoothing parameters of the smoothed path based on the smoothed path obtained through interpolation. For example, the mobile phone interpolates the smoothed path marked 1 and the smoothed path marked 10 to obtain a smoothed path marked 2. Based on the smoothed path marked 2, the smoothing parameters of the smoothed path marked 2 can be determined as (1.3, 3.5; 1.3, 3.5). Correspondingly, based on the smoothed path marked 3, the smoothing parameters of the smoothed path marked 3 can be determined as (1.8, 6.0; 1.8, 6.0). Based on the smoothed path marked 4, the smoothing parameters of the smoothed path marked 4 can be determined as (2.3, 8.5; 2.3, 8.5). Based on the smoothed path marked 5, the smoothing parameters of the smoothed path marked 5 can be determined as (2.8, 11.0; 2.8, 11.0). Based on the smoothed path marked 6, the smoothing parameters of the smoothed path marked 6 can be determined as (3.3, 13.5; 3.3, 13.5). Based on the smoothed path marked 7, the smoothing parameters for that path are (3.8, 16.0; 3.8, 16.0). Based on the smoothed path marked 8, the smoothing parameters for that path are (4.3, 18.5; 4.3, 18.5). Based on the smoothed path marked 9, the smoothing parameters for that path are (4.8, 21.0; 4.8, 21.0).

[0215] In one embodiment, the mobile phone can sequentially perform motion compensation and image transformation (also known as cropping or trimming) on ​​the video frames in the video to obtain a reconstructed transformation matrix H for each video frame. The reconstructed transformation matrix H can represent the corresponding transformation relationship between the video frames after image transformation and the video frames before jitter reduction.

[0216] For each smooth path, the mobile phone can use multiple second asynchronous threads to perform motion compensation and image transformation on the video frames corresponding to the smooth path, so as to obtain the reconstruction transformation matrix H of each video frame corresponding to the smooth path.

[0217] In this embodiment, image pairs are used as input units, and video frames in the image pairs are used as processing units. Motion compensation and image transformation are performed on the video frames in the image pairs to obtain the reconstruction transformation matrix H corresponding to the video frames in each image pair.

[0218] Optionally, the mobile phone can use four second asynchronous threads to perform motion compensation and image transformation on the video frames in the image pairs corresponding to the smooth path in parallel, to obtain the reconstruction transformation matrix H of the video frames in each image pair. The mobile phone can also use three, more, or fewer second asynchronous threads to perform motion compensation and image transformation on the video frames in the image pairs corresponding to the smooth path in parallel, to obtain the reconstruction transformation matrix H of the video frames in each image pair. This application embodiment is not limited in this way.

[0219] For example, the first asynchronous thread processes image pair 1, the second asynchronous thread processes image pair 2, the third asynchronous thread processes image pair 3, and the fourth asynchronous thread processes image pair 4. Multiple asynchronous threads process multiple image pairs simultaneously, resulting in multiple reconstruction transformation matrices H for the multiple image pairs.

[0220] Among them, image pair 1 includes the first and second frames of the video; image pair 2 includes the second and third frames of the video; image pair 3 includes the third and fourth frames of the video; and image pair 4 includes the fourth and fifth frames of the video.

[0221] For example, the elements of the reconstruction transformation matrix H of the first frame are 1 by default. The reconstruction transformation matrix H of image pair 1 can be the reconstruction transformation matrix H of the second frame after image transformation relative to the jittered second frame; the reconstruction transformation matrix H of image pair 2 can be the reconstruction transformation matrix H of the third frame after image transformation relative to the jittered third frame; the reconstruction transformation matrix H of image pair 3 can be the reconstruction transformation matrix H of the fourth frame after image transformation relative to the jittered fourth frame; and the reconstruction transformation matrix H of image pair 4 can be the reconstruction transformation matrix H of the fifth frame after image transformation relative to the jittered fifth frame.

[0222] Alternatively, for example, the reconstruction transformation matrix H of image pair 1 can be the reconstruction transformation matrix H of the first frame after image transformation relative to the jittered first frame; the reconstruction transformation matrix H of image pair 2 can be the reconstruction transformation matrix H of the second frame after image transformation relative to the jittered second frame; the reconstruction transformation matrix H of image pair 3 can be the reconstruction transformation matrix H of the third frame after image transformation relative to the jittered third frame; and the reconstruction transformation matrix H of image pair 4 can be the reconstruction transformation matrix H of the fourth frame after image transformation relative to the jittered fourth frame. The reconstruction transformation matrix H of the fifth frame is 1 by default. This application does not limit this aspect.

[0223] Optionally, the mobile phone can perform motion compensation on the video frames corresponding to the smooth path to obtain motion residuals. Then, based on the motion residual estimation, the homography transformation matrix set between the jittery path and the smooth path is calculated.

[0224] Specifically, for each moment (each frame), based on the coordinates of the four vertices of each video frame corresponding to the jitter path and the coordinates of the four vertices of the video frame corresponding to the smoothing path, the homography transformation matrix Homo is calculated between each frame of the jittered video and the corresponding video frame of the smoothed video. Optionally, the homography transformation matrix Homo is a 3*3 matrix.

[0225] For example, the first asynchronous thread in the mobile phone can calculate the homography transformation matrix Homo1 between the second frame of the jittered video and the second frame of the smoothed video, based on the coordinates of the four vertices of the second frame of the jittered video and the coordinates of the four vertices of the second frame of the smoothed video. Figure 7B Assuming Figure 7B The coordinates of the top-left vertex (a1, b1), top-right vertex (a2, b2), bottom-left vertex (a3, b3), and bottom-right vertex (a4, b4) of the second frame b shown in (b) are given. After smoothing, aligning the water cup in the second frame b with the water cup in the first frame, the coordinates of the top-left vertex of the smoothed second frame c are (c1, d1), the top-right vertex (a2', b2'), the bottom-left vertex (a3, b3), and the bottom-right vertex (c4, d4). The first asynchronous thread in the mobile phone can calculate the homography transformation matrix Homo1 between the jittered second frame b and the smoothed second frame c based on the coordinates of the four vertices of the second frame of the jittered video and the four vertices of the second frame of the smoothed video.

[0226] Similarly, during the process of motion compensation performed by the first asynchronous thread to determine the homography transformation matrix Homo1 between the second frame of the jittery video and the corresponding second frame of the smoothed video, at least one other asynchronous thread in the phone can determine the homography transformation matrix Homo between at least one other jittery video frame and its corresponding smoothed video frame. In this way, the homography transformation matrix Homo between each jittery video frame and its corresponding smoothed video frame can be obtained asynchronously and in parallel.

[0227] The multiple homography transformation matrices (Homo) corresponding to multiple jittered video frames and shaky video frames can be combined into a homography transformation matrix group. This homography transformation matrix group can represent the corresponding transformation relationship between the jittered video and the smoothed video.

[0228] After a mobile phone smooths multiple video frame pairs using the MAPS algorithm, black borders will appear at the edges between the video frames. These black borders represent the misaligned portions between multiple video frames after smoothing.

[0229] The mobile phone can perform image transformation on each video frame in a motion-compensated image pair, removing the black borders from each smoothed video frame. Specifically, after obtaining the homography transformation matrix Homo of the video frame through motion compensation, the phone can analyze the Homo to determine the cropping coordinates of the smoothed video frame using an adaptive cropping algorithm. The phone then performs a cropping transformation on the smoothed video frame and enlarges the cropped video frame to the same size as the jittered video frame, resulting in the image-transformed video frame. The phone can obtain the translation matrix Ht and scaling matrix Hs of the smoothed video frame relative to the image-transformed video frame. For example, both the translation matrix Ht and the scaling matrix Hs are 3x3 matrices.

[0230] For example, such as Figure 7B As shown in (c), the smoothed video frame is cropped to obtain video frame c, as follows. Figure 7B As shown in (d), the video frame with the black borders cropped is enlarged to the same size as the jittered video frame to obtain the transformed video frame c'.

[0231] The stronger the MAPS algorithm's shaking reduction, the higher the alignment between smoothed video frames, resulting in larger black borders. This leads to a larger portion of the black border being cropped by the phone, ultimately resulting in smaller pixel sizes and lower resolution for the shaking-reduced video frames. For example, a smoothed video frame with a pixel size of 100*100 and frame dimensions of 10cm (length x width) will have a shaking-reduced video frame with a pixel size of 80*80 and frame dimensions of 10cm (length x width).

[0232] For example, such as Figure 11 As shown, the video frame with the largest pixel count is the one with shaky video. When the shaking level is 1, corresponding to smoothing path 1, the shaking intensity of the MAPS algorithm is relatively low, resulting in a larger pixel count for the cropped shaky video frame. When the shaking level is 5, corresponding to smoothing path 5, the shaking intensity of the MAPS algorithm is relatively high, resulting in a smaller pixel count for the cropped shaky video frame. When the shaking level is 10, corresponding to smoothing path 10, the shaking intensity of the MAPS algorithm is the highest, resulting in the lowest pixel count for the cropped shaky video frame.

[0233] The mobile phone can calculate the reconstruction transformation matrix H of each video frame in the smoothing path based on the translation matrix Ht, scaling matrix Hs, and homography transformation matrix Homo of each video frame in the smoothing path. The reconstruction transformation matrix H = Hs * Ht * Homo. Multiple jittery video frames can yield multiple reconstruction transformation matrices H.

[0234] The reconstructed transformation matrix H can represent the corresponding transformation relationship between the image-transformed video frames and the jittered video frames. The reconstructed transformation matrix H can also represent the corresponding transformation relationship between the decluttered video frames and the jittered video frames. Then, the mobile phone can process the decoded video frames again using the reconstructed transformation matrix H to obtain the decluttered video frames.

[0235] Optionally, the mobile phone can sort multiple reconstruction transformation matrices H to obtain a group of reconstruction transformation matrices.

[0236] For example, the second frame in image pair 1 corresponds to the reconstruction transformation matrix H1; the third frame in image pair 2 corresponds to the reconstruction transformation matrix H2; the fourth frame in image pair 3 corresponds to the reconstruction transformation matrix H3; and the fifth frame in image pair 4 corresponds to the reconstruction transformation matrix H4.

[0237] In some cases, the order of the reconstructed transformation matrices H obtained by the mobile phone through parallel processing by multiple second asynchronous threads may be reconstructed transformation matrices H1, H4, H3, and H2. This results in an unordered reconstructed transformation matrix H.

[0238] Optionally, the mobile phone can dynamically sort multiple reconstruction transformation matrices H using a stream sorting algorithm to obtain an ordered reconstruction transformation matrix H.

[0239] For example, the order of the sorted reconstructed transformation matrices H is: reconstructed transformation matrix H1, reconstructed transformation matrix H2, reconstructed transformation matrix H3, and reconstructed transformation matrix H4.

[0240] For each image pair corresponding to a smooth path marked as 1, one video frame corresponds to one reconstruction transformation matrix H, and multiple reconstruction transformation matrices H form a reconstruction transformation matrix group for a smooth path.

[0241] For example, the smoothed video consists of five video frames, that is, four image pairs. The video includes four reconstructed transformation matrices H. For example, it includes reconstructed transformation matrices H1, H2, H3, and H4. Reconstructed transformation matrices H1, H2, H3, and H4 can form a reconstructed transformation matrix group, which corresponds to the smoothed path marked as 1.

[0242] Optionally, the phone can access things such as Figure 10 The 10 smooth paths shown (smooth paths 1-10) are subjected to motion compensation and image transformation in sequence to obtain the reconstruction transformation matrix group 1-reconstruction transformation matrix group 10 corresponding to the 10 smooth paths.

[0243] The mobile phone can store 10 reconstruction transformation matrix groups in text format on the local disk. Optionally, reconstruction transformation matrix groups 1 to 10 can correspond to de-jitter levels 1 to 10.

[0244] Optionally, the mobile phone can render and reconstruct the jittered video frames based on the reconstruction transformation matrix group to obtain shaky video frames. The mobile phone can then encode the shaky video frames to obtain a shaky video, which is displayed as shown below. Figure 4 In the video window 408 shown in (d).

[0245] like Figure 12 As shown, the user clicks as follows Figure 4 Following the de-shake control 413 shown in (d), the user can slide the de-shake intensity bar to select the de-shake level. Responding to the user's operation, the phone can use the MAPS algorithm to de-shake the video, resulting in a de-shake video.

[0246] For example, such as Figure 5A As shown in (b), the user first slides the intensity bar to the stun level 3 to stun the video. In response to the user's action of sliding the intensity bar to stun level 3, the phone processes the 10 smoothing paths corresponding to the 10 levels, obtaining reconstruction transformation matrix groups 1-10 for each smoothing path, which are stored in text format on the local disk. Afterwards, the phone can decode the shaky video again to obtain multiple video frames, and perform affine transformations on the shaky video frames according to the reconstruction transformation matrix group 3 stored on the local disk to obtain the rendered and reconstructed stunned video frames. The phone encodes the stunned video frames to obtain the stunned video, displayed as shown below. Figure 5A In the video window 408 shown in (d).

[0247] Or, such as Figure 5A As shown in (b), the user first slides the intensity bar to the stun level 3 to stun the video. In response to the user's action of sliding the intensity bar to stun level 3, the phone first processes the smoothing path corresponding to level 3 to obtain the reconstruction transformation matrix group 3 corresponding to that smoothing path. Then, the phone can decode the shaky video again to obtain multiple video frames. The reconstruction transformation matrix group 3 performs an affine transformation on the shaky video frames to obtain the rendered and reconstructed stunned video frames. The phone encodes the stunned video frames to obtain the stunned video, as shown in [example image]. Figure 5A In video window 408 shown in (d), the reconstructed transformation matrix group 3 is stored in text format on the local disk. Then, the smoothing paths corresponding to the remaining speeds are processed to obtain the reconstructed transformation matrix group corresponding to each smoothing path, which is also stored in text format on the local disk.

[0248] For example, such as Figure 5BAs shown in (a), the user clicks the de-jitter control 413 on the toolbar 411. In response to the user's click on the de-jitter control 413, the phone processes the 10 smoothing paths corresponding to the 10 speed settings, obtaining reconstruction transformation matrix groups 1-10 for each smoothing path, which are stored in text format on the local disk. Simultaneously, the phone can display... Figure 5B The "Shake stabilization complete" message box shown in (c) indicates to the user that the shake stabilization process is complete. Afterwards, as... Figure 5B As shown in (d), the user slides the intensity bar to the de-shake level 3. In response to this action, the phone performs an affine transformation on the jittery video frame based on the transformation matrix group 3 corresponding to level 3, obtaining the rendered and reconstructed de-shake video frame. The phone then encodes the de-shake video frame to obtain the de-shake video, which is displayed as shown in [example image]. Figure 5B In the video window 408 shown in (d).

[0249] Optionally, when the user slides the intensity bar again to select a new shading level, the phone can determine the corresponding reconstruction transformation matrix group based on the new shading level, and then render and reconstruct the shading video frames based on the reconstruction transformation matrix group to obtain the shading video.

[0250] For example, the user can slide the intensity bar to the shading level 7 again. In response, the phone determines the reconstruction transformation matrix group 7 corresponding to shading level 7 from the local disk. It then decodes the shaky video again to obtain multiple video frames. Based on the reconstruction transformation matrix H in the reconstruction transformation matrix group, it performs affine transformations on the video frames sequentially to obtain the rendered reconstructed shaky video frames. The phone encodes the shaky video frames to obtain the shaky video. This video can be displayed as shown in the image. Figure 6 (b) is in the video window 408.

[0251] In some embodiments, the mobile phone can perform the image stabilization method described in this application offline, such that the mobile phone does not need to utilize other devices (e.g., servers) to implement the image stabilization method. Instead, it can be done through... Figure 13 The application module and algorithm module implementation are shown.

[0252] For example, the process of implementing the debouncing method in the algorithm module of the above application module may include:

[0253] First, the application module can acquire the shaky video and decode it into shaky video frames. Then, the application module can preprocess the shaky video frames, converting color shaky video frames into grayscale shaky video frames. Next, two consecutive shaky video frames are packaged into an image pair. The application module can also send the image pair to the algorithm module.

[0254] Then, the algorithm module can receive image pairs from the application module and, within the streaming asynchronous shaking framework, use a hybrid-order adaptive path smoothing optimization algorithm to shake the video frames in the image pairs, obtaining smooth paths. The algorithm module can then use a stepless shaking method to perform differential processing on the two smooth paths, resulting in multiple smooth paths. These multiple smooth paths can correspond to multiple shaking levels with different intensities. The algorithm module can also perform motion compensation and image transformation on the multiple smooth paths to determine a set of reconstruction transformation matrices for each smooth path. Each set of reconstruction transformation matrices corresponds to a shaking level and includes the reconstruction transformation matrix for each video frame. The algorithm module can also send the multiple sets of reconstruction transformation matrices to the application module.

[0255] Finally, the application module can receive multiple sets of reconstruction transformation matrices from the algorithm module and store them on the local disk. The application module can also render and reconstruct the video frames sequentially according to the reconstruction transformation matrices in the set corresponding to the user-selected shading level, thus obtaining the shaded video.

[0256] The method of the embodiments of this application will be described below. Optionally, as... Figure 14 As shown, the process for shaky removal in mobile phone videos is as follows: steps S101-S106:

[0257] S101. The mobile phone obtains a portion of the video frame packets in the video through each of the multiple first asynchronous threads, and processes the portion of the video frame packets to obtain the jitter path of the reference pixel between the video frames of the video frame packets.

[0258] like Figure 2 As shown, the phone decodes the video, obtaining a video frame queue. The phone can then preprocess each video frame in the queue, converting the color RGB (Red, Green, Blue) video frames to grayscale. Afterward, the phone can package the current video frame and the next video frame into image pairs (video frame packets) using a data packetization operation, and add multiple image pairs to a relay queue.

[0259] Optionally, the phone can distribute multiple image pairs in the relay queue to multiple first asynchronous multithreads for processing to obtain the video jitter path. Different first asynchronous threads process different video frame packets.

[0260] Optionally, the mobile phone can use multiple first asynchronous threads to perform feature matching and motion estimation on video frames in the image pair to determine the jitter path of the reference pixel between video frames in the video frame packet.

[0261] Specifically, such as Figure 7AAs shown, the mobile phone can perform feature detection on two video frames in an image pair to identify multiple feature points. Then, it uses the FAST method to perform feature matching on the corresponding feature points in the two video frames. Afterward, the mobile phone can use the Lucas-Kanade optical flow method to track these multiple feature points. Based on the tracking results, it performs path estimation on the four vertices (reference pixels) of the video frames in the image pair to obtain the jitter paths of the four vertices in the video frames of the image pair.

[0262] Optionally, the number of reference pixels is greater than or equal to four.

[0263] S102, The mobile phone determines the video jitter path based on the jitter path of the reference pixel points in the partial video frames processed by each first asynchronous thread.

[0264] Optionally, the mobile phone can determine the jitter path of the top-left vertex based on the jitter path of the top-left vertex in each video frame. For example, the mobile phone can perform curve fitting on the coordinates of the top-left vertex of each video frame to obtain the jitter path of the top-left vertex. Similarly, the mobile phone can perform curve fitting on the coordinates of the other three vertices of each video frame to obtain the jitter paths of the other three vertices.

[0265] Optionally, the jitter paths of the four vertices of the video are the jitter paths of the video itself.

[0266] S103. The mobile phone smooths out the shaking amplitude that is less than the first threshold and the shaking frequency that is greater than the second threshold in the shaking path, and determines the first smoothing path.

[0267] Optionally, the mobile phone can use the MAPS algorithm to smooth the jitter path of the video, resulting in a smoothed path. The smoothed path S' at each moment (each frame) corresponds to a smoothed video frame.

[0268] Developers can determine the smoothing parameters, which include at least the maximum amplitude, minimum amplitude, and preset sliding window.

[0269] The mobile phone can use the MAPS algorithm to smooth jitter in the jitter path where the jitter amplitude is less than the first threshold and the jitter frequency is greater than the second threshold, based on the smoothing parameters, jitter path, and number of video frames, and determine the first smoothing path.

[0270] Optionally, the first smoothing path may include a smoothing path marked as 1 obtained according to a first set of smoothing parameters (0.8, 1.0; 0.8, 1.0) and a smoothing path marked as 10 obtained according to a second set of smoothing parameters (5.3, 23.5; 5.3, 23.5).

[0271] For details, please refer to the smoothing process of the MAPS algorithm mentioned above, which will not be repeated here.

[0272] S104. The mobile phone determines at least one second smooth path based on at least two first smooth paths and an interpolation algorithm.

[0273] Optional, such as Figure 9 As shown, an asynchronous parallel approach is used to interpolate the smooth path marked as 1 and the smooth path marked as 10. That is, multiple threads simultaneously perform interpolation on the smooth path marked as 1 and the smooth path marked as 10, which can quickly obtain n smooth paths, where n is an integer greater than or equal to 3.

[0274] For example, such as Figure 10 As shown, the smooth paths (second smooth paths) marked 2-9 are eight second smooth paths obtained by interpolating the smooth path marked 1 and the smooth path marked 10.

[0275] S105. The mobile phone determines the reconstruction transformation matrix group corresponding to the candidate smooth path through the second asynchronous thread.

[0276] The candidate smoothing paths include at least two first smoothing paths and at least one second smoothing path. For example, candidate smoothing paths include... Figure 10 The smooth paths shown are labeled 1-10.

[0277] Optionally, motion compensation can be performed on the video frames in the corresponding image pair for each candidate smoothing.

[0278] Specifically, for each moment (each frame), based on the coordinates of the four vertices of each video frame corresponding to the jitter path and the coordinates of the four vertices of the video frame corresponding to the smoothing path, the homography transformation matrix Homo is calculated between each video frame of the jittered video and the corresponding video frame of the smoothed video. For example, as shown... Figure 7B Video frame b shown in (c) and Figure 7B The homography transformation matrix Homo between (b) and (b).

[0279] Next, the phone can perform image transformation on the video frames in the smoothed image pair based on the homography transformation matrix Homo. The phone analyzes the homography transformation matrix Homo of each video frame, determines the cropping coordinates of the smoothed video frame using an adaptive cropping algorithm, performs cropping transformation on the smoothed video frame, and enlarges the video frame with the black borders removed to the same size as the jittery video frame to obtain the image-transformed video frame.

[0280] For example, such as Figure 7B The video frame c' shown in (d) is the video frame after image transformation.

[0281] Optionally, the reconstructed transformation matrix group includes at least one reconstructed transformation matrix corresponding to a video frame packet. Each video frame packet corresponds to one reconstructed transformation matrix. The reconstructed transformation matrix represents the correspondence between the corresponding pixels in the video frame and the image-transformed video frame.

[0282] Optionally, the phone can store the reconstructed transformation matrix group in text format on the local disk.

[0283] S106. The mobile phone performs video stabilization by reconstructing the transformation matrix group and determines the stabilized video.

[0284] Optionally, the mobile phone can decode the video to obtain multiple video frames, and then render and reconstruct these multiple video frames using the corresponding reconstruction transformation matrices in the reconstruction transformation matrix group to obtain decluttered video frames. The mobile phone can then encode these decluttered video frames to obtain a decluttered video. This can be displayed as shown in the image. Figure 4 In the video window 408 shown in (d).

[0285] It should be noted that some operations in the processes of the above method embodiments may be optionally combined, and / or the order of some operations may be optionally changed. Furthermore, the execution order between the steps of each process is merely exemplary and does not constitute a limitation on the execution order between steps; other execution orders are also possible. It is not intended to indicate that the execution order is the only possible order in which these operations can be performed. Those skilled in the art will conceive of various ways to reorder the operations described herein. Additionally, it should be pointed out that the process details involved in one embodiment of this document are similarly applicable to other embodiments, or different embodiments may be combined.

[0286] Furthermore, some steps in the method embodiments can be equivalently replaced with other possible steps. Alternatively, some steps in the method embodiments may be optional and can be deleted in certain use cases. Or, other possible steps may be added to the method embodiments.

[0287] Furthermore, the above-described method embodiments can be implemented individually or in combination.

[0288] Other embodiments of this application provide an apparatus that may be the second electronic device described above, or a first electronic device, or a component of the first electronic device, or a component of the second electronic device (such as a chip system).

[0289] The device may include a display screen, a memory, and one or more processors. The display screen, memory, and processors are coupled. The memory stores computer program code, including computer instructions. When the processor executes the computer instructions, the electronic device can perform various functions or steps performed by the mobile phone in the above method embodiments.

[0290] The core structure of this electronic device can be represented as follows: Figure 15 The structure shown indicates that the electronic device includes: a processing module 151, an input module 152, a storage module 153, a display module 154, and a communication module 155.

[0291] Processing module 151 may include at least one of a central processing unit (CPU), an application processor (AP), or a communication processor (CP). Processing module 151 can perform operations or data processing related to control and / or communication with at least one of other components of the user electronic device. Optionally, processing module 151 is used to support the execution of functions by the first electronic device 100 and / or the second electronic device 200. Figure 14 S101-S105 in the middle;

[0292] Input module 152 is used to acquire user input commands or data and transmit the acquired commands or data to other modules of the electronic device. Specifically, the input method of input module 152 may include touch, gesture, proximity to the screen, or voice input. For example, the input module may be the screen of the electronic device, acquiring user input operations and generating input signals based on the acquired input operations, and transmitting the input signals to processing module 151. Optionally, input module 152 is used to acquire user-input jittery video, which can be referred to... Figure 4 The diagram shows a jittery video input interface.

[0293] Storage module 153 may include volatile memory and / or non-volatile memory. The storage module is used to store at least one related instruction or data from other modules of the user terminal device. Optionally, storage module 153 is used by the first electronic device 100 and / or the second electronic device 200 to store media interface status parameters and attribute information of the first electronic device.

[0294] Display module 154 may include, for example, a liquid crystal display (LCD), a light-emitting diode (LED) display, an organic light-emitting diode (OLED) display, a microelectromechanical system (MEMS) display, or an electronic paper display. It is used to display user-viewable content (e.g., text, images, videos, icons, symbols, etc.). Optionally, display module 154 is used to display, for example, Figure 4 , Figure 5A , Figure 5B , Figure 6 The content shown.

[0295] Communication module 155 is used to support communication between a personal terminal and other personal terminals (via a communication network). For example, the communication module can connect to a network via wireless or wired communication to communicate with other personal terminals or network servers. Wireless communication can employ at least one of the following cellular communication protocols: Long Term Evolution (LTE), LTE-A Advanced, Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Universal Mobile Telecommunications System (UMTS), Wi-Fi, or Global System for Mobile Communications (GSM). Wireless communication may include, for example, short-range communication. Short-range communication may include at least one of Wi-Fi, Bluetooth, Near Field Communication (NFC), Magnetic Stripe Transmission (MST), or GNSS. For example, a mobile phone can receive jittery video from other electronic devices.

[0296] Figure 15 The apparatus shown may also include more or fewer components, or split some components, or have other layouts of components, and the embodiments of this application do not limit this.

[0297] This application also provides a chip system, such as... Figure 16 As shown, the chip system includes at least one processor 161 and at least one interface circuit 162. The processor 161 and the interface circuit 162 are interconnected via lines. For example, the interface circuit 162 can be used to receive signals from other devices (e.g., the memory of an electronic device). As another example, the interface circuit 162 can be used to send signals to other devices (e.g., the processor 161). Exemplarily, the interface circuit 162 can read instructions stored in memory and send those instructions to the processor 161. When the instructions are executed by the processor 161, the electronic device can perform the steps in the above embodiments. Of course, the chip system may also include other discrete devices, and this application embodiment does not specifically limit this.

[0298] This application also provides a computer storage medium that includes computer instructions. When the computer instructions are executed on the electronic device, the electronic device causes the electronic device to perform various functions or steps performed by the mobile phone in the above method embodiment.

[0299] This application also provides a computer program product that, when run on a computer, causes the computer to perform the various functions or steps performed by the mobile phone in the above method embodiments.

[0300] Through the above description of the embodiments, those skilled in the art will clearly understand that, for the sake of convenience and brevity, the division of the above functional modules is only used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device and module described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0301] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0302] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0303] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0304] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, essentially or in other words, the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0305] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A video stabilization display method, characterized in that, include: Each of the multiple asynchronous threads acquires a portion of video frame packets from the video and processes the portion of video frame packets to obtain the jitter path of the reference pixel between the video frames in the video frame packets; the video frame packets processed by different asynchronous threads are different; the video frame packets include at least two adjacent video frames; The jitter path of the video is determined based on the jitter path of the reference pixel in the partial video frame packet obtained by each first asynchronous thread. For jitters in the jitter path where the jitter amplitude is less than a first threshold and the jitter frequency is greater than a second threshold, smoothing is performed to determine at least two first smoothing paths; Based on the at least two first smooth paths, at least one second smooth path is determined using an interpolation algorithm; A second asynchronous thread determines a group of reconstruction transformation matrices corresponding to candidate smoothing paths; the candidate smoothing paths include at least two first smoothing paths and at least one second smoothing path; the group of reconstruction transformation matrices includes a reconstruction transformation matrix corresponding to at least one video frame packet; the reconstruction transformation matrix is ​​obtained by performing image transformation on the video frames in the video frame packet; the reconstruction transformation matrix represents the correspondence between the video frame and the corresponding pixel points in the image-transformed video frame. The video is de-shaken by using the reconstructed transformation matrix group to determine the de-shaken video.

2. The method according to claim 1, characterized in that, The step of obtaining a portion of video frame packets from the video through each of a plurality of first asynchronous threads, and processing the portion of video frame packets to obtain the jitter path of the reference pixel point between the video frames of the video frame packets includes: Each first asynchronous thread determines the position of the reference pixel on each video frame in the at least one video frame packet, and the jitter path is obtained by curve fitting the position of the reference pixel determined by each first asynchronous thread.

3. The method according to claim 1 or 2, characterized in that, For jitter in the jitter path where the jitter amplitude is less than a first threshold and the jitter frequency is greater than a second threshold, smoothing is performed to determine at least two first smoothing paths, including: Determine the smoothing parameters, which include at least the maximum amplitude, the minimum amplitude, and a preset sliding window; Based on the smoothing parameters, the jitter path, and the number of video frames, smoothing is performed on jitters in the jitter path whose jitter amplitude is less than a first threshold and whose jitter frequency is greater than a second threshold, thereby determining at least two first smoothing paths.

4. The method according to claim 3, characterized in that, The method further includes: A first sliding window is determined based on the maximum amplitude and the minimum amplitude, and the first sliding window is used to remove jitter in the jitter path whose jitter amplitude is less than a first threshold. A second sliding window is determined based on the maximum amplitude, the minimum amplitude, and the preset sliding window. The second sliding window is used to remove jitter in the jitter path whose jitter frequency is greater than a second threshold. A third sliding window is determined based on the first sliding window and the second sliding window. The third sliding window is used to remove jitter in the jitter path where the jitter amplitude is less than the first threshold and the jitter frequency is greater than the second threshold. Based on the third sliding window and the jittering path, the jittering path is smoothed using Gaussian filtering to determine at least two first smoothed paths.

5. The method according to claim 1, characterized in that, The step of determining at least one second smooth path based on the at least two first smooth paths and an interpolation algorithm includes: determining at least one second smooth path based on the at least two first smooth paths and a linear interpolation algorithm.

6. The method according to claim 1 or 2, characterized in that, The reconstruction transformation matrix set corresponding to the candidate smooth path is determined through multiple second asynchronous threads, including: For a candidate smoothing path, the reconstruction transformation matrix of each video frame corresponding to the candidate smoothing path is determined by multiple second asynchronous threads; The reconstructed transformation matrix of each video frame is sorted using a stream sorting algorithm to determine the reconstructed transformation matrix group corresponding to a candidate smooth path.

7. The method according to claim 1 or 2, characterized in that, It also includes storing the reconstructed transformation matrix group.

8. The method according to claim 1 or 2, characterized in that, Also includes: Display a first interface, which includes a stabilization control; Receive a first operation from the user on the image stabilization control, the first operation being used to input a first image stabilization level; De-shaking the video using the reconstructed transformation matrix group includes: In response to receiving the first operation, the video is de-shaken according to the reconstruction transformation matrix group corresponding to the first image stabilization level; the reconstruction transformation matrix group corresponding to the first image stabilization level represents the reconstruction transformation matrix group corresponding to the candidate smoothing path corresponding to the first image stabilization level; wherein, the image stabilization level and the candidate smoothing path correspond one-to-one.

9. An electronic device, characterized in that, include: A processor and a memory, the memory being coupled to the processor, the memory being used to store computer program code, the computer program code including computer instructions, which, when the processor reads from the memory, cause the electronic device to perform the method as described in any one of claims 1-8.

10. A computer-readable storage medium storing instructions, characterized in that, When the instructions are executed on an electronic device, the electronic device causes the electronic device to perform the method as described in any one of claims 1-8.

Images