Image anti-shake system and method, electronic terminal and computer readable storage medium

Abstract

The invention provides an image anti-shake system and method, an electronic terminal and a computer readable storage medium, the image anti-shake system comprises a detection module and a hardware accelerator, and the hardware accelerator is connected with the detection module. The detection module is used for detecting the global motion of the current video frame relative to the reference video frame to obtain a displacement difference value corresponding to the current video frame; the reference video frame is a previous video frame adjacent to the current video frame; and the hardware accelerator is used for carrying out anti-shake processing on the current video frame based on the displacement difference value corresponding to the current video frame. According to the invention, the hardware accelerator inherent on the control chip carries out anti-shake processing on the current video frame, and a special processing unit only existing on a high-end integrated chip, such as a geometric deformation corrector or a high-performance digital signal processor, does not need to be added to carry out electronic anti-shake processing on the image; and the electronic anti-shake of the image is realized by fully utilizing a hardware accelerator generally integrated on a middle-end chip and a low-end chip, so that the optical anti-shake cost is reduced.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image stabilization system and method, an electronic terminal, and a computer-readable storage medium. Background Technology

[0002] When taking photos or videos using a terminal device, the image formation by the device's lens is not instantaneous; there is a certain exposure time. During this exposure time, the terminal device may be unstable, causing it to vibrate during the photo or video recording process. This vibration will result in blurry photos or shaky video content. Summary of the Invention

[0003] The main technical problem addressed by this application is to provide an image stabilization system and method, electronic terminal, and computer-readable storage medium, thereby solving the problem of excessively high costs in existing optical image stabilization technologies.

[0004] To solve the above-mentioned technical problems, the first technical solution adopted in this application is: to provide an image stabilization system, the image stabilization system comprising: The detection module is used to detect the global motion of the current video frame relative to the reference video frame and obtain the displacement difference corresponding to the current video frame; the reference video frame is the previous video frame adjacent to the current video frame. The hardware accelerator, connected to the detection module, is used to perform image stabilization on the current video frame based on the displacement difference corresponding to the current video frame.

[0005] The image stabilization system also includes: The preprocessing module, connected to the detection module and the hardware accelerator respectively, is used to preprocess the displacement difference of the current video frame. The preprocessing includes low-pass filtering and / or clipping.

[0006] The image stabilization system according to claim 1 is characterized in that the displacement difference includes the displacement change in a first direction and the displacement change in a second direction, wherein the first direction and the second direction are different; The hardware accelerator is used to correct the current video frame based on the displacement change in the first direction and / or the displacement change in the second direction.

[0007] The image stabilization system also includes a geometric transformation module, and the displacement difference includes the displacement change in the first direction and the displacement change in the second direction, wherein the first direction and the second direction are different. In response to the first direction being the line direction of the current video frame, the hardware accelerator is used to correct the current video frame based on the displacement change of the current video frame in the first direction. In response to the fact that the second direction is the column direction of the current video frame, the geometric transformation module is used to correct the current video frame based on the displacement change of the current video frame in the second direction.

[0008] The hardware accelerator is equipped with multiple correction neural networks, each corresponding to a different preset deviation value. The hardware accelerator is used to compare the displacement change of the current video frame in the target direction with each preset deviation value, and to correct the current video frame in the target direction based on the correction neural network corresponding to the preset deviation value that is consistent with the displacement change.

[0009] The geometric transformation module is used to correct the current video frame in the target direction based on the displacement change of the current video frame in the target direction.

[0010] The preprocessing module is also used to align and correct the memory access start address calculated based on the displacement difference to an integer multiple of the access step size of the geometric transformation module, based on the memory access requirements of the geometric transformation module.

[0011] To solve the above-mentioned technical problems, the second technical solution adopted in this application is: to provide an image stabilization method, applicable to the above-mentioned image stabilization system, comprising: Get the current video frame; The displacement difference corresponding to the current video frame is obtained by detecting the global motion of the current video frame relative to the reference video frame; the reference video frame is the previous video frame adjacent to the current video frame. The current video frame is stabilized using a hardware accelerator based on the current video frame and the corresponding displacement difference.

[0012] To solve the above-mentioned technical problems, the third technical solution adopted in this application is: to provide an electronic terminal, which includes a memory and a processor coupled to each other, the processor being used to execute program instructions stored in the memory, and the processor being used to execute program data to implement the steps in the above-mentioned image stabilization method.

[0013] To solve the above-mentioned technical problems, the fourth technical solution adopted in this application is: to provide a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, to implement the steps in the above-mentioned image stabilization method.

[0014] The beneficial effects of this application are as follows: Unlike existing technologies, this application provides an image stabilization system and method, an electronic terminal, and a computer-readable storage medium. The image stabilization system includes a detection module and a hardware accelerator. The hardware accelerator is connected to the detection module. The detection module detects the global motion of the current video frame relative to a reference video frame to obtain the displacement difference corresponding to the current video frame. The reference video frame is the previous video frame adjacent to the current video frame. The hardware accelerator performs stabilization processing on the current video frame based on the displacement difference corresponding to the current video frame. This application achieves stabilization processing on the current video frame based on the current video frame and its corresponding displacement difference using the hardware accelerator inherent on the control chip. It eliminates the need for dedicated processing units such as geometric distortion correctors or high-performance digital signal processors, which are only found on high-end integrated chips, to perform electronic image stabilization. Instead, it fully utilizes the hardware accelerators commonly integrated on mid-to-low-end chips to achieve electronic image stabilization, thereby reducing the cost of optical image stabilization. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a schematic block diagram of an embodiment of the image stabilization system provided in this application; Figure 2 This is a flowchart illustrating an embodiment of the image stabilization method provided in this application; Figure 3 This is a schematic diagram of the framework of an embodiment of the electronic terminal provided in this application; Figure 4 This is a schematic diagram of a framework of an embodiment of the computer-readable storage medium provided in this application.

[0017] In the figure: Image stabilization system 100; detection module 10; preprocessing module 20; hardware accelerator 30; geometric transformation module 40. Detailed Implementation

[0018] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0019] In the following description, specific details such as particular system architectures, interfaces, and technologies are presented for illustrative purposes rather than for limiting purposes, in order to provide a thorough understanding of this application.

[0020] In this article, the term "and / or" simply describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. Additionally, the character " / " generally indicates that the preceding and following related objects have an "or" relationship. Furthermore, "more" in this article means two or more objects.

[0021] To enable those skilled in the art to better understand the technical solution of this application, the image stabilization system and image stabilization method provided in this application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0022] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0023] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0024] Optical Image Stabilization (OIS), also known as optical image stabilization, refers to the process of using motion sensors (such as gyroscopes and accelerometers) to detect camera shake during exposure. The OIS controller then uses this data to move the lens or image sensor, controlling the motor that drives the OIS to maintain a stable optical path throughout the exposure, resulting in a sharp image. In essence, it compensates for shake by physically moving the lens or sensor, which is costly and has limited effectiveness against large-angle rotations or rapid shifts.

[0025] Electronic Image Stabilization (EIS), also known as electronic image stabilization, is a technology that uses motion sensor data for image stabilization. It calculates the motion between image frames in a sequence by using data collected by the motion sensor during each frame's exposure, and corrects this motion to generate a relatively stable image sequence. Specifically, stabilization is achieved by cropping and enlarging the central region of the image, but this significantly reduces effective resolution or field of view, especially in high-magnification zoom or low-light scenes where image quality deteriorates noticeably. It requires resources such as a GDC (Gas Dynamics Center) or a high-performance DSP processor.

[0026] The Geometric Distortion Corrector (DGC) performs high-speed spatial geometric transformations on images, including translation, rotation, scaling, lens distortion correction, and perspective correction. In image stabilization, it remaps pixels of each frame in real time based on motion vectors or correction maps calculated upstream, "pulling" shaky images back to a stable position.

[0027] A high-performance digital signal processor (DSP) is a dedicated microprocessor optimized for performing high-speed mathematical operations (especially multiply-accumulate (MAC) operations). It calculates precise camera motion trajectories by analyzing gyroscope data and / or images from preceding and following frames; it separates the calculated motion trajectory into "intentional motion" (user tracking) and "unintentional jitter," typically using complex algorithms such as Kalman filtering; it generates smooth camera motion paths and calculates the specific correction amounts required for each frame (i.e., the parameters needed to generate GDC).

[0028] SoC is an abbreviation for System on a Chip, often translated as "system-on-a-chip" or "system-level chip." SoC is an integrated circuit design methodology that aims to integrate a complete electronic system or its main functions onto a single chip. Traditionally, this "system" requires a circuit board that carries multiple independent chips.

[0029] Existing optical image stabilization technologies rely on mechanical components within the lens module for physical shake compensation. This approach is not only costly and power-intensive, but also has a limited compensation range and requires additional hardware support, making it difficult to popularize in compact or low-cost, low-performance devices. Traditional electronic image stabilization technologies, such as algorithms based on optical flow or feature point matching, typically rely on dedicated processing units such as GDC modules or high-performance DSPs to perform image remapping. However, these high-performance hardware units are often lacking in low-to-mid-range embedded chips (such as mainstream security SoCs), preventing the deployment of image stabilization functionality on resource-constrained platforms.

[0030] Therefore, the image stabilization system and image stabilization method provided in this application can perform efficient and low-power electronic image stabilization on low- and mid-range chips that lack high-performance processing units such as GDC and DSP.

[0031] Please see Figure 1 , Figure 1 This is a schematic block diagram of an embodiment of the image stabilization system provided in this application.

[0032] This application provides an image stabilization system 100, which includes a detection module 10 and a hardware accelerator 30, with the hardware accelerator 30 connected to the detection module 10.

[0033] The detection module 10 is used to detect the global motion of the current video frame relative to the reference video frame to obtain the displacement difference corresponding to the current video frame; the reference video frame is the previous video frame adjacent to the current video frame.

[0034] The hardware accelerator 30 is used to perform image stabilization on the current video frame based on the displacement difference corresponding to the current video frame. The hardware accelerator 30 can be a Neural Network Processing Unit (NPU), which becomes the core engine supporting intelligent computing.

[0035] In one embodiment, the displacement difference includes the displacement change in a first direction and the displacement change in a second direction, wherein the first direction and the second direction are different.

[0036] In one embodiment, the first direction is the row direction of the current video frame; the second direction is the column direction of the current video frame. In a specific embodiment, the first direction can be a horizontal direction, for example, the X-axis direction. The second direction can be a vertical direction, for example, the Y-axis direction.

[0037] In one embodiment, the current video frame and the reference video frame are continuously acquired by the same image acquisition device. The detection module 10 can be used to perform time integration based on the angular velocity data between the reference video frame and the current video frame detected by the gyroscope sensor in the image acquisition device to obtain the rotation angle between the reference video frame and the current video frame, and then map the rotation angle to the displacement difference (dx, dy) on the image plane based on the parameter information and projection model of the image acquisition device.

[0038] In one embodiment, the detection module 10 can be used for motion estimation based on optical flow. By calculating the optical flow between the current video frame and the reference video frame, pixel-level motion is analyzed, global main motion direction is statistically analyzed, and the displacement difference (dx, dy) between the current video frame and the reference video frame is fitted.

[0039] In one embodiment, the detection module 10 can be used to extract feature points of the current video frame and feature points of the reference video frame (such as ORB, SIFT or SURF) based on feature point matching and RANSAC algorithm fitting, perform feature matching, use the RANSAC algorithm to remove mismatched points, estimate global motion through homography matrix or affine transformation model, and finally extract the corresponding displacement difference (dx, dy) between the current video frame and the reference video frame.

[0040] Where dx represents the displacement change in the first direction and dy represents the displacement change in the second direction. The unit of displacement change can be a pixel or a sub-pixel, depending on the actual situation.

[0041] The detection module 10 can also obtain the displacement difference between the current video frame and the reference video frame through other means.

[0042] The image stabilization system 100 also includes a preprocessing module 20, which is connected to the detection module 10 and the hardware accelerator 30 respectively.

[0043] In one embodiment, the preprocessing module 20 is further configured to preprocess the displacement difference of the current video frame, the preprocessing including low-pass filtering and / or clipping.

[0044] Specifically, in order to improve the large jumps in image frames and maintain the continuity of the video, the preprocessing module 20 performs amplitude limiting processing on the displacement changes of the current video frame in the first direction and the displacement changes in the second direction.

[0045] In one specific embodiment, the preprocessing module 20 compares the displacement change of the current video frame in the first direction and the displacement change in the second direction with preset ranges, respectively. The preset ranges corresponding to the first direction and the second direction can be the same or different, depending on the actual situation.

[0046] The preset range can be pre-set or calculated based on the image size of the current video frame. For example, the preset range for the first direction is [A, B]. When dx < A < B, then dx is A; when dx > B > A, then dx is B; when B > dx > A, then dx remains unchanged. The preset range for the second direction is [C, D]. When dy < C < D, then dy is C; when dy > D > C, then dy is D; when D > dy > C, then dy remains unchanged.

[0047] In one specific embodiment, the safe region of the current video frame is 90% of the center area of ​​the current video frame. The width of the current video frame is W, and the length of the current video frame is H. Then, the maximum allowable offset of the current video frame in the first direction is dx_max = (W*0.1) / 2, and dy_max = (H*0.1) / 2. The preset range in the first direction is [-dx_max, dx_max], and the preset range in the second direction is [-dy_max, dy_max]. After clipping, dx_cliped = clip(dx, -dx_max, dx_max), and after clipping, dy_cliped = clip(dy, -dy_max, dy_max). By clipping the current video frame in the first and second directions, the phenomenon of black borders or out-of-bounds errors in the corrected video frame can be improved.

[0048] In one embodiment, the preprocessing module 20 is used to perform low-pass filtering on the displacement change of the current video frame in the first direction and the displacement change in the second direction, respectively, to prevent the current video frame from jumping relative to the reference video frame.

[0049] In one embodiment, the hardware accelerator 30 is used to correct the current video frame based on the amount of displacement change in a first direction and / or the amount of displacement change in a second direction.

[0050] In one embodiment, the image stabilization system 100 further includes a geometric transformation module 40. The geometric transformation module 40 is connected to the preprocessing module 20.

[0051] In one embodiment, the preprocessing module 20 is used to align and correct the memory access start address calculated based on the displacement difference to an integer multiple of the access step size of the geometric transformation module 40 based on the memory access requirements of the geometric transformation module 40.

[0052] In the hardware processing of image stabilization, the displacement change is the original input and fundamental basis for calculating the access start address. Since the current video frame is not stored in memory as a straightforward two-dimensional matrix, but rather "flattened" into a one-dimensional linear sequence, this mapping relationship is the formula connecting the displacement difference and the access address. For example, the access step size of the geometric transformation module 40 can be 4, 8, or 16 pixels. When the access step size of the geometric transformation module 40 is 4 pixels, and the current video frame is stored at position 0, 1, 2, 3, 4, 5, 6, 7, 8, ..., the memory access start address corresponding to the displacement difference is corrected to 4 or 8, etc., so that when the geometric transformation module 40 accesses the image in memory, it can simultaneously extract the four pixels stored in the same storage block, thereby improving the access efficiency and accuracy of the geometric transformation module 40.

[0053] In other words, the initial access address corresponding to the displacement difference after correction by the preprocessing module 20 is rounded down to the alignment boundary. This is equivalent to performing a quantization correction on the address implied by the displacement difference. Although the displacement difference remains unchanged, the initial access address initiated based on it has changed.

[0054] In response to the first direction being the line direction of the current video frame, the hardware accelerator 30 is used to correct the current video frame based on the displacement change of the current video frame in the first direction.

[0055] In response to the second direction being the column direction of the current video frame, the geometric transformation module 40 is used to correct the current video frame based on the displacement change of the current video frame in the second direction. The geometric transformation module 40 can be a hardware scaling module, which is a fixed-function hardware unit used for scaling, rotating, format conversion, and display output of image or video frames.

[0056] In one embodiment, the hardware accelerator 30 is equipped with multiple correction neural networks, each corresponding to a different preset deviation value. The hardware accelerator 30 is used to compare the displacement change of the current video frame in the target direction with each preset deviation value, and to correct the current video frame in the target direction based on the correction neural network corresponding to the preset deviation value that is consistent with the displacement change.

[0057] In one specific embodiment, if the preset range of the first direction is [-5, 5], then 11 correction neural networks are deployed on the hardware accelerator 30, with preset deviation values ​​of -5, -4, -3, -2, -1, 0, 1, 2, 3, 4, 5. When the displacement change dx corresponding to the current video frame in the first direction is -3, the current video frame is input into the correction neural network corresponding to the preset deviation value of -3, so that the correction neural network corrects the current video frame in the first direction and outputs a stable video frame.

[0058] In one embodiment, the geometric transformation module 40 is used to correct the current video frame in the target direction based on the amount of displacement change of the current video frame in the target direction.

[0059] In one specific embodiment, when the displacement change dy of the center point of the current video frame relative to the center point of the reference video frame in the second direction is greater than zero, the center point of the current video frame is shifted downward by dy pixels so that the updated center point of the current video frame is closer to the center point of the reference video frame; a sub-region of a preset size of the current video frame is extracted with the updated pixel point of the current video frame as the center, and the sub-region is enlarged to the original size of the current video frame to obtain a stable image.

[0060] In one specific embodiment, when the displacement change dy of the center point of the current video frame relative to the center point of the reference video frame in the second direction is less than zero, the center point of the current video frame is shifted upward by dy pixels so that the updated center point of the current video frame is closer to the center point of the reference video frame; a sub-region of a preset size of the current video frame is extracted with the updated pixel point of the current video frame as the center, and the sub-region is enlarged to the original size of the current video frame to obtain a stable image.

[0061] The preset size can be 95%, 90%, 85%, or 80% of the current video frame, etc.

[0062] The image stabilization system 100 provided in this embodiment does not rely on dedicated processing units such as GDC or high-performance DSPs that only exist in high-end SoCs. Instead, it fully utilizes the hardware scaling module and hardware accelerator 30 commonly integrated in mid-to-low-end chips to achieve electronic image stabilization through a heterogeneous collaborative architecture. For example, the CPU is responsible for scheduling, the hardware scaling unit is responsible for Y-direction correction, the hardware accelerator 30 is responsible for X-direction correction, and the detection module 10 is responsible for image shift estimation. This image stabilization system 100 significantly lowers the hardware threshold for image stabilization functions, enabling image stabilization capabilities to be widely deployed in cost-constrained end products such as cameras in specific fields, thus promoting the widespread adoption of image stabilization technology from high-end to affordable.

[0063] The image stabilization system 100 provided in this embodiment uses a learnable multi-path neural network based on a hardware accelerator 30 to resample the image in the X direction, completely bypassing the limitations of traditional hardware scaling modules. Since the hardware accelerator 30 supports pixel-level access to the input feature map and activates the corresponding offset convolution path through a conditional execution mechanism, it can achieve offset correction accurate to one pixel in the X direction, and even further support sub-pixel-level compensation by introducing deformable convolutions or sub-pixel convolutions.

[0064] The image stabilization system 100 provided in this embodiment supports large-shift stabilization, overcoming the physical limitations of optical image stabilization. Traditional optical image stabilization compensates for minute shakes by moving the lens assembly, which is costly and consumes a lot of power. This solution achieves dual-axis compensation on the X and Y axes electronically, and combined with a hardware scaling module and a hardware accelerator 30, it supports a wide range of shift capabilities and sub-pixel-level sampling achievable by the hardware accelerator 30, effectively dealing with severe shakes and compensating for the limited compensation range of optical image stabilization.

[0065] The image stabilization system 100 provided in this embodiment is entirely based on the existing scaling module and hardware accelerator 30 on the chip, without the need for additional optical image stabilization lenses or high-end ISP chips, thus significantly reducing costs.

[0066] Please see Figure 2 , Figure 2 This is a flowchart illustrating an embodiment of the image stabilization method provided in this application.

[0067] This embodiment provides an image stabilization method, which includes the following steps.

[0068] S1: Get the current video frame.

[0069] S2: Detect the global motion of the current video frame relative to the reference video frame to obtain the displacement difference corresponding to the current video frame; the reference video frame is the previous video frame adjacent to the current video frame.

[0070] S3: The current video frame is stabilized using a hardware accelerator based on the current video frame and the displacement difference corresponding to the current video frame.

[0071] The image stabilization method provided in this embodiment is based on the image stabilization system in the above embodiment.

[0072] The image stabilization method provided in this embodiment achieves image stabilization processing of the current video frame based on the current video frame and the displacement difference corresponding to the current video frame through the hardware accelerator inherent on the optical image stabilization control chip. It does not require the addition of a geometric distortion corrector or a high-performance digital signal processor, which are dedicated processing units that only exist on high-end integrated chips, to perform electronic image stabilization. Instead, it makes full use of the hardware accelerators that are commonly integrated on mid-to-low-end chips to achieve electronic image stabilization, thereby reducing the cost of optical image stabilization.

[0073] Please see Figure 3 , Figure 3 This is a schematic diagram of the framework of an embodiment of the electronic terminal provided in this application. The electronic terminal 80 includes a memory 81 and a processor 82 coupled to each other. The processor 82 is used to execute program instructions stored in the memory 81 to implement the steps of any of the above-described image stabilization method embodiments. In a specific implementation scenario, the electronic terminal 80 may include, but is not limited to, security devices such as embedded network cameras, smart bullet cameras, dome cameras, network video recorders (NVRs), and edge computing gateways; in addition, it may also include mobile security terminals (such as portable recorders) equipped with embedded chips, which are not limited here.

[0074] Specifically, processor 82 controls itself and memory 81 to implement the steps of any of the above-described image stabilization method embodiments. Processor 82 can also be referred to as a CPU (Central Processing Unit). Processor 82 may be an integrated circuit chip with signal processing capabilities. Processor 82 can also be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor or any conventional processor. Furthermore, processor 82 can be implemented using integrated circuit chips.

[0075] Please see Figure 4 , Figure 4This is a schematic diagram of a framework of an embodiment of the computer-readable storage medium provided in this application. The computer-readable storage medium 90 stores program instructions 901 that can be executed by a processor. The program instructions 901 are used to implement the steps of any of the above-described image stabilization method embodiments.

[0076] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0077] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0078] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.

[0079] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0080] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0081] The above are merely embodiments of this application and do not limit the scope of patent protection of this application. Any equivalent structural or procedural changes made using the content of this application’s specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of this application.

Claims

1. An image stabilization system, characterized in that, The image stabilization system includes: The detection module is used to detect the global motion of the current video frame relative to the reference video frame to obtain the displacement difference value corresponding to the current video frame; the reference video frame is the previous video frame adjacent to the current video frame. A hardware accelerator, connected to the detection module, is used to perform image stabilization on the current video frame based on the displacement difference corresponding to the current video frame.

2. The image stabilization system according to claim 1, characterized in that, The image stabilization system also includes: A preprocessing module, connected to the detection module and the hardware accelerator respectively, is used to preprocess the displacement difference of the current video frame. The preprocessing includes low-pass filtering and / or amplitude limiting.

3. The image stabilization system according to claim 1, characterized in that, The displacement difference includes the displacement change in a first direction and the displacement change in a second direction, wherein the first direction is different from the second direction; The hardware accelerator is used to correct the current video frame based on the displacement change of the current video frame in the first direction and / or the displacement change in the second direction.

4. The image stabilization system according to claim 1, characterized in that, The image stabilization system further includes a geometric transformation module, and the displacement difference includes the displacement change in a first direction and the displacement change in a second direction, wherein the first direction and the second direction are different. In response to the first direction being the line direction of the current video frame, the hardware accelerator is used to correct the current video frame based on the displacement change of the current video frame in the first direction; In response to the second direction being the column direction of the current video frame, the geometric transformation module is used to correct the current video frame based on the displacement change of the current video frame in the second direction.

5. The image stabilization system according to claim 3 or 4, characterized in that, The hardware accelerator is equipped with multiple correction neural networks, each of which corresponds to a different preset deviation value. The hardware accelerator is used to compare the displacement change of the current video frame in the target direction with each of the preset deviation values, and to correct the current video frame in the target direction based on the correction neural network corresponding to the preset deviation value that is consistent with the displacement change.

6. The image stabilization system according to claim 3 or 4, characterized in that, The geometric transformation module is used to correct the current video frame in the target direction based on the displacement change of the current video frame in the target direction.

7. The image stabilization system according to claim 4, characterized in that, The preprocessing module is further configured to, based on the memory access requirements of the geometric transformation module, align and correct the memory access start address calculated according to the displacement difference to an integer multiple of the access step size of the geometric transformation module.

8. An image stabilization method, characterized in that, The image stabilization system applicable to any one of claims 1 to 7 comprises: Get the current video frame; The displacement difference value corresponding to the current video frame is obtained by detecting the global motion of the current video frame relative to the reference video frame; the reference video frame is the previous video frame adjacent to the current video frame. The current video frame is stabilized using a hardware accelerator based on the current video frame and the corresponding displacement difference.

9. An electronic terminal, characterized in that, The electronic terminal includes a memory and a processor coupled to each other. The processor is used to execute program instructions stored in the memory and to execute program data to implement the steps in the image stabilization method as described in claim 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the image stabilization method as described in claim 8.

Images