Video shake processing method and apparatus, and electronic device and computer-readable storage medium

Abstract

Provided in the present application are a video shake processing method and apparatus, and an electronic device and a computer-readable storage medium. The method comprises: acquiring a video to be processed, and determining a shake compensation matrix for each original image frame in the video to be processed; on the basis of the shake compensation matrix for the original image, performing compensation processing on the original image to obtain a compensated image and vertex information of the compensated image; on the basis of the vertex information, determining a bounding box of the compensated image, wherein two adjacent edges of the bounding box are respectively parallel to two coordinate axes of an image coordinate system of the original image; on the basis of boundary information of the original image and the compensated image, determining an image to be displayed, and on the basis of the bounding box, determining a region to be cropped in the image to be displayed; and on the basis of a corresponding region to be cropped, cropping each image frame to be displayed to obtain a video subjected to anti-shake processing. The present application can improve the flexibility and accuracy of black-border processing in a video anti-shake process.

Description

Video jitter processing method and device, electronic equipment and computer readable storage medium

[0001] The present application claims priority from the Chinese patent application No. 202410870131.9, filed on June 28, 2024 in the China Patent Office, and entitled "Video jitter processing method and device, electronic equipment and computer readable storage medium", the whole content of which is incorporated herein by reference. TECHNICAL FIELD

[0002] The present application relates to the technical field of image processing, and in particular to a video jitter processing method and device, electronic equipment and computer readable storage medium. BACKGROUND

[0003] In the process of camera shooting, due to the possible movement of the camera, the video shot by the camera may have jitter. Intelligent anti-shake is an image anti-shake technology based on image algorithm, which can use feature matching, smoothing and compensation methods to reduce the picture blur caused by jitter during camera shooting or after shooting the video, and obtain a more stable and clear picture. In order to compensate for the imaging jitter caused by camera movement, intelligent anti-shake technology introduces black edges at the edges of the image.

[0004] In related technologies, the edges are directly cut off by default cutting ratio, and then the image is enlarged to the original size to cut off the black edges, so as to obtain the processed video. However, the flexibility and accuracy of using the default cutting ratio to process the black edges in the image are low. TECHNICAL PROBLEM

[0005] The embodiments of the present application provide a video jitter processing method and device, electronic equipment and computer readable storage medium, which can improve the flexibility and accuracy of black edge processing in the video anti-shake process. TECHNICAL SOLUTION

[0006] The technical scheme of the embodiments of the present application is implemented as follows:

[0007] The embodiment of the present application provides a video jitter processing method, which comprises the following steps: obtaining a video to be processed, and determining a jitter compensation matrix of each original image in the video to be processed; performing compensation processing on the original image based on the jitter compensation matrix of the original image, to obtain a compensated image and vertex information of the compensated image; determining a bounding box of the compensated image based on the vertex information; two adjacent edges of the bounding box are parallel to two coordinate axes of an image coordinate system of the original image; determining a to-be-displayed image based on boundary information of the original image and the compensated image, and determining a to-be-cropped region in the to-be-displayed image based on the bounding box; cropping each frame of the to-be-displayed image according to the corresponding to-be-cropped region, to obtain a video after jitter processing.

[0008] The embodiment of the present application provides a video jitter processing device, which comprises: a matrix determination module, which is used for obtaining a video to be processed, and determining a jitter compensation matrix of each original image in the video to be processed; an image compensation module, which is used for performing compensation processing on the original image based on the jitter compensation matrix of the original image, to obtain a compensated image and vertex information of the compensated image; a bounding box determination module, which is used for determining a bounding box of the compensated image based on the vertex information; two adjacent edges of the bounding box are parallel to two coordinate axes of an image coordinate system of the original image; a to-be-cropped region determination module, which is used for determining a to-be-displayed image based on boundary information of the original image and the compensated image, and determining a to-be-cropped region in the to-be-displayed image based on the bounding box; and a cropping module, which is used for cropping each frame of the to-be-displayed image according to the corresponding to-be-cropped region, to obtain a video after jitter processing.

[0009] In the above scheme, the vertex information comprises first pixel coordinates of each first vertex of the compensated image; the bounding box determination module is further used for determining a first coordinate point and a second coordinate point based on the first pixel coordinates of the each first vertex; the first coordinate point and the second coordinate point are taken as diagonal vertices, and the bounding box of the compensated image is constructed based on the diagonal vertices.

[0010] In the above scheme, the bounding box determination module is further used for determining a first horizontal coordinate value as a minimum horizontal coordinate value in the first pixel coordinates of the each first vertex, determining a first vertical coordinate value as a minimum vertical coordinate value in the first pixel coordinates of the each first vertex, determining a second horizontal coordinate value as a maximum horizontal coordinate value in the first pixel coordinates of the each first vertex, and determining a second vertical coordinate value as a maximum vertical coordinate value in the first pixel coordinates of the each first vertex; a first coordinate point is determined based on the first horizontal coordinate value and the first vertical coordinate value, and a second coordinate point is determined based on the second horizontal coordinate value and the second vertical coordinate value.

[0011] In the above scheme, the clipping region determination module is further configured to determine the maximum value between the first horizontal coordinate value and the origin horizontal coordinate value of the image coordinate system as the initial clipping pixel number corresponding to the first edge of the to-be-displayed image; determine the maximum value between the first vertical coordinate value and the origin vertical coordinate value of the image coordinate system as the initial clipping pixel number corresponding to the second edge of the to-be-displayed image; obtain a first difference value between the width of the compensated image and the second horizontal coordinate value, and determine the maximum value between the first difference value and the origin horizontal coordinate value as the initial clipping pixel number corresponding to the third edge of the to-be-displayed image; obtain a second difference value between the height of the compensated image and the second vertical coordinate value, and determine the maximum value between the second difference value and the origin vertical coordinate value as the initial clipping pixel number corresponding to the fourth edge of the to-be-displayed image; perform smoothing processing on the initial clipping pixel number corresponding to each edge to obtain the clipping pixel number corresponding to each edge; and determine the to-be-clipped region in the to-be-displayed image based on the clipping pixel number corresponding to each edge.

[0012] In the above scheme, the clipping region determination module is further configured to obtain a first covariance matrix and a second covariance matrix of the i-th to-be-displayed image; for each edge of the i-th to-be-displayed image, determine an observation vector based on the initial clipping pixel number corresponding to the edge and a change speed, the change speed being a difference between the initial clipping pixel number corresponding to the edge of the i-th to-be-displayed image and the initial clipping pixel number corresponding to the edge of the (i-1)-th to-be-displayed image; determine the clipping pixel number corresponding to the edge based on the first covariance matrix, the second covariance matrix, the observation vector, and the clipping pixel number of the (i-1)-th to-be-displayed image; wherein i is an integer greater than 1, and the clipping pixel number corresponding to the edge of the first to-be-displayed image is the initial clipping pixel number.

[0013] In the above scheme, the clipping region determination module is further configured to obtain a preset first covariance matrix of the i-th to-be-displayed image; determine a second covariance prediction matrix of the (i-1)-th to-be-displayed image based on a first covariance matrix and a second covariance matrix of the (i-1)-th to-be-displayed image; determine the second covariance matrix of the i-th to-be-displayed image based on a preset coefficient matrix and the second covariance prediction matrix of the (i-1)-th to-be-displayed image; and the second covariance matrix of the first to-be-displayed image is a unit matrix.

[0014] In the foregoing solution, the clipping region determination module is further configured to determine a clipping state vector of the i-1th frame of to-be-displayed image based on a number of clipped pixels and a change speed of the i-1th frame of to-be-displayed image; determine a clipping state prediction vector of the i-th frame of to-be-displayed image based on a preset coefficient matrix and the clipping state vector of the i-1th frame of to-be-displayed image; determine a difference between an observation vector of the i-th frame of to-be-displayed image and the clipping state prediction vector as a target vector; determine an inverse matrix of a sum of the first covariance matrix and the second covariance matrix as a target matrix; obtain a clipping state vector corresponding to the edge in the i-th frame of to-be-displayed image by adding a product of the first covariance matrix, the target matrix, and the target vector to the clipping state prediction vector of the i-th frame of to-be-displayed image; and obtain the number of clipped pixels corresponding to the edge from the clipping state vector corresponding to the edge in the i-th frame of to-be-displayed image.

[0015] In the foregoing solution, the image compensation module is further configured to obtain pixel coordinates of each second vertex of the original image in the image coordinate system; determine homogeneous coordinates corresponding to the pixel coordinates of each second vertex; perform transformation processing on the homogeneous coordinates of each second vertex based on the jitter compensation matrix to obtain vertex information of the compensated image; and perform movement on the original image based on the vertex information of the compensated image to obtain the compensated image.

[0016] An electronic device is provided in an embodiment of the present application, and the electronic device includes a memory configured to store computer executable instructions or computer programs, and a processor configured to execute the computer executable instructions or computer programs stored in the memory to implement the jitter processing method for a video provided in the embodiments of the present application.

[0017] A computer readable storage medium is provided in an embodiment of the present application, and the computer readable storage medium stores computer programs or computer executable instructions, and is configured to be executed by a processor to implement the jitter processing method for a video provided in the embodiments of the present application. Advantages

[0018] The embodiments of the present application have the following advantages:

[0019] In the process of video anti-shake, the original image in the video to be processed can be first processed by a shake compensation matrix transformation to obtain a compensated image. The image to be displayed is determined based on the boundary information of the original image and the compensated image. Since the position range of the displayable video, i.e., the boundary information of the original image, is fixed and unchanged, the positions of the compensated image and the original image are different, and therefore only part of the compensated image is in the image to be displayed, and the rest is black edge. The bounding box of the compensated image can be determined based on the vertex information of the compensated image, and two adjacent edges of the bounding box are parallel to two coordinate axes of the image coordinate system of the original image. The region outside the bounding box in the image to be displayed does not have the compensated image, and therefore the region to be cropped in the image to be displayed can be determined based on the bounding box. Finally, each frame of the image to be displayed is cropped according to the corresponding region to be cropped to obtain the video after anti-shake processing. The embodiments of the present application automatically calculate the region to be cropped in the image to be displayed based on the bounding box of the compensated image, realize the image to be displayed corresponding to different original images, can obtain different regions to be cropped, improve the flexibility and accuracy of black edge processing, and further improve the video quality of the video after anti-shake processing. BRIEF DESCRIPTION OF DRAWINGS

[0020] FIG. 1 is a schematic diagram of the architecture of a video shake processing system according to an embodiment of the present application;

[0021] FIG. 2 is a schematic diagram of the structure of an electronic device according to an embodiment of the present application;

[0022] FIG. 3 is a first flowchart of a video shake processing method according to an embodiment of the present application;

[0023] FIG. 4 is a second flowchart of a video shake processing method according to an embodiment of the present application;

[0024] FIG. 5 is a third flowchart of a video shake processing method according to an embodiment of the present application;

[0025] FIG. 6 is a fourth flowchart of a video shake processing method according to an embodiment of the present application;

[0026] FIG. 7 is a fifth flowchart of a video shake processing method according to an embodiment of the present application;

[0027] FIG. 8 is a flowchart of an image anti-shake system according to an embodiment of the present application;

[0028] FIG. 9 is a schematic diagram of an original image and a compensated image in the process of video shake processing according to an embodiment of the present application.

[0029] It should be noted that the above-mentioned "first" and "second" are only used to distinguish different schemes, and do not represent the degree of superiority or priority in the implementation process. Embodiments of the present application

[0030] In order to make the purposes, technical solutions and advantages of the present application clearer, the following will further describe the present application in conjunction with the accompanying drawings, and the described embodiments should not be regarded as limiting the present application. All other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0031] In the following description, "some embodiments" are referred to, which describe a subset of all possible embodiments, but it can be understood that "some embodiments" can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict.

[0032] In the following description, the terms "first\second\third" are only to distinguish similar objects, and do not represent a specific order of the objects. It can be understood that "first\second\third" can be interchanged in a specific order or sequence as allowed, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0033] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined target, and can be implemented entirely or partially by using software, hardware (such as a processing circuit or a memory) or a combination thereof. Similarly, one processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be a part of an integral module or unit that includes the functions of the module or unit.

[0034] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0035] The relevant data collection process in the embodiments of the present application should strictly comply with the requirements of relevant laws and regulations, obtain the informed consent or separate consent of the personal information subject, and within the scope of authorization of laws and regulations and the personal information subject, carry out subsequent data use and processing.

[0036] Before the embodiments of the present application are further described in detail, the terms and terms related to the embodiments of the present application are explained, and the terms and terms related to the embodiments of the present application are applicable to the following explanations.

[0037] 1) Video stabilization: A technique to reduce or eliminate the image blur and jitter caused by the image acquisition device (e.g., camera, etc.) or environmental shaking during video recording. Video stabilization can improve the viewing experience of the video, making the captured video more stable and clear. Video stabilization is usually achieved through optical stabilization, electronic stabilization, and intelligent stabilization methods.

[0038] 2) Border: A blank area introduced at the edge of the image to compensate for the image blur caused by motion during video recording. These blank areas do not display video content and are usually filled with black color.

[0039] 3) Bounding box: A geometric shape used to contain and represent the boundaries of an image. The embodiments of the present application refer to a quadrilateral used to contain the compensated image obtained after compensating the original image with a jitter compensation matrix.

[0040] 4) Original image: An image in a video directly recorded by an image acquisition device.

[0041] 5) Jitter compensation matrix: A mathematical tool used to describe the spatial transformation relationship between images or video frames. This transformation relationship reflects the movement of one image relative to another in space, including translation, rotation, scaling, etc. Through the jitter compensation matrix, the image can be transformed to eliminate or reduce the image jitter caused by camera shaking or other factors, thereby improving the quality and clarity of the image.

[0042] 6) Covariance matrix: In the Kalman filter algorithm, the covariance matrix is used to describe the uncertainty of the predicted state, representing the covariance of the state estimation error. In the update process of the Kalman filter, the covariance matrix is updated according to the prediction error and observation error to reflect the latest state estimation uncertainty. The update of the covariance matrix helps to improve the accuracy of state estimation and establishes a balance between prediction and observation.

[0043] 7) Image coordinate system: A coordinate system used in computer vision and image processing to describe the position of pixels in an image. In the image coordinate system, there are usually two main coordinate axes, namely the horizontal axis (u-axis) and the vertical axis (v-axis), which correspond to the width and height of the image, respectively. The origin (0, 0) of the image coordinate system is usually located at the top left corner of the image.

[0044] 8) Homogeneous coordinates: A coordinate system used to represent points, vectors, or matrices, which has a wide range of applications in mathematics and computer science, especially in computer graphics and computer vision. In the homogeneous coordinate system, vectors can be represented by three or four components, with the last component usually being 1. This representation method introduces an additional dimension, making mathematical operations more convenient.

[0045] In the related art, during the process of photographing by a camera, due to the possible movement of the camera, the video photographed by the camera can have jitter. In order to remove the jitter of the video, mainstream technologies are divided into three categories: optical anti-shake, electronic anti-shake and intelligent anti-shake. Among them, optical anti-shake is to drive the anti-shake component to move the lens module in the opposite direction of the jitter quickly when photographing jitter, so as to offset the jitter and realize the final stable imaging. The biggest problem of this method is that the camera itself needs to be modified. For cameras that cannot be modified or videos that have been photographed and have jitter, optical anti-shake cannot be used to process. Electronic anti-shake is to realize anti-shake through software algorithm. The operation principle of electronic anti-shake technology is to detect the amplitude of mobile phone jitter through acceleration sensor and gyroscope module, and then dynamically adjust the ISO, shutter and imaging algorithm to make blur correction. This method does not need to modify the imaging module itself, but also needs to increase multiple sensors on the camera, which is more expensive and cannot process the already photographed video. Intelligent anti-shake is an image anti-shake technology based on image algorithm, which can use feature matching, smoothing and compensation methods to reduce the picture blur caused by jitter during camera shooting or after shooting the video, and obtain more stable and clear pictures.

[0046] In order to compensate for the imaging jitter caused by camera movement, intelligent anti-shake technology introduces black edges at the edges of the image. Therefore, in general, a default cropping ratio is set to directly cut off a certain proportion of the edges, and then the image is enlarged to the original size for processing. Setting a default cropping ratio for cutting has the following problems: the cropping ratio of the top, bottom, left and right of the image is the same; for frames with small jitter and less black edges, the cropping ratio can be too large; for frames with large jitter and more black edges, the cropping ratio can be too small.

[0047] Based on the problems in the related art, the embodiments of the present application provide a video jitter processing method, device and equipment and computer readable storage medium, which can improve the flexibility and accuracy of black edge processing in the video anti-shake process.

[0048] In the method for processing video jitter provided in the embodiments of the present application, first, a video to be processed is obtained, and a jitter compensation matrix of each original image in the video to be processed is determined; then, the original image is compensated based on the jitter compensation matrix of the original image to obtain a compensated image and vertex information of the compensated image; then, the bounding box of the compensated image is determined based on the vertex information; two adjacent edges of the bounding box are parallel to two coordinate axes of an image coordinate system of the original image; then, a to-be-displayed image is determined based on the boundary information of the original image and the compensated image, and a to-be-cropped region in the to-be-displayed image is determined based on the bounding box; finally, each to-be-displayed image is cropped according to the corresponding to-be-cropped region to obtain a video processed by anti-jitter.

[0049] The following describes an exemplary application of the video jitter processing device provided in the embodiments of the present application. The video jitter processing device provided in the embodiments of the present application is an electronic device for implementing the method for processing video jitter. The electronic device provided in the embodiments of the present application can be implemented as various types of terminals such as a notebook computer, a tablet computer, a desktop computer, a set-top box, a smart phone, a smart speaker, a smart watch, a smart television, a vehicle-mounted terminal, and the like, or can be implemented as a server. The server can be a standalone physical server, or a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content distribution networks (CDN), and basic cloud computing services such as big data and artificial intelligence platforms. The terminals and the server can be connected directly or indirectly through wired or wireless communication, which is not limited in the embodiments of the present application. In the following, exemplary applications of the video jitter processing device implemented as a server or a terminal are described.

[0050] Referring to FIG. 1, FIG. 1 is an architecture schematic diagram of a video jitter processing system provided in the embodiments of the present application. To implement anti-jitter processing on a video to be processed, a video jitter processing application can be provided. For example, the video jitter processing application can be an application dedicated to video jitter processing, or can be a functional module in another application (for example, a video jitter processing module in a video recording application, etc.). The video jitter processing system 100 in the embodiments of the present application at least includes a terminal 400, a network 300, and a server 200. The server 200 is a server of a video processing application. The server 200 can constitute a video jitter processing device in the embodiments of the present application, that is, the server 200 implements the method for processing video jitter in the embodiments of the present application. The terminal 400 connects to the server 200 through the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0051] Referring to FIG. 1, a user can perform an interactive operation, such as a click shooting operation, and the like, on a client of a terminal 400 in an application of a video shake processing. After receiving the interactive operation of the user, the client can collect a to-be-processed video through the terminal, encapsulate the to-be-processed video into a video shake processing request, and send the video shake processing request to a server 200 through a network 300. After receiving the video shake processing request, the server 200 acquires the to-be-processed video in response to the video shake processing request, and determines a shake compensation matrix of each frame of an original image in the to-be-processed video. The server 200 compensates the original image based on the shake compensation matrix of the original image, to obtain a compensated image and vertex information of the compensated image. The server 200 determines a bounding box of the compensated image based on the vertex information. Two adjacent edges of the bounding box are parallel to two coordinate axes of an image coordinate system of the original image. The server 200 determines a to-be-displayed image based on boundary information of the original image and the compensated image, and determines a to-be-cropped region in the to-be-displayed image based on the bounding box. The server 200 crops each frame of the to-be-displayed image according to the corresponding to-be-cropped region, to obtain a video after shake processing. After determining the video after shake processing, the server 200 can also send the video after shake processing to the terminal 400, to show the video after shake processing to the user.

[0052] In some embodiments, the terminal 400 can also perform the video shake processing method of the embodiments of the present application by itself, that is, after the terminal 400 receives the interactive operation input by the user through the client, the terminal 400 acquires the to-be-processed video, and determines a shake compensation matrix of each frame of an original image in the to-be-processed video. The terminal 400 compensates the original image based on the shake compensation matrix of the original image, to obtain a compensated image and vertex information of the compensated image. The terminal 400 determines a bounding box of the compensated image based on the vertex information. Two adjacent edges of the bounding box are parallel to two coordinate axes of an image coordinate system of the original image. The terminal 400 determines a to-be-displayed image based on boundary information of the original image and the compensated image, and determines a to-be-cropped region in the to-be-displayed image based on the bounding box. The terminal 400 crops each frame of the to-be-displayed image according to the corresponding to-be-cropped region, to obtain a video after shake processing. After determining the video after shake processing, the terminal 400 displays the video after shake processing on a client interface of the terminal 400.

[0053] The video jitter processing method provided in the embodiments of the present application can also be implemented based on a cloud platform and through cloud technology. For example, the server 200 can be a cloud server. The cloud server is used to obtain a to-be-processed video, and determine a jitter compensation matrix of each frame of original image in the to-be-processed video; or the cloud server is used to compensate the original image based on the jitter compensation matrix of the original image, to obtain a compensated image and vertex information of the compensated image; or the cloud server is used to determine a bounding box of the compensated image based on the vertex information; or the cloud server is used to determine a to-be-displayed image based on the boundary information of the original image and the compensated image, and determine a to-be-cropped region in the to-be-displayed image based on the bounding box; or the cloud server is used to crop each frame of to-be-displayed image according to the corresponding to-be-cropped region, to obtain a video after jitter processing.

[0054] It should be noted that the cloud technology refers to a hosting technology that unifies a series of resources such as hardware, software, network, and the like in a wide area network or a local area network, to realize data calculation, storage, processing and sharing. The cloud technology is a general term of network technology, information technology, integration technology, management platform technology, application technology and the like applied based on a cloud computing business model, can form a resource pool, and is used on demand, flexibly and conveniently. The cloud computing technology will become an important support. The background service of a technical network system needs a large amount of calculation and storage resources, such as a video website, a picture website and more portals. With the high development and application of the Internet industry, in the future, each item is likely to have its own identification mark, and needs to be transmitted to a background system for logical processing. Different levels of data will be processed separately, and data of various industries needs strong system support, which can be realized through cloud computing.

[0055] Referring to FIG. 2, FIG. 2 is a structural schematic diagram of an electronic device provided in the embodiments of the present application. The electronic device shown in FIG. 2 includes at least one processor 410, a memory 450, at least one network interface 420 and a user interface 430. The various components in the terminal 400 are coupled together through a bus system 440. It can be understood that the bus system 440 is used to realize the connection and communication between the components. The bus system 440 includes not only a data bus, but also a power bus, a control bus and a status signal bus. However, for the purpose of clear illustration, all kinds of buses are marked as the bus system 440 in FIG. 2.

[0056] The processor 410 can be an integrated circuit chip with signal processing capability, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0057] The user interface 430 includes one or more output devices 431 that enable presentation of media content, including one or more speakers and one or more visual display screens. The user interface 430 also includes one or more input devices 432 that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0058] The memory 450 can be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, and the like. The memory 450 optionally includes one or more storage devices remotely located from the processor 410 in a physical location.

[0059] The memory 450 includes volatile memory or non-volatile memory, and can also include both volatile and non-volatile memory. Non-volatile memory can be read only memory (ROM), volatile memory can be random access memory (RAM). The memory 450 described in embodiments of the present application is intended to include any suitable type of memory.

[0060] In some embodiments, the memory 450 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or a subset or superset thereof, which are illustratively described below.

[0061] The operating system 451 includes systems programs for handling various basic system services and for performing hardware dependent tasks, such as a framework layer, a core library layer, a driver layer, and the like, for implementing various basic services as well as handling hardware dependent tasks;

[0062] The network communication module 452 is for communicating to other electronic devices via one or more (wired or wireless) network interfaces 420, example network interfaces 420 include: Bluetooth, wireless fidelity (WiFi), universal serial bus (USB), and the like;

[0063] The presentation module 453 is for enabling presentation of information via one or more output devices 431 associated with the user interface 430 (e.g., display screens, speakers, and the like) (e.g., user interfaces for operating peripheral devices and displaying content and information);

[0064] The input processing module 454 is for detecting and interpreting one or more user inputs or interactions from one or more input devices 432.

[0065] In some embodiments, the video shake processing apparatus provided by the embodiments of the present application can be implemented in a software manner. FIG. 2 shows a video shake processing apparatus 455 stored in a memory 450, which can be software in the form of programs and plug-ins, etc., including the following software modules: a matrix determination module 4551, an image compensation module 4552, a bounding box determination module 4553, a cropping region determination module 4554 and a cropping module 4555. These modules are logical, and thus can be combined or further split according to the implemented functions. The functions of the various modules will be described below.

[0066] In some other embodiments, the video shake processing apparatus provided by the embodiments of the present application can be implemented in a hardware manner. As an example, the apparatus provided by the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to perform the video shake processing method provided by the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can use one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Programmable Logic Devices (PLDs), Complex Programmable Logic Devices (CPLDs), Field-Programmable Gate Arrays (FPGAs) or other electronic elements.

[0067] It should be noted that the examples of the video shake processing method below are illustrated by the scenario of video stabilization when a user records a video through a terminal due to movement. Based on the understanding below, those skilled in the art can also apply the video processing method provided by the embodiments of the present application to other arbitrary scenarios, such as a vehicle-mounted video recording scenario, a drone photography scenario, a conference recording scenario, a photo burst shooting scenario, etc.

[0068] FIG. 3 is a first flowchart of the video shake processing method provided by the embodiments of the present application. The following will be described in combination with the steps shown in FIG. 3. As shown in FIG. 3, the execution subject of the video shake processing method is taken as a server as an example, and the method includes the following steps S101 to S105:

[0069] In step S101, a to-be-processed video is acquired, and a shake compensation matrix of each original image in the to-be-processed video is determined.

[0070] Here, the to-be-processed video is a video recorded by an image acquisition device, and the original image is an image directly acquired by the image acquisition device. Alternatively, the to-be-processed video can also be a video stored in the terminal, and the original image is an image in the to-be-processed video that has not been processed by the anti-shake algorithm. The to-be-processed video includes multiple original images. For example, a user records a video by using a camera of the terminal, obtains a to-be-processed video of 10 seconds, and the frame rate of the video recording is 60 frames per second, and thus the to-be-processed video includes 600 original images. The shake compensation matrix is a matrix used to describe a spatial transformation relationship of the original image, and the spatial transformation relationship reflects a movement of an image obtained after the original image is compensated by the shake compensation matrix, relative to the original image in space, including but not limited to translation, rotation, scaling, and the like. The shake compensation matrix is a 3*3 matrix. The method for determining the shake compensation matrix of the original image is not specifically limited in this embodiment of the present application, and can be referred to related technologies of video anti-shake. For example, a method such as an optical flow method, a feature matching method, a block matching method, a gray projection method, or a gyroscope estimation method can be used to perform motion estimation on the original images of adjacent frames in the to-be-processed video, to obtain a global motion model. Then, motion filtering is performed, for example, the global motion model is filtered based on a Kalman filtering algorithm, and a high-frequency shake vector is separated, that is, the shake compensation matrix.

[0071] In step S102, the original image is compensated based on the shake compensation matrix of the original image, to obtain a compensated image and vertex information of the compensated image.

[0072] Here, for any one original image, the compensated image of the original image is an image obtained by transforming the original image by the shake compensation matrix. The coordinates of each pixel in the original image can be transformed by the shake compensation matrix to obtain the coordinates of a compensated pixel. Each pixel in the original image is moved to the coordinates of the corresponding compensated pixel to obtain the compensated image. The vertex information of the compensated image includes the coordinates of each vertex of the four vertices in the compensated image. In this embodiment of the present application, the coordinates of each vertex of the four vertices in the original image are transformed by the shake compensation matrix to obtain the vertex information of the compensated image. The original image is moved according to the vertex information of the compensated image to obtain the compensated image.

[0073] In some embodiments, referring to FIG. 4, the step S102 of compensating the original image based on the shake compensation matrix of the original image to obtain the compensated image and the vertex information of the compensated image can be implemented by the following steps S1021 to S1024:

[0074] In step S1021, pixel coordinates of each second vertex of the original image in the image coordinate system are obtained.

[0075] Here, the original image can include four second vertices. The image coordinate system of the original image is a two-dimensional coordinate system constructed with the second vertex at the left upper corner of the original image as the origin (0, 0), the extension of the left upper corner second vertex in the width as the horizontal axis u, and the extension of the left upper corner second vertex in the height as the vertical axis v. The pixel coordinate is a coordinate used to represent the position of each pixel in the original image. The pixel coordinate is usually represented by a horizontal coordinate u and a vertical coordinate v. The horizontal coordinate u and the vertical coordinate v are both integers. For example, if the width of the original image is 100 px, the pixel coordinate of the second vertex at the right upper corner of the original image is (100, 0).

[0076] In step S1022, the homogeneous coordinate corresponding to the pixel coordinate of each second vertex is determined.

[0077] Here, the homogeneous coordinate is a coordinate used to represent a pixel in the original image in a three-dimensional space. The homogeneous coordinate of each pixel in the original image includes a horizontal coordinate u, a vertical coordinate v, and a scalar. For any second vertex, the horizontal coordinate u in the homogeneous coordinate of the second vertex is equal to the horizontal coordinate u in the pixel coordinate, the vertical coordinate v in the homogeneous coordinate is equal to the vertical coordinate v in the pixel coordinate, and the scalar is 1. The homogeneous coordinate of the second vertex is (u, v, 1). For example, the pixel coordinate of the second vertex at the right upper corner of the original image is (100, 0), and the homogeneous coordinate of the second vertex is (100, 0, 1).

[0078] In step S1023, the homogeneous coordinate of each second vertex is transformed based on the dithering compensation matrix to obtain the vertex information of the compensated image.

[0079] Here, the compensated image includes four first vertices. The vertex information of the compensated image includes the first pixel coordinates of the first vertices of the compensated image. The first pixel coordinates are also represented in the form of homogeneous coordinates. The homogeneous coordinate can be regarded as a three-dimensional vector. For any second vertex, the product of the dithering compensation matrix and the transpose of the homogeneous coordinate of the second vertex is determined, and the transpose of the product is obtained to obtain the first pixel coordinate of the first vertex corresponding to the second vertex. The first pixel coordinate of the first vertex can satisfy the following formula (1).

[0080] where p' n is the first pixel coordinate of the nth first vertex, K is the dithering compensation matrix, and n 3×3 is the transpose of the homogeneous coordinate of the nth second vertex.

[0081] In step S1024, the original image is moved based on the vertex information of the compensated image to obtain the compensated image.

[0082] ​​Here, after obtaining the vertex information of the compensated image, for any second vertex of the original image, the second vertex can be moved to the first pixel coordinate corresponding to the second vertex. Moving the four second vertices to the corresponding first pixel coordinates respectively, that is, obtaining the compensated image composed of the four first vertices.

[0083] The embodiment of the present application compensates the four second vertices in the original image through the jitter compensation matrix to obtain the vertex information of the compensated image, and can realize the movement of the original image to obtain the compensated image based on the vertex information, without position transformation for each pixel, thereby saving the calculation resources and improving the image compensation efficiency.

[0084] In step S103, the bounding box of the compensated image is determined based on the vertex information.

[0085] The two adjacent edges of the bounding box are parallel to the two coordinate axes of the image coordinate system of the original image.

[0086] Here, the bounding box is the smallest box that can enclose the four first vertices in the compensated image. That is, the four first vertices in the compensated image are located at one edge of the bounding box. The bounding box includes two groups of parallel edges, one group of parallel edges (the width of the bounding box) is parallel to the horizontal axis u in the image coordinate system of the original image, and the other group of parallel edges (the height of the bounding box) is parallel to the vertical axis v in the image coordinate system of the original image. The bounding box of the compensated image can be determined based on the first pixel coordinates of the four first vertices in the compensated image.

[0087] In some embodiments, the vertex information includes the first pixel coordinates of each first vertex of the compensated image. Referring to FIG. 5, FIG. 5 shows that the bounding box of the compensated image can be determined based on the vertex information in step S103, which can be implemented through the following steps S1031 to S1032:

[0088] In step S1031, the first coordinate point and the second coordinate point are determined based on the first pixel coordinates of each first vertex.

[0089] Here, the first coordinate point and the second coordinate point are opposite vertices. The first coordinate point is the vertex of the upper left corner of the bounding box. The second coordinate point is the vertex of the lower right corner of the bounding box. The horizontal coordinate values of the first coordinate point and the second coordinate point can be determined based on the horizontal coordinate values in the first pixel coordinates of each first vertex, and the vertical coordinate values of the first coordinate point and the second coordinate point can be determined based on the vertical coordinate values in the first pixel coordinates of each first vertex.

[0090] In the embodiments of this application, the first coordinate point and the second coordinate point can be determined based on the first pixel coordinates of each first vertex in step S1031. The first coordinate point and the second coordinate point can be determined in the following manner. First, the minimum horizontal coordinate value in the first pixel coordinates of each first vertex is determined as the first horizontal coordinate value, and the minimum vertical coordinate value in the first pixel coordinates of each first vertex is determined as the first vertical coordinate value. Then, the maximum horizontal coordinate value in the first pixel coordinates of each first vertex is determined as the second horizontal coordinate value, and the maximum vertical coordinate value in the first pixel coordinates of each first vertex is determined as the second vertical coordinate value. Finally, the first coordinate point is determined based on the first horizontal coordinate value and the first vertical coordinate value, and the second coordinate point is determined based on the second horizontal coordinate value and the second vertical coordinate value.

[0091] Here, the first coordinate point and the second coordinate point satisfy the following formula (2).

[0092] wherein p tl [0] is the first horizontal coordinate value of the first coordinate point, p tl [1] is the first vertical coordinate value of the first coordinate point, p n [0] is the horizontal coordinate value in the first pixel coordinates of the nth first vertex, p n [1] is the vertical coordinate value in the first pixel coordinates of the nth first vertex, p br [0] is the second horizontal coordinate value of the second coordinate point, p br [1] is the second vertical coordinate value of the second coordinate point.

[0093] For example, the first pixel coordinates of the four first vertices of the compensated image are (20, 20, 1), (30, 120, 1), (120, 20, 1) and (130, 120, 1). The minimum horizontal coordinate value in the first pixel coordinates of each first vertex is 20, which is determined as the first horizontal coordinate value. The minimum vertical coordinate value in the first pixel coordinates of each first vertex is 20, which is determined as the first vertical coordinate value. The maximum horizontal coordinate value in the first pixel coordinates of each first vertex is 130, which is determined as the second horizontal coordinate value. The maximum vertical coordinate value in the first pixel coordinates of each first vertex is 120, which is determined as the second vertical coordinate value. The first coordinate point is (20, 20), and the second coordinate point is (130, 120).

[0094] In the embodiments of this application, the first coordinate point is determined based on the minimum horizontal coordinate value and the minimum vertical coordinate value in the first pixel coordinates of each first vertex in the compensated image, and the first coordinate point is determined based on the maximum horizontal coordinate value and the maximum vertical coordinate value, so that the smallest bounding box that can enclose each first vertex of the compensated image is quickly determined, and the accuracy of the to-be-clipped region determined based on the bounding box is improved.

[0095] In step S1032, the first coordinate point and the second coordinate point are taken as diagonal vertices, and a bounding box of the compensated image is constructed based on the diagonal vertices.

[0096] Here, the first coordinate point can be taken as a top-left corner vertex of the bounding box, and the second coordinate point can be taken as a bottom-right corner vertex of the bounding box. An extension line is constructed from the first coordinate point to the direction of the horizontal axis u and the direction of the vertical axis v in the image coordinate system of the original image, respectively, and an extension line is constructed from the second coordinate point to the opposite direction of the horizontal axis u and the opposite direction of the vertical axis v in the image coordinate system of the original image, respectively. When the two extension lines intersect, the bounding box of the compensated image is constructed.

[0097] In the embodiments of the present application, the first coordinate point is determined based on the minimum horizontal coordinate value and the minimum vertical coordinate value in the first pixel coordinates of each first vertex in the compensated image, and the first coordinate point is determined based on the maximum horizontal coordinate value and the maximum vertical coordinate value, so that the smallest bounding box that can enclose each first vertex of the compensated image is quickly determined, and the accuracy of the to-be-clipped region determined based on the bounding box is improved.

[0098] In step S104, the to-be-displayed image is determined based on the boundary information of the original image and the compensated image, and the to-be-clipped region in the to-be-displayed image is determined based on the bounding box.

[0099] Here, the boundary information of the original image is determined as the boundary information of the to-be-displayed image, and the part of the compensated image within the boundary information of the to-be-displayed image is normally displayed in the to-be-displayed image. The to-be-displayed image also includes blank regions where the compensated image does not exist, and these blank regions are filled with black, that is, black edges. For any pixel in the to-be-displayed image, if the coordinate of the pixel exists in the first pixel coordinates of the compensated image, the pixel is displayed as the pixel at the corresponding position in the compensated image; or if the coordinate of the pixel does not exist in the first pixel coordinates of the compensated image, the pixel is displayed in black. The to-be-clipped region in the to-be-displayed image does not contain pixels in the compensated image, and the pixels in the to-be-clipped region are all black. The to-be-clipped region in the to-be-displayed image can be determined based on the coordinates of the vertices of the bounding box.

[0100] In some embodiments, referring to FIG. 6, the to-be-clipped region in the to-be-displayed image based on the bounding box in step S104 can be implemented through the following steps S1041 to S1046.

[0101] In step S1041, the maximum value of the first horizontal coordinate value and the horizontal coordinate value of the origin of the image coordinate system is determined as the initial clipping pixel number corresponding to the first edge in the to-be-displayed image.

[0102] The to-be-displayed image includes four edges: a first edge, a second edge, a third edge, and a fourth edge. The first edge and the third edge are parallel to the vertical axis v of the image coordinate system, that is, the height of the to-be-displayed image. The horizontal coordinate value of the first edge is less than that of the third edge. The second edge and the fourth edge are parallel to the horizontal axis u of the image coordinate system, that is, the width of the to-be-displayed image. The vertical coordinate value of the second edge is less than that of the fourth edge. For example, the upper left corner of the to-be-displayed image is taken as a starting point, and the four edges are sequentially set as the second edge, the third edge, the fourth edge, and the first edge in a clockwise direction. The initial number of pixels corresponding to the first edge is used to represent the number of pixel columns cropped from the first edge in the direction of the horizontal axis u of the image coordinate system. If the horizontal coordinate value of the origin of the image coordinate system is 0, the maximum value between the first horizontal coordinate value of the first coordinate point of the bounding box and 0 is determined as the initial number of pixels corresponding to the first edge.

[0103] For example, the first coordinate point of the bounding box is (20, 20), and the second coordinate point is (130, 120). The first horizontal coordinate value of the first coordinate point is 20, and thus the initial number of pixels corresponding to the first edge is 20. That is, 20 columns of pixels need to be cropped from the first edge in the direction of the horizontal axis u of the image coordinate system.

[0104] In step S1042, the maximum value between the first vertical coordinate value and the vertical coordinate value of the origin of the image coordinate system is determined as the initial number of pixels corresponding to the second edge in the to-be-displayed image.

[0105] The initial number of pixels corresponding to the second edge is used to represent the number of pixel rows cropped from the second edge in the direction of the vertical axis v of the image coordinate system. The vertical coordinate value of the origin of the image coordinate system is 0. Thus, the maximum value between the first vertical coordinate value of the first coordinate point of the bounding box and 0 is determined as the initial number of pixels corresponding to the second edge. For example, the first coordinate point of the bounding box is (20, 20), and the second coordinate point is (130, 120). The first vertical coordinate value of the first coordinate point is 20, and thus the initial number of pixels corresponding to the second edge is 20. That is, 20 rows of pixels need to be cropped from the second edge in the direction of the vertical axis v of the image coordinate system.

[0106] In step S1043, a first difference value between the width of the compensated image and the second horizontal coordinate value is obtained, and the maximum value between the first difference value and the horizontal coordinate value of the origin is determined as the initial number of pixels corresponding to the third edge in the to-be-displayed image.

[0107] Here, the initial cropping pixel number corresponding to the third edge is used to represent the number of pixel columns cropped from the third edge in the opposite direction of the horizontal axis u of the image coordinate system. The maximum value between the first difference value and 0 is determined as the initial cropping pixel number corresponding to the third edge. The width of the compensated image is the same as the width of the image to be displayed and the width of the original image. For example, the first coordinate point of the bounding box is (20, 20) and the second coordinate point is (130, 120). The second horizontal coordinate value of the second coordinate point is 130, and if the width of the compensated image is 100, the first difference value is 30, and the initial cropping pixel number corresponding to the third edge is 30. That is, 30 columns of pixels need to be cropped from the third edge in the opposite direction of the horizontal axis u of the image coordinate system.

[0108] In step S1044, the second difference value between the height of the compensated image and the second vertical coordinate value is obtained, and the maximum value between the second difference value and the maximum value of the origin vertical coordinate value is determined as the initial cropping pixel number corresponding to the fourth edge in the image to be displayed.

[0109] Here, the initial cropping pixel number corresponding to the fourth edge is used to represent the number of pixel rows cropped from the fourth edge in the opposite direction of the vertical axis v of the image coordinate system. Then the maximum value between the second difference value and 0 is determined as the initial cropping pixel number corresponding to the fourth edge. The height of the compensated image is the same as the height of the image to be displayed and the height of the original image. For example, the first coordinate point of the bounding box is (20, 20) and the second coordinate point is (130, 120). The second vertical coordinate value of the second coordinate point is 120, and if the height of the compensated image is 100, the second difference value is 20, and the initial cropping pixel number corresponding to the fourth edge is 20. That is, 20 rows of pixels need to be cropped from the fourth edge in the opposite direction of the vertical axis v of the image coordinate system.

[0110] In step S1045, the initial cropping pixel number corresponding to each edge is smoothed to obtain the cropping pixel number corresponding to each edge.

[0111] Here, since if each frame of the image to be displayed is directly cropped using the cropping region constituted by the initial cropping pixel number, the size of the imaging region of the cropped image may change dramatically, the initial cropping pixel number needs to be smoothed to obtain the actual cropping pixel number corresponding to each edge. The definition of the cropping pixel number can refer to the initial cropping pixel number, which will not be repeated here. The initial cropping pixel number corresponding to each edge can be smoothed by using a Kalman filtering algorithm to obtain the cropping pixel number corresponding to each edge.

[0112] In some embodiments, the smoothing processing on the initial number of cropped pixels corresponding to each edge in step S1045 is implemented in the following manner: first, the first covariance matrix and the second covariance matrix of the i-th frame of the to-be-displayed image are obtained; then, for each edge of the i-th frame of the to-be-displayed image, an observation vector is determined based on the initial number of cropped pixels corresponding to the edge and a change speed, the change speed being the difference between the initial number of cropped pixels corresponding to the edge of the i-th frame of the to-be-displayed image and the initial number of cropped pixels corresponding to the edge of the (i-1)-th frame of the to-be-displayed image; finally, the number of cropped pixels corresponding to the edge is determined based on the first covariance matrix, the second covariance matrix, the observation vector and the number of cropped pixels of the (i-1)-th frame of the to-be-displayed image; wherein i is an integer greater than 1, and the number of cropped pixels corresponding to the edge of the 1st frame of the to-be-displayed image is the initial number of cropped pixels.

[0113] Here, the first covariance matrix of the i-th frame of the to-be-displayed image is used to represent the change disturbance noise of the observation vector of the edge in the i-th frame of the to-be-displayed image and the observation vector of the corresponding edge in the (i-1)-th frame of the to-be-displayed image. That is, the first covariance matrix of the i-th frame of the to-be-displayed image is used to represent the disturbance noise of the change speed of the edge in the i-th frame of the to-be-displayed image. The second covariance matrix of the i-th frame of the to-be-displayed image is used to represent the disturbance noise of predicting the number of cropped pixels of the i-th frame of the to-be-displayed image based on the number of cropped pixels of the (i-1)-th frame of the to-be-displayed image. The first covariance matrix can be obtained based on measurement or experience. The first covariance matrix of each edge can be the same or different. The second covariance matrix of each edge can be the same or different.

[0114] For example, for the first edge a of the 2nd frame of the to-be-displayed image, the initial number of cropped pixels corresponding to the first edge a is 20, and the initial number of cropped pixels corresponding to the first edge a of the 1st frame of the to-be-displayed image is 10. Then the change speed of the 2nd frame of the to-be-displayed image is 10, and the vector obtained by splicing the initial number of cropped pixels 20 corresponding to the first edge a and the change speed 10 is transposed to obtain the observation vector z2 of the first edge a in the 2nd frame of the to-be-displayed image. T The number of cropped pixels corresponding to the first edge a is determined based on the first covariance matrix, the second covariance matrix of the 2nd frame of the to-be-displayed image, the observation vector of the first edge a and the initial number of cropped pixels of the first edge a in the 1st frame of the to-be-displayed image.

[0115] The embodiments of the present application determine the number of cropped pixels of the current frame by the number of cropped pixels of the last frame of the to-be-displayed image and the observation vector of the current frame of the to-be-displayed image, which can smooth the initial number of cropped pixels, avoid the imaging area jumping problem between adjacent frames, and improve the stability of the video after the final anti-shake processing.

[0116] In some embodiments, the first and second covariance matrices of the i-th frame of the image to be displayed can be obtained by the following method: first, obtaining a preset first covariance matrix of the i-th frame of the image to be displayed; and determining a second covariance prediction matrix of the i-1-th frame of the image to be displayed based on the first and second covariance matrices of the i-1-th frame of the image to be displayed; then, determining the second covariance matrix of the i-th frame of the image to be displayed based on a preset coefficient matrix and the second covariance prediction matrix of the i-1-th frame of the image to be displayed; and the second covariance matrix of the 1-th frame of the image to be displayed is a unit matrix.

[0117] Here, the preset coefficient matrix F includes a time interval At of the original images of adjacent frames in the video to be processed. The preset coefficient matrix F is determined according to the time interval At of the original images of adjacent frames in the video to be processed. The first and second covariance matrices of the i-1-th frame of the image to be displayed can be added, and the obtained sum can be operated by an inverse operation to obtain a covariance inverse matrix. The second covariance matrix of the i-1-th frame of the image to be displayed is multiplied by the covariance inverse matrix, and then multiplied by the second covariance matrix of the i-1-th frame of the image to be displayed to obtain a product matrix. The difference between the second covariance matrix of the i-1-th frame of the image to be displayed and the product matrix is determined as the second covariance prediction matrix of the i-1-th frame of the image to be displayed. The coefficient matrix F, the second covariance prediction matrix of the i-1-th frame of the image to be displayed, and the transpose of the coefficient matrix F are multiplied to obtain the second covariance matrix of the i-th frame of the image to be displayed.

[0118] By iteratively updating the second covariance matrix of each frame of the image to be displayed, the application embodiment can eliminate the influence of disturbance noise when calculating the number of cropped pixels, thereby improving the accuracy of the number of cropped pixels and improving the accuracy of processing the black border in the video.

[0119] In some embodiments, the number of cropped pixels corresponding to the edge can be determined based on the first and second covariance matrices, the observation vector, and the number of cropped pixels of the i-1-th frame of the image to be displayed by the following method: first, determining the cropped state vector of the i-1-th frame of the image to be displayed based on the number of cropped pixels of the i-1-th frame of the image to be displayed and the change speed; and determining the cropped state prediction vector of the i-th frame of the image to be displayed based on the preset coefficient matrix and the cropped state vector of the i-1-th frame of the image to be displayed; then, determining the difference between the observation vector of the i-th frame of the image to be displayed and the cropped state prediction vector as a target vector; and determining the inverse matrix of the sum of the first and second covariance matrices as a target matrix; finally, adding the product of the first covariance matrix, the target matrix, and the target vector to the cropped state prediction vector of the i-th frame of the image to be displayed to obtain the cropped state vector of the edge corresponding to the edge in the i-th frame of the image to be displayed; and obtaining the number of cropped pixels corresponding to the edge from the corresponding cropped state vector of the i-th frame of the image to be displayed.

[0120] Here, the rate of change in the cropping state vector of the (i-1)th frame of the image to be displayed is the difference between the number of cropped pixels corresponding to the edge in the (i-1)th frame and the number of cropped pixels corresponding to the edge in the (i-2)th frame. For example, for the first edge 'a' of the third frame of the image to be displayed, the number of cropped pixels corresponding to the first edge 'a' is 10, the number of cropped pixels corresponding to the first edge 'a' of the second frame is 9, and the number of cropped pixels corresponding to the first edge 'a' of the first frame is 10, so the rate of change is 1. Transposing the vector obtained by concatenating the number of cropped pixels (9) corresponding to the first edge 'a' of the second frame and the rate of change (1), we obtain the cropping state vector x′2 = [9,1] for the first edge 'a' in the second frame. T The preset coefficient matrix F is combined with the cropping state vector x′2=[9,1] of the first side a in the second frame image to be displayed. T The cropping state prediction vector x3 of the first edge 'a' in the third frame of the image to be displayed is determined by multiplying the observation vector and the cropping state prediction vector of the third frame of the image to be displayed. The target vector is determined by the difference between the observation vector and the cropping state prediction vector of the third frame of the image to be displayed. The inverse matrix of the sum of the first covariance matrix and the second covariance matrix of the third frame of the image to be displayed is determined as the target matrix. The product of the first covariance matrix, the target matrix, and the target vector of the third frame of the image to be displayed is summed with the cropping state prediction vector x3 of the third frame of the image to be displayed to obtain the cropping state vector of the edge corresponding to the first edge 'a' in the third frame of the image to be displayed. The cropping state vector of the edge corresponding to the first edge 'a' in the third frame of the image to be displayed includes the number of cropped pixels corresponding to the first edge 'a', and the number of cropped pixels can be directly obtained from the cropping state vector.

[0121] This application embodiment smooths the initial number of cropped pixels for each frame of the image to be displayed to obtain the actual number of cropped pixels, avoiding the problem of image area jumps between adjacent frames and improving the stability of the final video after image stabilization.

[0122] Step S1046: Determine the cropping area in the image to be displayed based on the number of cropping pixels corresponding to each edge.

[0123] Here, the number of cropped pixels corresponding to the first edge is used to represent the number of pixel columns cropped in the direction of the horizontal axis u of the image coordinate system from the first edge. The number of cropped pixels corresponding to the second edge is used to represent the number of pixel rows cropped in the direction of the vertical axis v of the image coordinate system from the second edge. The number of cropped pixels corresponding to the third edge is used to represent the number of pixel columns cropped in the opposite direction of the horizontal axis u of the image coordinate system from the third edge. The number of cropped pixels corresponding to the fourth edge is used to represent the number of pixel rows cropped in the opposite direction of the vertical axis v of the image coordinate system from the fourth edge. Therefore, for each edge, the edge and the number of cropped pixels corresponding to the edge form a sub-region, and the sub-regions of each edge are superimposed to obtain the to-be-cropped region in the to-be-displayed image. For example, the to-be-displayed image has a width and a height of 100px, the number of cropped pixels corresponding to the first edge is 20, the number of cropped pixels corresponding to the second edge is 0, the number of cropped pixels corresponding to the third edge is 0, and the number of cropped pixels corresponding to the fourth edge is 0. The sub-region corresponding to the first edge is the region between the leftmost pixel and the 20th pixel from the left in the to-be-displayed image. The to-be-cropped region in the to-be-displayed image is the region between the leftmost pixel and the 20th pixel from the left in the to-be-displayed image.

[0124] The number of cropped pixels corresponding to each edge obtained by smoothing the initial number of cropped pixels is used to determine the to-be-cropped region in the to-be-displayed image, thereby avoiding the problem of imaging region jumping between adjacent frames and improving the stability of the final video after the anti-shake processing.

[0125] In step S105, each frame of the to-be-displayed image is cropped according to the corresponding to-be-cropped region to obtain the video after the anti-shake processing.

[0126] Here, the to-be-cropped region in the to-be-displayed image can be cropped to obtain the remaining cropped image. The size of the to-be-displayed image is obtained, and the cropped image is enlarged to the size of the to-be-displayed image in a proportional manner to obtain the image after the anti-shake processing. Each frame of the image after the anti-shake processing is sorted in chronological order to obtain the video after the anti-shake processing.

[0127] In the processing of video anti-shake, the original image in the to-be-processed video can be first processed through a shake compensation matrix transformation to obtain a compensated image. A to-be-displayed image is determined based on the boundary information of the original image and the compensated image. Since the position range of the displayable video, i.e., the boundary information of the original image, is fixed and unchanged, the positions of the compensated image and the original image are different, and therefore only part of the compensated image is in the to-be-displayed image, and the remaining part is a black border. The bounding box of the compensated image can be determined based on the vertex information of the compensated image, and two adjacent edges of the bounding box are parallel to two coordinate axes of the image coordinate system of the original image. The region outside the bounding box in the to-be-displayed image does not have the compensated image, and therefore the to-be-clipped region in the to-be-displayed image can be determined based on the bounding box. Finally, each frame of the to-be-displayed image is clipped according to the corresponding to-be-clipped region to obtain the anti-shake-processed video. The embodiments of the present application automatically calculate the to-be-clipped region in the to-be-displayed image based on the bounding box of the compensated image, realize the to-be-displayed image corresponding to different original images, can obtain different to-be-clipped regions, and improve the flexibility and accuracy of black border processing.

[0128] FIG. 7 is a fifth flowchart of a video shake processing method provided by the embodiments of the present application. As shown in FIG. 7, the method includes the following steps S201 to S210:

[0129] Step S201, the terminal receives an interactive operation of the user.

[0130] Here, the interactive operation can be clicking to start shooting, clicking to perform anti-shake processing operation, etc. The user can perform any kind of interactive operation through the terminal in the client of the video shake processing application, for example, clicking to start the image acquisition device of the terminal, and obtaining the to-be-processed video through the image acquisition device.

[0131] Step S202, the terminal generates a video shake processing request in response to the interactive operation.

[0132] When the terminal receives the interactive operation of the user, the to-be-processed video is acquired and encapsulated into the video shake processing request in response to the interactive operation. The video shake processing request is used to request the server to perform anti-shake processing on the to-be-processed video.

[0133] Step S203, the terminal sends the video shake processing request to the server.

[0134] Step S204, the server acquires the to-be-processed video and determines the shake compensation matrix of each frame of original image in the to-be-processed video in response to the video shake processing request.

[0135] Here, the specific process of acquiring the to-be-processed video and determining the shake compensation matrix of each original image frame in the to-be-processed video can refer to step S101 in the above embodiment, which will not be repeated here.

[0136] In step S205, the server performs compensation processing on the original image based on the shake compensation matrix of the original image to obtain a compensated image and vertex information of the compensated image.

[0137] Here, the specific process of performing compensation processing on the original image based on the shake compensation matrix of the original image to obtain a compensated image and vertex information of the compensated image can refer to step S102 in the above embodiment, which will not be repeated here.

[0138] In step S206, the server determines a bounding box of the compensated image based on the vertex information.

[0139] Two adjacent edges of the bounding box are parallel to two coordinate axes of the image coordinate system of the original image.

[0140] Here, the specific process of determining a bounding box of the compensated image based on the vertex information can refer to step S103 in the above embodiment, which will not be repeated here.

[0141] In step S207, the server determines a to-be-displayed image based on the boundary information of the original image and the compensated image, and determines a to-be-cropped region in the to-be-displayed image based on the bounding box.

[0142] Here, the specific process of determining a to-be-displayed image based on the boundary information of the original image and the compensated image, and determining a to-be-cropped region in the to-be-displayed image based on the bounding box can refer to step S104 in the above embodiment, which will not be repeated here.

[0143] In step S208, the server crops each frame of the to-be-displayed image according to the corresponding to-be-cropped region to obtain a video after shake reduction processing.

[0144] Here, the specific process of cropping each frame of the to-be-displayed image according to the corresponding to-be-cropped region to obtain a video after shake reduction processing can refer to step S105 in the above embodiment, which will not be repeated here.

[0145] In step S209, the server sends the video after shake reduction processing to the terminal.

[0146] In step S210, the terminal displays the video after shake reduction processing on the current interface.

[0147] In the processing of video anti-shake, the original image in the video to be processed can be first processed by a shake compensation matrix to obtain a compensated image. The image to be displayed is determined based on the boundary information of the original image and the compensated image. Since the position range of the displayable video, i.e., the boundary information of the original image, is fixed, the position of the compensated image is different from that of the original image, and thus only part of the compensated image is in the image to be displayed, and the rest is black edges. The bounding box of the compensated image can be determined based on the vertex information of the compensated image, and two adjacent edges of the bounding box are parallel to two coordinate axes of the image coordinate system of the original image. The region to be cropped in the image to be displayed can be determined based on the bounding box since there is no compensated image outside the bounding box. Finally, each frame of the image to be displayed is cropped according to the corresponding region to be cropped to obtain the video after anti-shake processing. The embodiments of the present application automatically calculate the region to be cropped in the image to be displayed based on the bounding box of the compensated image, realize the image to be displayed corresponding to different original images, can obtain different regions to be cropped, and improve the flexibility and accuracy of black edge processing.

[0148] In the following, an exemplary application of the embodiments of the present application in an actual application scenario will be described.

[0149] To solve the problem caused by the default cropping ratio in the intelligent anti-shake algorithm, the embodiments of the present application provide a shake processing method for video, which is an automatic black edge removal video anti-shake algorithm, i.e., an automatic black edge removal method according to the camera motion amplitude. The method calculates the size of the black edge to be additionally introduced by using the motion compensation matrix obtained by the anti-shake algorithm, smoothes the size of the black edge by using the Kalman filtering algorithm, and finally realizes automatic black edge removal by cropping the smoothed black edge size. The method obtains the size of the black edge by matrix operation, and thus does not significantly increase the processing time of the anti-shake algorithm.

[0150] The video shot by a stable camera is continuous and smooth, while the video shot by a camera with shake often presents an irregular and discontinuous state, and thus eliminating camera shake is to eliminate the discontinuous change between adjacent image frames in the video. First, the global motion between the front and rear frames of the video needs to be estimated, and the global motion includes the normal active motion of the camera during shooting, which must be retained. The low-frequency normal scanning and high-frequency random shake of the camera need to be separated from the global motion, and then each frame of the video image is inversely compensated and corrected according to the random shake component.

[0151] FIG. 8 is a flowchart of an image anti-shake system according to an embodiment of the present application. In step S301, motion estimation. A global motion model between the front and back frames of the shaking video is obtained, which contains low-frequency scanning and high-frequency shaking. Common motion estimation methods include optical flow method, feature matching method, block matching method, gray projection method, gyroscope estimation method, etc. In step S302, motion filtering. Filtering is performed according to the estimated motion model to eliminate high-frequency shaking. In step S303, motion compensation. The direction and size of the separated high-frequency shaking vector are used to perform reverse transformation on the video image frame, thereby constructing a stable image sequence. In step S304, cropping and enlarging. The stable image sequence is cropped and enlarged to the image size in the original video to obtain a stable video. The video processing method provided by the embodiment of the present application is mainly applied to a certain frame image in step S304. After motion estimation and motion filtering, the obtained global motion compensation matrix (corresponding to the shaking compensation matrix in the above embodiment) is K 3×3 After automatically calculating the black border size and smoothing the black border size, the steps of cropping and enlarging the image based on the smoothed black border size are as follows:

[0152] For the original graph (corresponding to the original image in the video to be processed in the above embodiment), the pixel homogeneous coordinates of the four corner points (corresponding to the second vertex in the above embodiment) of the original graph in the original graph coordinate system (corresponding to the image coordinate system in the above embodiment) are p0, p1, p2, and p3, respectively. n n n , 1] (n = 0, 1, 2, 3), where u n is the horizontal axis coordinate of the nth corner point in the original graph coordinate system, v n is the vertical axis coordinate of the nth corner point in the original graph coordinate system, and u n and v n are integers). After the motion compensation matrix K 3×3 , the pixel homogeneous coordinates of the four corner points in the rotated image coordinate system can satisfy the following formula (1).

[0153] , where p' is the pixel homogeneous coordinates of the nth corner point of the rotated image, K n is the motion compensation matrix, 3×3 is the transpose of the pixel homogeneous coordinates of the nth corner point of the original graph.

[0154] Since the motion compensation matrix K 3×3 ​​​The obtained rotated image has its side not parallel to the image coordinate system, so a minimum frame enclosing the four corner points of the rotated image needs to be calculated. A minimum frame b (corresponding to the enclosing frame in the above embodiment) enclosing the four corner points of the rotated image is calculated. The width and height of the minimum frame b are parallel to the width and height of the rotated image coordinate system, respectively, and are called the minimum parallel frame. The coordinates of the top-left corner point p tl (corresponding to the first coordinate point in the above embodiment) and the bottom-right corner point p br (corresponding to the second coordinate point in the above embodiment) in the rotated image coordinate system satisfy formula (2).

[0155] where p tl [0] is the horizontal coordinate value of the top-left corner point p tl , p tl [1] is the vertical coordinate value of the top-left corner point p tl , p' n [0] is the horizontal coordinate value in the pixel homogeneous coordinates of the nth vertex in the rotated image, p' n [1] is the vertical coordinate value in the pixel homogeneous coordinates of the nth vertex in the rotated image, p br [0] is the horizontal coordinate value of the bottom-right corner point p br , p br [1] is the vertical coordinate value of the bottom-right corner point p br .

[0156] According to formula (2), in order to completely remove the black edges caused by the shaking of the frame image, the left side of the rotated image needs to be cropped by max(p tl [0], 0) pixels, the upper side needs to be cropped by max(p tl [1], 0) pixels, the right side needs to be cropped by max(img_width-p br [0], 0) pixels, and the lower side needs to be cropped by max(img_height-p br [1], 0) pixels, where img_width represents the image width and img_height represents the image height. However, if this method is used to crop every frame, the size of the imaging area of the image may change dramatically, so the number of actually cropped pixels needs to be smoothed using Kalman filtering.

[0157] The Kalman filtering is based on the observation value of a side calculated in the kth frame (corresponding to the observation vector in the above embodiment), where represents the number of pixels that need to be cropped according to the motion compensation matrix (corresponding to the initial number of cropped pixels in the above embodiment), represents the number of pixels that need to be cropped in the kth frame compared with the number of pixels that need to be cropped in the previous frame (the observation value may have noise, and it is assumed that the covariance matrix is R k (corresponding to the first covariance matrix in the above embodiment), which can be generally obtained by measurement), to estimate the actual number of pixels that need to be cropped in the kth frame after smoothing (corresponding to the number of cropped pixels in the above embodiment) as p' k , which changes with each frame at a speed of v' k , that is, the estimated cropping state is x' k = [p' k , v' k ] T (when k = 1, x' k is equal to z k ).

[0158] In addition to estimating x' k using the observation value z k , x' k can also be estimated using the estimated value x k-1 of the previous frame to obtain x k = [p k , v k ] T When there is no disturbance, the prediction satisfies the following formula (3).

[0159] Formula (3) is expressed as a matrix to obtain formula (4).

[0160] In fact, the estimation using the previous frame is also inaccurate, that is, the estimated state x k in formula (4) should conform to a Gaussian distribution, and the covariance matrix is assumed to be P K (corresponding to the second covariance matrix in the above embodiment), which is unknown and can be initialized as an identity matrix I in the first frame and updated in each subsequent frame to satisfy formula (5). P k = FP' k-1 F T formula (5);

[0161] Combining the two Gaussian noise estimates z k and x k , an optimal estimate x' k (corresponding to the number of cropped pixels in the above embodiment) and P' k (corresponding to the second covariance prediction matrix in the above embodiment) can be obtained.

[0162] The actual number of pixels to be cut after smoothing for the four edges of top, bottom, left and right is obtained by using formula (6), max (p tl ′ [0], 0) pixels are cut from the left edge, max (p tl ′ [1], 0) pixels are cut from the top edge, max (img_width-p br ′ [0], 0) pixels are cut from the right edge, and max (img_height-p br ′ [1], 0) pixels are cut from the bottom edge. The cut image is enlarged to the size of the original image.

[0163] Referring to FIG. 9, the left picture is an original picture 401 (original image), the gray area in the right thick line frame (the intersection area of the thin line frame and the thick line frame) is the picture area of the compensated image after the original picture is rotated or translated to prevent video jitter, the remaining area in the thick line frame is a black border, and the gray area outside the thick line frame (the image area of the thin line frame except the intersection area of the thin line frame and the thick line frame) is information exceeding the picture boundary after rotation or translation. The dashed line is the area to be cut. (a) shows a case where only the original picture 401 is translated to obtain the compensated image 402-1 to remove jitter, and only the upper and left edges need to be cut along the dashed line, and the lower and right edges do not need to be cut after calculation. (b) shows a case where only the original picture 401 is rotated to obtain the compensated image 402-2 to remove jitter, and the four edges do not need to be cut after calculation. (c) shows a case where the original picture 401 needs to be translated and rotated to obtain the compensated image 402-3 to remove jitter, and only the upper and left edges need to be cut along the dashed line, and the lower and right edges do not need to be cut after calculation.

[0164] The embodiment of the present application mainly aims at the problem that black borders are introduced at the image edges when the camera motion is compensated to remove the imaging jitter caused by the camera motion. An automatic black border removal method is proposed, which uses the motion compensation matrix calculated by the anti-shake algorithm to calculate the size of the additional black border, uses a filtering algorithm to smooth the size of the black border, and finally removes the black border by cutting the smoothed black border size. The size of the black border is obtained by matrix operation, so the processing time of the anti-shake algorithm is not significantly increased. The embodiment of the present application can be applied to a scene without a gyroscope, an acceleration sensor or other sensors, and only software algorithms are used to reduce the picture blur caused by camera jitter during camera shooting, to obtain a more stable and clear picture without introducing additional black borders.

[0165] It can be understood that, in the embodiments of the present application, related data such as user information is involved, and when the embodiments of the present application are applied to specific products or technologies, user permission or consent needs to be obtained, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards.

[0166] The following continues to illustrate an exemplary structure of the implementation of the video jitter processing apparatus 455 provided by the embodiments of the present application as a software module. In some embodiments, as shown in FIG. 2, the software module stored in the video processing apparatus 455 of the memory 450 can include: a matrix determination module 4551 configured to obtain a to-be-processed video, and determine a jitter compensation matrix of each frame of original image in the to-be-processed video; an image compensation module 4552 configured to compensate the original image based on the jitter compensation matrix of the original image to obtain a compensated image and vertex information of the compensated image; a bounding box determination module 4553 configured to determine a bounding box of the compensated image based on the vertex information; two adjacent edges of the bounding box are parallel to two coordinate axes of an image coordinate system of the original image; a cropping region determination module 4554 configured to determine a to-be-displayed image based on the original image and the compensated image, and determine a to-be-cropped region in the to-be-displayed image based on the bounding box; and a cropping module 4555 configured to crop each frame of the to-be-displayed image according to the corresponding to-be-cropped region to obtain a jitter-processed video.

[0167] In some embodiments, the vertex information includes first pixel coordinates of each first vertex of the compensated image; the bounding box determination module 4553 is further configured to determine a first coordinate point and a second coordinate point based on the first pixel coordinates of each first vertex; and the first coordinate point and the second coordinate point are taken as diagonal vertices, and the bounding box of the compensated image is constructed based on the diagonal vertices.

[0168] In some embodiments, the bounding box determination module 4553 is further configured to determine a minimum horizontal coordinate value in the first pixel coordinates of each first vertex as a first horizontal coordinate value, and determine a minimum vertical coordinate value in the first pixel coordinates of each first vertex as a first vertical coordinate value; determine a maximum horizontal coordinate value in the first pixel coordinates of each first vertex as a second horizontal coordinate value, and determine a maximum vertical coordinate value in the first pixel coordinates of each first vertex as a second vertical coordinate value; determine the first coordinate point based on the first horizontal coordinate value and the first vertical coordinate value, and determine the second coordinate point based on the second horizontal coordinate value and the second vertical coordinate value.

[0169] In some embodiments, the cropping region determination module 4554 is further configured to determine a maximum value between the first horizontal coordinate value and an origin horizontal coordinate value of the image coordinate system as an initial number of cropped pixels corresponding to the first side in the to-be-displayed image; determine a maximum value between the first vertical coordinate value and an origin vertical coordinate value of the image coordinate system as an initial number of cropped pixels corresponding to the second side in the to-be-displayed image; obtain a first difference value between a width of the compensated image and the second horizontal coordinate value, and determine a maximum value between the first difference value and the origin horizontal coordinate value as an initial number of cropped pixels corresponding to the third side in the to-be-displayed image; obtain a second difference value between a height of the compensated image and the second vertical coordinate value, and determine a maximum value between the second difference value and the origin vertical coordinate value as an initial number of cropped pixels corresponding to the fourth side in the to-be-displayed image; perform smoothing processing on the initial number of cropped pixels corresponding to each side to obtain a number of cropped pixels corresponding to each side; and determine a to-be-cropped region in the to-be-displayed image based on the number of cropped pixels corresponding to each side.

[0170] In some embodiments, the cropping region determination module 4554 is further configured to obtain a first covariance matrix and a second covariance matrix of an i-th to-be-displayed image; for each side of the i-th to-be-displayed image, determine an observation vector based on an initial number of cropped pixels corresponding to the side and a change speed, the change speed being a difference between the initial number of cropped pixels corresponding to the side and an initial number of cropped pixels corresponding to the side in an (i-1)-th to-be-displayed image; and determine a number of cropped pixels corresponding to the side based on the first covariance matrix, the second covariance matrix, the observation vector, and the number of cropped pixels of the (i-1)-th to-be-displayed image; wherein i is an integer greater than 1, and the initial number of cropped pixels corresponding to the side in the first to-be-displayed image is the initial number of cropped pixels.

[0171] In some embodiments, the cropping region determination module 4554 is further configured to obtain a preset first covariance matrix of an i-th to-be-displayed image; determine a second covariance prediction matrix of an (i-1)-th to-be-displayed image based on a first covariance matrix and a second covariance matrix of the (i-1)-th to-be-displayed image; and determine a second covariance matrix of the i-th to-be-displayed image based on a preset coefficient matrix and the second covariance prediction matrix of the (i-1)-th to-be-displayed image; and the second covariance matrix of the first to-be-displayed image is a unit matrix.

[0172] In some embodiments, the cropping region determination module 4554 is further configured to determine a cropping state vector of the i-1th frame to-be-displayed image based on the number of cropped pixels and the change speed of the i-1th frame to-be-displayed image, determine a cropping state prediction vector of the i th frame to-be-displayed image based on a preset coefficient matrix and the cropping state vector of the i-1th frame to-be-displayed image, determine a difference between the observation vector of the i th frame to-be-displayed image and the cropping state prediction vector as a target vector, determine an inverse matrix of a sum of the first covariance matrix and the second covariance matrix as a target matrix, and obtain a cropping state vector corresponding to the edge in the i th frame to-be-displayed image by adding a product of the first covariance matrix, the target matrix, and the target vector to the cropping state prediction vector of the i th frame to-be-displayed image. The number of cropped pixels corresponding to the edge is obtained from the cropping state vector corresponding to the edge in the i th frame to-be-displayed image.

[0173] In some embodiments, the image compensation module 4552 is further configured to obtain pixel coordinates of each second vertex of the original image in the image coordinate system, determine homogeneous coordinates corresponding to the pixel coordinates of each second vertex, perform transformation processing on the homogeneous coordinates of each second vertex based on the jitter compensation matrix to obtain vertex information of the compensated image, and move the original image based on the vertex information of the compensated image to obtain the compensated image.

[0174] The embodiment of the present application provides a computer program product, which includes a computer program or computer executable instructions stored in a computer readable storage medium. The processor of the electronic device reads the computer executable instructions from the computer readable storage medium, and the processor executes the computer executable instructions, so that the electronic device executes the video jitter processing method provided in the embodiment of the present application.

[0175] The embodiment of the present application provides a computer readable storage medium, which stores computer executable instructions or computer programs. When the computer executable instructions or computer programs are executed by the processor, the processor will execute the video jitter processing method provided in the embodiment of the present application, for example, the video jitter processing method shown in FIG. 3.

[0176] In some embodiments, the computer readable storage medium can be RAM, ROM, flash memory, magnetic surface memory, optical disc, or CD-ROM memory; or various devices including one or any combination of the above storage devices.

[0177] In some embodiments, the computer-executable instructions can be in the form of programs, procedures, modules, scripts, or code, and can be written in any programming language, including compiled or interpreted languages, or declarative or procedural languages, and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0178] By way of example, a computer-executable instruction can be, but is not necessarily, a file in a file system, can be stored in a part of a file that holds other programs or data, e.g., one or more scripts stored in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub programs, or code portions.

[0179] By way of example, a computer-executable instruction can be, but is not necessarily, a file in a file system, can be stored in a part of a file that holds other programs or data, e.g., one or more scripts stored in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub programs, or code portions.

[0180] In summary, through the embodiments of the present application, the specific pixel values that need to be cropped on the four edges of the image are automatically calculated, which is more flexible and does not increase the amount of calculation too much. In addition, the specific pixel values that need to be cropped are smoothed by Kalman filtering, which avoids the problem of imaging area jump between adjacent frames.

[0181] The above merely illustrates the embodiments of the present application, but is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, and improvement within the spirit and scope of the present application shall be included in the protection scope of the present application.

Claims

1. A method for processing video jitter, characterized in that, The method includes: Acquire the video to be processed and determine the jitter compensation matrix for each frame of the original image in the video to be processed; The original image is compensated based on the jitter compensation matrix of the original image to obtain the compensated image and the vertex information of the compensated image. The bounding box of the compensated image is determined based on the vertex information; the two adjacent sides of the bounding box are parallel to the two coordinate axes of the image coordinate system of the original image, respectively; Based on the boundary information of the original image and the compensated image, the image to be displayed is determined, and the cropping area in the image to be displayed is determined based on the bounding box. Each frame of the image to be displayed is cropped according to the corresponding cropping area to obtain the video after image stabilization.

2. The method according to claim 1, characterized in that, The vertex information includes the first pixel coordinates of each first vertex of the compensated image; Determining the bounding box of the compensated image based on the vertex information includes: The first coordinate point and the second coordinate point are determined based on the first pixel coordinates of each first vertex; Using the first and second coordinate points as diagonal vertices, a bounding box for the compensated image is constructed based on the diagonal vertices.

3. The method according to claim 2, characterized in that, The step of determining the first coordinate point and the second coordinate point based on the first pixel coordinates of each first vertex includes: The minimum horizontal coordinate value among the first pixel coordinates of each first vertex is determined as the first horizontal coordinate value, and the minimum vertical coordinate value among the first pixel coordinates of each first vertex is determined as the first vertical coordinate value. The maximum horizontal coordinate value among the first pixel coordinates of each first vertex is determined as the second horizontal coordinate value, and the maximum vertical coordinate value among the first pixel coordinates of each first vertex is determined as the second vertical coordinate value. A first coordinate point is determined based on the first abscissa value and the first ordinate value, and a second coordinate point is determined based on the second abscissa value and the second ordinate value.

4. The method according to claim 3, characterized in that, Determining the cropping region in the image to be displayed based on the bounding box includes: The maximum value between the first abscissa value and the abscissa value of the origin of the image coordinate system is determined as the initial number of cropped pixels corresponding to the first side in the image to be displayed. The maximum value between the first ordinate value and the ordinate value of the origin of the image coordinate system is determined as the initial number of cropped pixels corresponding to the second side in the image to be displayed. Obtain the first difference between the width of the compensated image and the second horizontal coordinate value, and determine the maximum value between the first difference and the horizontal coordinate value of the origin as the initial number of cropped pixels corresponding to the third side in the image to be displayed. Obtain the second difference between the height of the compensated image and the second ordinate value, and determine the maximum value between the second difference and the ordinate value of the origin as the initial number of cropped pixels corresponding to the fourth side in the image to be displayed. The initial number of cropped pixels corresponding to each edge is smoothed to obtain the number of cropped pixels corresponding to each edge; The cropping region in the image to be displayed is determined based on the number of cropping pixels corresponding to each edge.

5. The method according to claim 4, characterized in that, The step of smoothing the initial number of cropped pixels corresponding to each edge to obtain the number of cropped pixels corresponding to each edge includes: Obtain the first covariance matrix and the second covariance matrix of the i-th frame of the image to be displayed; For each edge of the i-th frame of the image to be displayed, an observation vector is determined based on the initial number of cropped pixels corresponding to the edge and the rate of change, wherein the rate of change is the difference between the initial number of cropped pixels corresponding to the edge and the initial number of cropped pixels corresponding to the edge in the (i-1)-th frame of the image to be displayed. Based on the first covariance matrix, the second covariance matrix, the observation vector, and the number of cropped pixels in the (i-1)th frame of the image to be displayed, the number of cropped pixels corresponding to the edge is determined; Where i is an integer greater than 1, and the number of cropped pixels corresponding to the edge in the first frame of the image to be displayed is the initial number of cropped pixels.

6. The method according to claim 5, characterized in that, The step of obtaining the first covariance matrix and the second covariance matrix of the i-th frame of the image to be displayed includes: Obtain the preset first covariance matrix of the i-th frame image to be displayed; Based on the first covariance matrix and the second covariance matrix of the i-1th frame of the image to be displayed, determine the second covariance prediction matrix of the i-1th frame of the image to be displayed. Based on the preset coefficient matrix and the second covariance prediction matrix of the (i-1)th frame image to be displayed, the second covariance matrix of the i-th frame image to be displayed is determined; the second covariance matrix of the 1st frame image to be displayed is the identity matrix.

7. The method according to claim 5 or 6, characterized in that, The step of determining the number of cropped pixels corresponding to the edge based on the first covariance matrix, the second covariance matrix, the observation vector, and the number of cropped pixels in the (i-1)th frame of the image to be displayed includes: The cropping state vector of the (i-1)th frame to be displayed is determined based on the number of cropped pixels and the rate of change of the (i-1)th frame to be displayed. Based on the preset coefficient matrix and the cropping state vector of the (i-1)th frame image to be displayed, the cropping state prediction vector of the i-th frame image to be displayed is determined. The difference between the observation vector and the cropping state prediction vector of the i-th frame image to be displayed is determined as the target vector; The inverse matrix of the sum of the first covariance matrix and the second covariance matrix is ​​determined as the target matrix; The product of the first covariance matrix, the target matrix, and the target vector is summed with the cropping state prediction vector of the i-th frame image to be displayed to obtain the cropping state vector corresponding to the edge in the i-th frame image to be displayed. The number of cropped pixels corresponding to the edge is obtained from the cropping state vector corresponding to the edge in the i-th frame of the image to be displayed.

8. The method according to any one of claims 1 to 5, characterized in that, The process of compensating the original image based on the jitter compensation matrix to obtain the compensated image and the vertex information of the compensated image includes: Obtain the pixel coordinates of each second vertex of the original image in the image coordinate system; Determine the homogeneous coordinates corresponding to the pixel coordinates of each second vertex; The homogeneous coordinates of each second vertex are transformed based on the jitter compensation matrix to obtain the vertex information of the compensated image. The original image is moved based on the vertex information of the compensated image to obtain the compensated image.

9. A video jitter processing device, characterized in that, The device includes: The matrix determination module is used to acquire the video to be processed and determine the jitter compensation matrix of each frame of the original image in the video to be processed. The image compensation module is used to perform compensation processing on the original image based on the jitter compensation matrix of the original image to obtain the compensated image and the vertex information of the compensated image; A bounding box determination module is used to determine the bounding box of the compensated image based on the vertex information; the two adjacent sides of the bounding box are parallel to the two coordinate axes of the image coordinate system of the original image, respectively; The cropping region determination module is used to determine the image to be displayed based on the boundary information of the original image and the compensated image, and to determine the cropping region in the image to be displayed based on the bounding box; The cropping module is used to crop each frame of the image to be displayed according to the corresponding cropping area to obtain the video after image stabilization.

10. An electronic device, characterized in that, The electronic device includes: Memory is used to store executable instructions or computer programs. A processor, when executing computer-executable instructions or computer programs stored in the memory, implements the video jitter processing method according to any one of claims 1 to 8.

11. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by a processor, the video jitter processing method according to any one of claims 1 to 8 is implemented.

Images