Video coding method and apparatus, and electronic device and readable storage medium

Abstract

The present application belongs to the technical field of videos. Disclosed are a video coding method and apparatus, and an electronic device and a readable storage medium. The method comprises: acquiring an affine transformation matrix on the basis of image data of at least two collected video frames and first data, wherein the affine transformation matrix is used for indicating a mapping relationship between macro blocks corresponding to every two adjacent video frames among the at least two video frames, and the first data includes acceleration data and angular velocity data of an electronic device in the process of collecting the at least two video frames; determining a first object on the basis of the affine transformation matrix; and coding the at least two video frames on the basis of the first object, wherein the first object comprises at least one of the following: a coding search range of each macro block in the at least two video frames, and an internal coding frame in the at least two video frames.

Description

Video encoding method, device, electronic device and readable storage medium

[0001] Cross-references

[0002] This application claims priority to the Chinese patent application filed with the China Patent Office on June 26, 2023, with application number 202310763796.5 and titled “Video Coding Method, Device, Electronic Device and Readable Storage Medium”. The entire contents of that application are incorporated herein by reference. Technical Field

[0003] The present application belongs to the field of video technology, and specifically relates to a video encoding method, device, electronic device and readable storage medium. Background Art

[0004] Currently, when encoding multiple captured video frames, electronic devices need to perform at least one of the following: determining the encoding search range of each macroblock in the multiple video frames to identify redundant information between the multiple video frames, and then compressing the redundant information; determining the internal coding frames in the multiple video frames based on the degree of change in the image content of each video frame to increase the video frame compression rate and reduce the size of the encoded video.

[0005] Specifically, for the above-mentioned coding search range, the electronic device can determine it by traversing multiple pixel points around each pixel point in each of the above-mentioned macroblocks through a search algorithm; and for the above-mentioned internal coding frame, the electronic device can determine it by comparing the pixel changes of the macroblocks in the above-mentioned multiple video frames.

[0006] However, since a large amount of calculation is required to traverse the above-mentioned multiple pixel points and compare the above-mentioned pixel changes, the electronic device has a large computing power when determining the above-mentioned coding search range and at least one of the above-mentioned internal coding frames, which leads to high power consumption of the electronic device during the video encoding process.

[0007] Summary of the Invention

[0008] The purpose of the embodiments of the present application is to provide a video encoding method, device, electronic device and readable storage medium, which can solve the problem of high power consumption of electronic devices during video encoding.

[0009] In a first aspect, an embodiment of the present application provides a video encoding method, which includes: obtaining an affine transformation matrix based on image data of at least two captured video frames and first data, wherein the affine transformation matrix is ​​used to indicate a mapping relationship between corresponding macroblocks between each two adjacent video frames in at least two video frames, and the first data is: acceleration data and angular velocity data of an electronic device during the process of capturing at least two video frames; determining a first object based on the affine transformation matrix; encoding at least two video frames based on the first object; wherein the first object includes at least one of the following: a coding search range for each macroblock in at least two video frames; and an internal coding frame in at least two video frames.

[0010] In second aspect, an embodiment of the present application provides a video encoding device, which includes an acquisition module, a determination module and an encoding module; the acquisition module is used to acquire an affine transformation matrix based on the acquired image data of at least two video frames and first data, the affine transformation matrix being used to indicate the mapping relationship between corresponding macroblocks between each two adjacent video frames in at least two video frames, and the first data being: acceleration data and angular velocity data of the electronic device during the process of acquiring at least two video frames; the determination module is used to determine a first object based on the affine transformation matrix acquired by the acquisition module; the encoding module is used to encode at least two video frames based on the first object determined by the determination module; wherein the first object includes at least one of the following: an encoding search range for each macroblock in at least two video frames; an internal encoding frame in at least two video frames.

[0011] In a third aspect, an embodiment of the present application provides an electronic device comprising a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0012] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0013] In a fifth aspect, an embodiment of the present application provides a chip, which includes a processor and a communication interface, the communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the method described in the first aspect.

[0014] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the method described in the first aspect.

[0015] In an embodiment of the present application, an affine transformation matrix can be obtained based on image data of at least two captured video frames and first data. The affine transformation matrix is ​​used to indicate the mapping relationship between corresponding macroblocks between each two adjacent video frames in the at least two video frames. The first data is acceleration data and angular velocity data of the electronic device during the capture of the at least two video frames. A first object is determined based on the affine transformation matrix. The at least two video frames are encoded based on the first object. The first object includes at least one of the following: a coding search range for each macroblock in the at least two video frames; or an intra-coded frame in the at least two video frames. With this solution, the electronic device can directly determine the coding search range for each macroblock in the at least two captured video frames based on the affine transformation matrix indicating the mapping relationship between corresponding macroblocks in adjacent video frames, without traversing multiple pixels surrounding each pixel in each macroblock. Furthermore, the electronic device can also directly determine the intra-coded frame in the at least two video frames based on the affine transformation matrix, without comparing pixel changes in the macroblocks in the at least two video frames. In this way, when encoding the at least two video frames based on the encoding search range and at least one of the internal encoding frames, the computing power of the electronic device can be greatly reduced, and the power consumption of the electronic device can be reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] FIG1 is a schematic diagram of the diamond search method in the traditional encoding process;

[0017] FIG2 is a flowchart of a video encoding method according to an embodiment of the present application;

[0018] FIG3 is a schematic diagram of affine transformation of pixels in a video encoding method provided in an embodiment of the present application;

[0019] FIG4 is a second flowchart of the video encoding method provided in an embodiment of the present application;

[0020] FIG5 is a third flowchart of the video encoding method provided in an embodiment of the present application;

[0021] FIG6 is a schematic diagram of determining a coding search range of a macroblock in a video coding method provided by an embodiment of the present application;

[0022] FIG7 is a fourth flowchart of the video encoding method provided in an embodiment of the present application;

[0023] FIG8 is a fifth flowchart of the video encoding method provided in an embodiment of the present application;

[0024] FIG9 is a sixth flowchart of the video encoding method provided in an embodiment of the present application;

[0025] FIG10 is a schematic diagram of a video encoding device provided in an embodiment of the present application;

[0026] FIG11 is a schematic diagram of an electronic device provided in an embodiment of the present application;

[0027] FIG12 is a hardware schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] The following will be combined with the accompanying drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the embodiments described are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field are within the scope of protection of this application.

[0029] The terms "first," "second," and the like in the specification and claims of this application are used to distinguish similar objects, and are not used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of this application can be implemented in an order other than that illustrated or described herein, and that the objects distinguished by "first," "second," and the like are generally of the same type, and do not limit the number of objects; for example, the first object can be one or more. In addition, the term "and / or" in the specification and claims refers to at least one of the connected objects, and the character " / " generally indicates that the objects connected are in an "or" relationship.

[0030] The terms "at least one" and "at least one of" in the specification and claims of this application refer to any one, any two, or a combination of more than two of the objects included. For example, at least one of a, b, and c can be represented by: "a", "b", "c", "a and b", "a and c", "b and c", and "a, b, and c", where a, b, and c can be single or multiple. Similarly, "at least two" means two or more, and its meaning is similar to "at least one".

[0031] First, some nouns or terms involved in the description and claims of this application are explained below.

[0032] Macroblock: It is a basic concept in video coding technology. It divides the picture into blocks of different sizes to implement different compression strategies at different locations. In video coding, a video frame image is usually divided into several blocks of the same size, called macroblocks. The size of a macroblock can be 16×16, 16×8, 8×16 or 8×8, etc., but the minimum can be 4×4.

[0033] Intra-coded frame (i.e., I-frame): also known as intra-frame or keyframe, it is an important frame in inter-frame compression coding and is an intra-frame compression. The image of the I-frame will be completely preserved, and only the data of this frame is needed to reconstruct the complete image when decoding the I-frame. The I-frame method is an intra-frame compression method, also known as the keyframe compression method. The I-frame method is a compression technology based on the Discrete Cosine Transform (DCT). Using I-frame compression can achieve a compression ratio of 1 / 6 without obvious compression traces.

[0034] Unidirectional predictive coding frame (i.e. P frame): also known as difference frame, belongs to inter-frame compression. After encoding, the P frame represents the difference information between the current frame and the I frame, or the P frame before the current frame. When decoding the P frame, it is necessary to superimpose the coded difference information defined by the current frame with the cached image of the P frame or I frame before the current frame to reconstruct the image of the current frame. The P frame method compresses the current frame data based on the differences between the current frame and the adjacent previous frame (I frame or P frame). The joint compression method of P frame and I frame can achieve higher compression without obvious compression traces.

[0035] Bidirectionally predictive coded frames (B-frames): Also known as bidirectional difference frames, B-frames record the difference between the current frame (i.e., the current frame) and the preceding and following frames. In other words, decoding a B-frame requires not only accessing the previous cached image but also decoding the following image, reconstructing the current frame using the coded data of the preceding and following frames and the current frame. The B-frame method is a bidirectionally predictive inter-frame compression algorithm. When compressing a frame into a B-frame, it compresses the frame based on the differences between the data of the previous, current, and following frames. Only using B-frame compression can achieve a high compression ratio of 200:1.

[0036] Electronic anti-shake algorithm: It is an algorithm that compensates for the displacement of the image in the video frames captured by the camera to achieve a smooth effect. The electronic anti-shake algorithm usually includes the following steps: (1) image stabilization processing is performed on the video frames captured by the camera. Specifically, it can be achieved by calculating the displacement of each video frame relative to the previous frame. The stabilization process will make the image transition smooth and avoid the jitter caused by camera shaking; (2) the movement of the camera is estimated. Specifically, the displacement change between two adjacent frames can be calculated to estimate the acceleration, speed and direction of the camera during the movement; (3) based on the results of the motion estimation, the pixels in the video frame are compensated. The method usually adopted is to perform transformations such as displacement, rotation and scaling on the video frame so that the pixel position in the video frame can be aligned with the previous frame.

[0037] The video encoding method, device, electronic device, and readable storage medium provided in the embodiments of the present application are described in detail below with reference to the accompanying drawings through specific embodiments and their application scenarios.

[0038] Mainstream encoding methods mainly include the Moving Pictures Experts Group (Mpeg) 1 encoding method, the Mpeg2 encoding method, the Mpeg4 encoding method, and the H.26X encoding method.

[0039] MPEG1 is the first lossy video and audio compression standard developed by the MPEG organization. It primarily utilizes block-based motion compensation, discrete cosine transform, and quantization technologies, and is optimized for a 1.2Mbps transmission rate. MPEG1 features include random access, flexible frame rates, variable image size, the definition of I-frames, P-frames, and B-frames, motion compensation that can span multiple frames, half-pixel precision motion vectors, and quantization matrices.

[0040] MPEG2 is a lossy video and audio compression standard developed by the MPEG organization, following MPEG1. Compared to MPEG1, MPEG2 offers higher image quality, a wider range of image formats, and higher bitrates. MPEG2 is a compression scheme for standard digital television and high-definition television in various applications, with a transmission rate between 3 and 10 Mbps. MPEG2 exploits two characteristics of images: spatial correlation and temporal correlation. Any scene within a frame is composed of several pixels, so a pixel typically has a certain relationship with its surrounding pixels in terms of brightness and color. This relationship is called spatial correlation. A program's plot often consists of a sequence of consecutive frames, and the frames within a sequence also have a certain relationship. This relationship is called temporal correlation. These two correlations result in a large amount of redundant information in the image. MPEG2 can remove this redundant information, retaining only a small amount of irrelevant information for transmission, significantly saving transmission bandwidth and improving coding efficiency.

[0041] MPEG4 is another lossy video and audio compression standard developed by the MPEG organization following MPEG2. Compared to MPEG1 and MPEG2, MPEG4 not only targets video and audio encoding at specific bit rates, but also places greater emphasis on the interactivity and flexibility of multimedia systems. MPEG4 is primarily used in video telephony, video email, and electronic news, requiring a relatively low transmission rate of 4,800-64,000 bits / s. MPEG4 utilizes very narrow bandwidth, using frame reconstruction, compression, and data transmission to achieve optimal image quality with minimal data. MPEG4 introduces several innovative new key technologies, including video object extraction, video object planar video coding, video coding scalability, and motion estimation and compensation.

[0042] H.26X is a next-generation digital video coding standard jointly proposed by the International Organization for Standardization and the International Telecommunication Union. Taking H.264 as an example, its key components include access unit delimiters, additional enhancement information, base picture coding, redundant picture coding, instantaneous decoding refresh, hypothetical reference decoding, and a hypothetical bitstream scheduler. H.264 is built on MPEG-4 technology, and its encoding and decoding process primarily includes five components: inter-frame and intra-frame prediction, transform and inverse transform, quantization and inverse quantization, loop filtering, and entropy coding. The advantages of using the H.264 encoding method for encoding are as follows: 1. Low bit rate: Under the same image quality, the amount of data compressed by H.264 technology is only 1 / 8 of Mpeg2 and 1 / 3 of Mpeg4; 2. High-quality image: H.264 can provide continuous, smooth, high-quality images; 3. Strong fault tolerance: H.264 provides the necessary tools to solve errors such as packet loss that are prone to occur in unstable network environments; 4. Strong network adaptability: H.264 provides a network abstraction layer, which allows H.264 files to be easily transmitted on different networks.

[0043] Currently, when an electronic device uses any of the above encoding methods to encode multiple video frames captured by a camera, it is usually necessary to perform at least one of the following steps A and B:

[0044] Step A: The electronic device determines the coding search range of each macroblock in the above-mentioned multiple video frames to identify redundant information between the multiple video frames, and compresses the identified redundant information through data compression technology to reduce the amount of data and the burden of storage information during transmission.

[0045] Among them, for the above-mentioned coding search range (including coding search radius and coding search direction), the electronic device can determine it through a search algorithm by traversing multiple pixel points around each pixel point in each macroblock of the above-mentioned multiple video frames. The search algorithm mainly includes diamond search method, hexagonal search method, full search method, etc.

[0046] For example, using the diamond search method described above, as shown in FIG1 , pixel 11 is a vertex of a macroblock. The electronic device can traverse four pixels in four directions around pixel 11, namely, pixel 12 to the left of pixel 11, pixel 13 below pixel 11, pixel 14 to the right of pixel 11, and pixel 15 above pixel 11. Thus, after traversing the four pixels surrounding each pixel of the macroblock, the electronic device can determine the encoding search range of the macroblock by comparing pixel values.

[0047] It should be noted that the up, down, left, and right in the embodiments of the present application are all illustrated by taking the screen of the electronic device displaying an interface as an example where the screen is facing the user.

[0048] It can be understood that since the electronic device determines the coding search range of each macroblock by traversing multiple pixel points around each pixel point in each macroblock of the above-mentioned multiple video frames, the electronic device needs to perform a large amount of calculation when determining the coding search range of each macroblock; especially when the motion amplitude of two adjacent frames is large, the large difference in image content will increase the number of pixel points traversed by the electronic device, which will further increase the computing power of the electronic device and consume additional coding performance, thus resulting in higher power consumption of the electronic device during the video encoding process.

[0049] Step B: The electronic device determines the internal coding frame among the plurality of video frames according to the degree of change of the image content of each video frame, so as to increase the video frame compression rate and reduce the size of the encoded video.

[0050] Specifically, the electronic device can first set a maximum value for the interval between internal coding frames, and then select video frames with drastically changed image content within the frame interval as internal coding frames; wherein, the method for determining video frames with drastically changed image content is to compare the pixel changes of macroblocks in the above-mentioned multiple video frames, and determine the video frames containing macroblocks with pixel changes exceeding a certain threshold as video frames with drastically changed image content. Obviously, since comparing the pixel changes of macroblocks in the multiple video frames also requires the electronic device to perform a large amount of calculations, the electronic device also needs to consume a large amount of computing power to determine the appropriate content coding frame among the multiple video frames, which also results in high power consumption of the electronic device during the video encoding process.

[0051] In order to solve the problem of high power consumption of electronic devices during video encoding, the embodiments of the present application provide a video encoding method, device, electronic device and readable storage medium. The video encoding method provided by the embodiments of the present application can be applied to the scenario where a mobile phone encodes the collected video frames.

[0052] Exemplarily, a mobile phone captures N (N is an integer greater than or equal to 2) video frames (e.g., at least two video frames in the embodiment of the present application), and in the process of capturing the N video frames, obtains acceleration data and angular velocity data of the mobile phone (e.g., the first data in the embodiment of the present application); then, based on the image data of the N video frames and the obtained acceleration data and angular velocity data of the mobile phone, the mobile phone can obtain a matrix (e.g., the affine transformation matrix in the embodiment of the present application) indicating the mapping relationship between corresponding macroblocks between every two adjacent video frames in the N video frames; then, based on the matrix, the mobile phone can determine at least one of the following: a coding search range for each macroblock in the N video frames; or an intra-coded frame in the N video frames. Thus, the mobile phone can encode the N video frames based on at least one of the determined coding search range and the intra-coded frame.

[0053] Through the solution of this application, on the one hand, when determining the coding search range for each macroblock in the N captured video frames, the mobile phone can directly use the above-mentioned matrix used to indicate the mapping relationship between the corresponding macroblocks of adjacent video frames, without having to traverse the multiple pixels around each pixel in each macroblock. On the other hand, when determining the internal coding frame in the N video frames, the mobile phone can also directly use the matrix, without having to compare the pixel changes of the macroblocks in the N video frames. In this way, when encoding the N video frames based on the coding search range and at least one of the internal coding frames, the computing power of the electronic device can be greatly reduced, and the power consumption of the electronic device can be reduced.

[0054] It should be noted that the video encoding method provided in the embodiments of the present application can be executed by a video encoding device, an electronic device, or a functional module in an electronic device, etc. In some embodiments of the present application, the video encoding method provided in the embodiments of the present application is described by taking an electronic device executing the video encoding method as an example.

[0055] FIG2 shows a flow chart of a video encoding method provided by an embodiment of the present application. As shown in FIG2 , the video encoding method provided by an embodiment of the present application may include the following steps 201 to 203 .

[0056] Step 201: The electronic device obtains an affine transformation matrix based on the collected image data of at least two video frames and first data.

[0057] The affine transformation matrix is ​​used to indicate the mapping relationship between corresponding macroblocks between every two adjacent video frames in the at least two video frames, and the first data is: acceleration data and angular velocity data of the electronic device during the process of collecting the at least two video frames.

[0058] For example, assuming that the at least two video frames include: video frame 1, video frame 2, video frame 3 and video frame 4; then each two adjacent video frames include: video frame 1 and video frame 2, video frame 2 and video frame 3, video frame 3 and video frame 4.

[0059] Optionally, in an embodiment of the present application, the at least two video frames are video frames continuously captured by the electronic device.

[0060] Optionally, in an embodiment of the present application, the at least two video frames may be captured by the electronic device through a camera in the electronic device.

[0061] Optionally, in an embodiment of the present application, the above-mentioned camera can be a traditional optical camera, an infrared camera, or a time of flight (TOF) camera in an electronic device.

[0062] For example, taking the above-mentioned camera as the above-mentioned traditional optical camera as an example, the camera can be a telephoto camera, a short-focus camera or a zoom camera, etc.

[0063] Optionally, in an embodiment of the present application, the at least two video frames may be a sequence of video frames arranged in a fixed order, and the fixed order may be determined according to the time sequence in which each of the at least two video frames is captured.

[0064] For example, assuming the at least two video frames include: video frame A captured at time a, video frame B captured at time b, and video frame C captured at time c, and the order of time a, time b, and time c is: time b, time a, time c; then the video frame sequence arranged in the fixed order can be: video frame B, video frame A, video frame C. It can be seen that the fixed order is determined based on the order of time a, time b, and time c.

[0065] Optionally, in an embodiment of the present application, the above-mentioned image data may include: pixel coordinate data of pixel points or pixel values ​​of pixel points, etc.

[0066] Optionally, in an embodiment of the present application, the first data may be acquired by an inertial measurement unit (IMU) sensor in the electronic device during the process of collecting the at least two video frames.

[0067] It should be noted that the above-mentioned IMU sensor can be used to collect data such as acceleration, angular velocity, tilt, impact, vibration, rotation or multi-degree-of-freedom motion.

[0068] Optionally, in an embodiment of the present application, the IMU sensor may include a timer and a first-in-first-out (FIFO) stack; the IMU sensor may generate interrupts at fixed time intervals through the timer to collect data, and after each data collection, the collected sampled data is recorded and saved in the FIFO stack, while simultaneously sending an interrupt signal to a processor in the electronic device to notify the processor to read the sampled data collected by the IMU sensor at fixed time intervals. In this way, the processor can read the sampled data after receiving the interrupt signal.

[0069] Optionally, in an embodiment of the present application, the above-mentioned acceleration data can be all acceleration data sampled by the above-mentioned IMU sensor during the process of acquiring the above-mentioned at least two video frames; or it can be an acceleration data obtained after linear fitting of all the acceleration data; or it can be an acceleration data obtained after averaging all the acceleration data.

[0070] Optionally, in an embodiment of the present application, the angular velocity data may be all angular velocity data sampled by the IMU sensor during the process of acquiring the at least two video frames; or may be an angular velocity data obtained after linear fitting of all angular velocity data; or may be an angular velocity data obtained after averaging all angular velocity data.

[0071] Optionally, in an embodiment of the present application, the mapping relationship between corresponding macroblocks between every two adjacent video frames in the at least two video frames is an affine transformation relationship between the corresponding macroblocks.

[0072] Optionally, in an embodiment of the present application, the corresponding macroblocks between two adjacent video frames include multiple groups of macroblocks, each group of macroblocks includes: a macroblock in one video frame of the two adjacent video frames (hereinafter referred to as macroblock 1), and a macroblock in the other video frame of the two adjacent video frames (hereinafter referred to as macroblock 2), and the degree of similarity between the pixel values ​​of the pixel points in macroblock 1 and the pixel values ​​of the pixel points in macroblock 2 is greater than or equal to a preset degree of similarity.

[0073] It can be understood that the two macroblocks in each group of macroblocks correspond to the same object in the image of the respective video frame.

[0074] For example, assuming that video frame a and video frame b are two adjacent video frames, and the two video frames record the flight trajectory of a flying bird, if the macroblock A in the video frame a corresponds to the flying bird in the image of the video frame a, then the macroblock B in the video frame b corresponding to the macroblock A corresponds to the flying bird in the image of the video frame b.

[0075] It should be noted that an affine transformation, also known as an affine mapping, refers to a geometric transformation in which a vector space undergoes a nonsingular linear transformation followed by a translation, transforming it into another vector space. In the finite-dimensional case, each affine transformation can be given by a matrix A and a vector b, which can be written as A with an additional column b. An affine transformation corresponds to the multiplication of a matrix and a vector, while the composition of affine transformations corresponds to ordinary matrix multiplication, simply by adding an extra row to the bottom of the matrix. This row is all zeros except for a 1 on the rightmost side, and a 1 is added to the bottom of the column vector.

[0076] The following uses a pixel point in a macroblock as an example to illustrate the above affine transformation relationship.

[0077] Exemplarily, assume that any two adjacent video frames are the i-th video frame and the i+1-th video frame in the at least two video frames, and i is a positive integer; then, as shown in FIG3 , pixel 31 is the center pixel of macroblock 32 in the i-th video frame, and since the electronic device jitters during the process of acquiring the at least two video frames, the macroblock 34 corresponding to macroblock 32 in the i+1-th video frame is distorted, so that the geometric position of pixel 33 of macroblock 34 corresponding to pixel 31 is transformed, and the transformation relationship of the geometric position is the affine transformation relationship between pixel 31 and pixel 33.

[0078] Optionally, in an embodiment of the present application, the above-mentioned affine transformation relationship can be recorded through the above-mentioned affine transformation matrix, so that the affine transformation matrix can indicate the mapping relationship between corresponding macroblocks between any two adjacent video frames in the above-mentioned at least two video frames.

[0079] Optionally, in an embodiment of the present application, the above-mentioned affine transformation matrix may include at least two sub-matrices, each of the at least two sub-matrices corresponds to one of the at least two video frames, and each sub-matrix is ​​used to indicate the geometric position of the macroblock in the corresponding video frame.

[0080] Optionally, in an embodiment of the present application, the geometric position of a macroblock in a video frame can be determined by the coordinate values ​​of the pixel points in the macroblock.

[0081] Optionally, in the embodiment of the present application, the values ​​in each of the above sub-matrices may include: coordinate values ​​of a pixel point in a corresponding video frame.

[0082] For example, assume that macroblock 5 in video frame 5 includes pixel 1, pixel 2, pixel 3 and pixel 4, and the four pixel points are the four vertices of macroblock 5; if the coordinate value of pixel 1 is (2, 2), the coordinate value of pixel 2 is (4, 2), the coordinate value of pixel 3 is (2, 4), and the coordinate value of pixel 4 is (4, 4), then the geometric position of macroblock 5 in the video frame 5 can obviously be determined by the coordinate values ​​of the four pixel points.

[0083] Optionally, in an embodiment of the present application, the at least two sub-matrices may also be arranged in the above fixed order.

[0084] Optionally, in an embodiment of the present application, any two adjacent sub-matrices in the at least two sub-matrices may indicate a mapping relationship between corresponding macroblocks between two adjacent video frames in the at least two video frames.

[0085] In an embodiment of the present application, since the above-mentioned affine transformation matrix may include: the above-mentioned at least two sub-matrices corresponding one-to-one to the above-mentioned at least two video frames, and each sub-matrix can be used to indicate the geometric position of the macroblock in the corresponding video frame, when the electronic device needs to obtain the mapping relationship between the corresponding macroblocks between any two adjacent video frames, it only needs to use the sub-matrices corresponding to the any two adjacent video frames without considering other sub-matrices, thereby reducing the computing power consumption of the electronic device.

[0086] Optionally, in an embodiment of the present application, in combination with FIG1 , as shown in FIG4 , the above step 201 may be specifically implemented through the following steps 201a to 201c.

[0087] Step 201a: The electronic device inputs the image data and the first data into an electronic image stabilization algorithm.

[0088] In the embodiment of the present application, the above-mentioned image data is the image data of the above-mentioned at least two video frames.

[0089] In the embodiment of the present application, the above-mentioned electronic anti-shake algorithm is a commonly used algorithm for improving the shaking problem of electronic devices when shooting videos. The electronic anti-shake algorithm can work in conjunction with the gyroscope in the electronic device (that is, a device for detecting angular motion of a high-speed rotating body relative to the inertial space around one or two axes orthogonal to the axis of rotation using the angular motion of the sensitive shell of the angular motion of the high-speed rotating body).

[0090] Specifically, during the video shooting process, when the above-mentioned gyroscope detects the vibration of the electronic device, the above-mentioned electronic anti-shake algorithm can analyze and collect the image on the sensor, and calculate the rotational changes of the electronic device's posture, dynamically adjust the sensitivity, shutter, etc. to make blur corrections, and dynamically crop the video frames to reduce the impact of the electronic device shaking on the shooting, effectively improving the stability of the video picture.

[0091] Optionally, in an embodiment of the present application, the above-mentioned electronic anti-shake algorithm can obtain the motion vector of the pixel based on the image content of the video frame or the IMU sensor to calculate the rotation change of the electronic device's posture.

[0092] For a detailed description of the electronic image stabilization algorithm, please refer to the relevant description in the related art. To avoid repetition, it will not be repeated here.

[0093] Step 201b: The electronic device obtains pixel coordinate data of feature points in at least two video frames from the image data using an electronic anti-shake algorithm.

[0094] Optionally, in an embodiment of the present application, the feature points in the at least two video frames include: feature points in each of the at least two video frames.

[0095] Optionally, in an embodiment of the present application, a feature point in a video frame may be: a vertex, a corner point, a center point in the video frame, or a point in the video frame where the grayscale value changes dramatically, etc.

[0096] Optionally, in an embodiment of the present application, the feature points of an image can reflect the essential features of the image and can identify the target object in the image, and image matching can be completed by matching the feature points.

[0097] Optionally, in an embodiment of the present application, pixel coordinate data of a pixel point is used to indicate: the geometric position of the pixel point in the video frame.

[0098] For example, if the coordinate data of a pixel point is (a, b), then the geometric position of the pixel point in the video frame is: a pixel away from the origin (usually the upper left corner of the video frame) in the X-axis direction and b pixels away from the origin in the Y-axis direction.

[0099] Step 201c: The electronic device calculates an affine transformation matrix based on the pixel coordinate data and the first data using an electronic anti-shake algorithm.

[0100] In the embodiment of the present application, the above-mentioned pixel coordinate data is the pixel coordinate data of the feature points in the above-mentioned at least two video frames.

[0101] For the specific method of calculating the affine transformation matrix of the electronic device based on pixel coordinate data, angular velocity data and acceleration data, please refer to the relevant description in the relevant technology. In order to avoid repetition, it will not be repeated here.

[0102] In an embodiment of the present application, since the electronic device can input the image data of the above-mentioned at least two video frames and the first data into the above-mentioned electronic anti-shake algorithm to obtain the above-mentioned affine transformation matrix, it can be ensured based on the motion estimation function of the electronic anti-shake algorithm that the obtained affine transformation matrix can accurately indicate the mapping relationship between corresponding macroblocks between any two adjacent video frames in the at least two video frames.

[0103] Step 202: The electronic device determines a first object based on an affine transformation matrix.

[0104] The first object includes at least one of the following:

[0105] a coding search range for each macroblock in the at least two video frames;

[0106] An intra-coded frame among the at least two video frames.

[0107] In an embodiment of the present application, the encoding search range of a macroblock is used to determine: the macroblock in the next video frame of the video frame where the macroblock is located that is most similar to the macroblock (ie, the degree of pixel value matching is greater than or equal to a preset threshold).

[0108] Optionally, in an embodiment of the present application, the above-mentioned coding search range may include: a coding search radius and a coding search direction.

[0109] Optionally, in an embodiment of the present application, the coding search radius of a macroblock is used to indicate the size of the coding search range of the macroblock.

[0110] For example, assuming that the encoding search radius of macroblock 1 is a pixels, the encoding search range of macroblock 1 is: a circular range with a radius of a pixels and a geometric position of macroblock 1 in the video frame as the origin.

[0111] Optionally, in an embodiment of the present application, the encoding search direction of a macroblock is used to indicate: the encoding search range of the macroblock is compared to the direction where the macroblock is located.

[0112] For example, assuming that the encoding search direction of macroblock 2 is the lower right, then the direction of the encoding search range of macroblock 2 is the lower right of macroblock 2.

[0113] It can be seen that through the coding search radius and coding search direction of a macroblock, the coding search range of the macroblock can be narrowed down to a smaller range, so as to determine the macroblock that is most similar to the macroblock in the next video frame of the video frame where the macroblock is located.

[0114] For a detailed description of the above-mentioned internal coding frame, reference may be made to the relevant descriptions in the explanations of some nouns or terms involved in the specification and claims of this application. In order to avoid repetition, they will not be repeated here.

[0115] The specific method for the electronic device to determine the first object is described in detail below.

[0116] Optionally, in an embodiment of the present application, the first object includes a coding search range of each macroblock in the at least two video frames. Exemplarily, in conjunction with FIG1 , as shown in FIG5 , the step 202 can be implemented by the following steps 202a and 202b.

[0117] Step 202a: The electronic device determines a motion vector corresponding to each macroblock according to at least two sub-matrices.

[0118] Each motion vector indicates a search range.

[0119] Optionally, in an embodiment of the present application, the electronic device may determine a motion vector corresponding to each macroblock of one of the at least two video frames based on any two adjacent sub-matrices of the at least two sub-matrices.

[0120] Specifically, the process of the electronic device determining the motion vector corresponding to each macroblock of the at least two video frames may include the following steps 1 to N:

[0121] 1. The electronic device determines a motion vector for each macroblock in a first video frame of the at least two video frames corresponding to the first submatrix by comparing changes between values ​​of a first submatrix of the at least two submatrices and values ​​of a second submatrix of the at least two submatrices;

[0122] 2. The electronic device determines a motion vector for each macroblock in a second video frame of the at least two video frames corresponding to the second submatrix by comparing changes between values ​​of the second submatrix of the at least two submatrices and values ​​of a third submatrix of the at least two submatrices;

[0123] 3. The electronic device determines the motion vector of each macroblock in the third of the at least two video frames corresponding to the third submatrix by comparing the values ​​of the third submatrix of the at least two submatrices with the changes between the values ​​of the fourth submatrix of the at least two submatrices;

[0124] …

[0125] N-1. The electronic device determines the motion vector of each macroblock in the video frame preceding the last of the at least two video frames corresponding to the previous submatrix by comparing the change between the values ​​of the previous submatrix of the last submatrix and the values ​​in the last submatrix;

[0126] N. Since the macroblocks of the last video frame of the at least two video frames do not need to perform motion estimation, the electronic device may determine the vectors corresponding to each macroblock of the last video frame to be 0;

[0127] In this way, the electronic device can determine the motion vector corresponding to each macroblock in the at least two video frames.

[0128] Step 202b: For each macroblock, the electronic device determines the search range indicated by the motion vector corresponding to the macroblock as the encoding search range of the macroblock, thereby obtaining the encoding search range of each macroblock.

[0129] In the embodiment of the present application, the motion vector includes size and direction, and a search range can be determined by the size and direction of a motion vector.

[0130] Exemplarily, as shown in FIG6 , the xth video frame and the x+1th video frame are any two adjacent video frames among the at least two video frames, where x is a positive integer. The electronic device can obtain the geometric positions of pixels in the xth video frame and the x+1th video frame based on the xth submatrix corresponding to the xth video frame and the x+1th submatrix corresponding to the x+1th video frame. For example, the coordinates of pixel 62 in the xth video frame are (50, 50), and the coordinates of pixel 63 in the x+1th video frame corresponding to pixel 62 are (100, 100). Thus, the motion vector corresponding to macroblock 61 in the xth video frame can be determined. It can be seen that the search range indicated by the motion vector is: a range with a radius of 50 pixels and a direction toward the lower right of macroblock 61. Thus, the electronic device can determine this range as the encoding search range for macroblock 61.

[0131] It can be understood that the electronic device obtains the encoding search range of each macroblock after determining the search range indicated by the motion vector corresponding to each macroblock as the encoding search range of the corresponding macroblock.

[0132] In an embodiment of the present application, since the electronic device can first determine the motion vector corresponding to each macroblock, and then determine the search range indicated by each motion vector as the encoding search range of the corresponding macroblock, the process of determining the encoding search range of each macroblock can be simplified, thereby greatly reducing the complexity of the calculation.

[0133] Optionally, in an embodiment of the present application, the first object includes an intra-coded frame in the at least two video frames. For example, in conjunction with FIG1 , as shown in FIG7 , the step 202 can be implemented by the following steps 202c and 202d.

[0134] Step 202c: The electronic device determines a first change rate corresponding to each sub-matrix based on the at least two sub-matrices.

[0135] The first change rate corresponding to a sub-matrix includes: a change rate of values ​​in the sub-matrix compared to values ​​in a previous sub-matrix of the sub-matrix.

[0136] Optionally, in an embodiment of the present application, the first change rate corresponding to a submatrix can be determined based on each value in the submatrix and the corresponding value in the previous submatrix of the submatrix.

[0137] For example, assuming that submatrix 1 and submatrix 2 are two adjacent submatrices among the at least two submatrices mentioned above, and submatrix 1 is located before submatrix 2, submatrix 1 is [a], and submatrix 2 is [b]; then a and b are corresponding values, so that the electronic device can determine the first change rate corresponding to submatrix 2 as (b a) / a based on a in submatrix 1 and b in submatrix 2.

[0138] For another example, assuming that submatrix 3 and submatrix 4 are two adjacent submatrices among the at least two submatrices mentioned above, and submatrix 3 is located before submatrix 4, submatrix 3 is [c1, d1], and submatrix 4 is [c2, d2]; then c1 and c2 are corresponding values, and d1 and d2 are corresponding values. Therefore, the electronic device can first obtain the change rates of the two groups of corresponding values, namely (c2-c1) / c1 and (d2-d1) / d1, respectively, based on c1 in submatrix 1 and c2 in submatrix 2, as well as d1 in submatrix 1 and d2 in submatrix 2, and then determine the average value of (c2-c1) / c1 and (d2-d1) / d1 as the first change rate corresponding to submatrix 4.

[0139] Optionally, in the embodiment of the present application, the process of the electronic device determining the first change rate corresponding to each of the at least two sub-matrices may include the following steps a to n:

[0140] a. Since no submatrix exists before a first submatrix of the at least two submatrices, the values ​​in the first submatrix do not have a change rate compared to the values ​​in the submatrix preceding the first submatrix, and thus the electronic device determines the first change rate corresponding to the first submatrix to be 0;

[0141] b. The electronic device determines, based on the values ​​in a first submatrix of the at least two submatrices and the values ​​in a second submatrix of the at least two submatrices, a first change rate corresponding to the second submatrix;

[0142] c. The electronic device determines, based on the values ​​in the second submatrix of the at least two submatrices and the values ​​in the third submatrix of the at least two submatrices, a first change rate corresponding to the third submatrix;

[0143] d. The electronic device determines a first change rate corresponding to the fourth submatrix based on the values ​​in the third submatrix of the at least two submatrices and the values ​​in the fourth submatrix of the at least two submatrices;

[0144] …

[0145] n. The electronic device determines a first change rate corresponding to the last submatrix of the at least two submatrices based on the values ​​in the submatrix preceding the last submatrix and the values ​​in the last submatrix;

[0146] In this way, the electronic device can determine the first change rate corresponding to each of the above sub-matrices.

[0147] Step 202d: The electronic device determines an intra-coded frame based on the first change rate.

[0148] In the embodiment of the present application, the above-mentioned first change rate is the first change rate corresponding to each of the above-mentioned sub-matrices.

[0149] In an embodiment of the present application, since the electronic device can directly determine the internal coding frame in the above-mentioned at least two video frames based on the first change rate corresponding to each of the above-mentioned sub-matrices, without having to compare the pixel changes of the macroblocks in the at least two video frames, the process of determining the internal coding frame can be simplified, thereby reducing the complexity of the calculation.

[0150] The specific method for the electronic device to determine the above-mentioned intra-coded frame based on the first change rate corresponding to each of the above-mentioned sub-matrices is described in detail below.

[0151] Optionally, in an embodiment of the present application, the electronic device may determine the above-mentioned internal coding frame based on the first change rate corresponding to each of the above-mentioned sub-matrices through the following method 1 or method 2.

[0152] Method 1

[0153] Optionally, in the embodiment of the present application, in combination with FIG. 7 , as shown in FIG. 8 , the above step 202d may be specifically implemented through the following step 202d1 .

[0154] Step 202d1: For at least one first video frame corresponding to a submatrix having a first change rate greater than or equal to a first threshold, the electronic device determines the at least one first video frame as an intra-coded frame.

[0155] Optionally, in an embodiment of the present application, the first threshold value may be a system default value, or may be set by a user according to actual usage requirements.

[0156] Optionally, in the embodiment of the present application, the first threshold may be any value such as 50%, 60% or 70%.

[0157] Exemplarily, assuming that the at least two sub-matrices are sub-matrix A, sub-matrix B and sub-matrix C, respectively, and the first threshold is 60%, then if the electronic device determines that the first change rate corresponding to sub-matrix A is 0, the first change rate corresponding to sub-matrix B is 50%, and the first change rate corresponding to sub-matrix C is 68%, then the electronic device can determine the video frame corresponding to sub-matrix C as an internal coding frame.

[0158] In an embodiment of the present application, the at least one first video frame is a video frame among the at least two video frames.

[0159] Optionally, in an embodiment of the present application, each submatrix in the at least two submatrices whose first change rate is greater than or equal to the first threshold corresponds to one first video frame in the at least one first video frame.

[0160] Method 2

[0161] Optionally, in an embodiment of the present application, in combination with FIG. 7 , as shown in FIG. 9 , the above step 202d may be specifically implemented through the following steps 202d2 and 202d3 .

[0162] Step 202d2: For at least one second video frame corresponding to a submatrix having a first change rate less than a first threshold, the electronic device calculates a second change rate corresponding to each second video frame.

[0163] The second change rate corresponding to a second video frame includes: a change rate of a pixel value of a pixel in the second video frame compared to a pixel value of a pixel in a video frame preceding the second video frame.

[0164] For a detailed description of how the electronic device calculates the second change rate corresponding to each second video frame, reference may be made to the relevant description in the related art, which will not be repeated here to avoid repetition.

[0165] Step 202d3: The electronic device determines all second video frames whose second change rate is greater than or equal to the second threshold as intra-coded frames.

[0166] Optionally, in an embodiment of the present application, the second threshold value may be a system default value, or may be set by a user according to actual usage requirements.

[0167] Optionally, in the embodiment of the present application, the second threshold may be any value such as 65%, 75% or 80%.

[0168] Exemplarily, assuming that the at least two sub-matrices are sub-matrix a, sub-matrix b and sub-matrix c, respectively, and the first threshold is 60%, and the second threshold is 75%, then if the electronic device determines that the first change rate corresponding to the sub-matrix a is 0, the first change rate corresponding to the sub-matrix b is 50%, and the first change rate corresponding to the sub-matrix c is 68%, then the electronic device can directly determine the video frame corresponding to the sub-matrix c as an internal coding frame, and then calculate the second change rate corresponding to the video frame corresponding to the sub-matrix a (hereinafter referred to as video frame a), and the second change rate corresponding to the video frame corresponding to the sub-matrix b (hereinafter referred to as video frame b); if the second change rate corresponding to the video frame a calculated by the electronic device is 0 and the second change rate corresponding to the video frame b is 78%, then the electronic device can also determine the video frame b as an internal coding frame.

[0169] In an embodiment of the present application, the at least one second video frame is a video frame among the at least two video frames.

[0170] Optionally, in an embodiment of the present application, the electronic device may determine, as a P / B frame, a second video frame in which the second change rate of the at least one second video frame is less than the second threshold.

[0171] In an embodiment of the present application, since the electronic device can directly determine at least one first video frame corresponding to a submatrix whose first change rate is greater than or equal to the first threshold as the above-mentioned internal coding frame; or can first calculate the second change rate of each second video frame corresponding to a submatrix whose first change rate is less than the first threshold, and determine all second video frames whose second change rate is greater than or equal to the second threshold as the internal coding frame; therefore, the flexibility of determining the internal coding frame can be improved.

[0172] Optionally, in an embodiment of the present application, after obtaining the above-mentioned affine transformation matrix, the electronic device can transmit the affine transformation matrix and the image data of the above-mentioned at least two video frames to an encoder in the electronic device, and then determine the above-mentioned first object based on the above-mentioned affine transformation matrix through the encoder.

[0173] Optionally, in an embodiment of the present application, the above-mentioned encoder is a device that compiles and converts signals (such as bit streams) or data into a signal form that can be used for communication, transmission and storage. The encoder converts angular displacement or linear displacement into an electrical signal. The former is called a code disk and the latter is called a code ruler.

[0174] Optionally, in an embodiment of the present application, the above-mentioned encoder can be an incremental encoder or an absolute encoder; wherein, the incremental encoder converts the displacement into a periodic electrical signal, and then converts the electrical signal into a counting pulse, and uses the number of pulses to represent the magnitude of the displacement; each position of the absolute encoder corresponds to a certain digital code, so its indication is only related to the starting and ending positions of the measurement, and has nothing to do with the intermediate process of the measurement.

[0175] Step 203: The electronic device encodes at least two video frames based on the first object.

[0176] Optionally, in an embodiment of the present application, the electronic device encodes the at least two video frames based on the encoding search range, and can identify redundant information between the at least two video frames to compress the redundant information.

[0177] Optionally, in an embodiment of the present application, the electronic device encodes the at least two video frames based on the internal coding frame, which can increase the video frame compression rate and reduce the volume of the encoded video.

[0178] For the specific method of encoding video frames for electronic devices, reference can be made to the relevant description in the related art, and in order to avoid repetition, it will not be described here.

[0179] In the video encoding method provided in an embodiment of the present application, on the one hand, when determining the encoding search range for each macroblock in at least two captured video frames, the electronic device can directly base its determination on an affine transformation matrix indicating the mapping relationship between corresponding macroblocks in adjacent video frames, without having to traverse multiple pixels around each pixel in each macroblock; on the other hand, when determining the internal encoding frame in the at least two video frames, the electronic device can also directly base its determination on the affine transformation matrix, without having to compare pixel changes in the macroblocks in the at least two video frames. In this way, when encoding the at least two video frames based on at least one of the encoding search range and the internal encoding frame, the computing power of the electronic device can be greatly reduced, thereby reducing the power consumption of the electronic device.

[0180] The above-mentioned method embodiments, or various possible implementation methods in each method embodiment, can be executed separately, or, under the premise that there is no contradiction, can also be executed in combination with each other. The specific implementation can be determined according to actual usage requirements, and the embodiments of the present application do not limit this.

[0181] The video encoding method provided in the embodiment of the present application can be executed by a video encoding device. In the embodiment of the present application, the video encoding device provided in the embodiment of the present application is described by taking the video encoding method performed by the video encoding device as an example.

[0182] As shown in FIG10 , an embodiment of the present application provides a video encoding device 10 , which may include an acquisition module 11 , a determination module 12 , and an encoding module 13 .

[0183] The acquisition module 11 can be used to acquire an affine transformation matrix based on the image data of at least two acquired video frames and first data, wherein the affine transformation matrix is ​​used to indicate the mapping relationship between corresponding macroblocks between each two adjacent video frames in the at least two video frames, and the first data is: acceleration data and angular velocity data of the electronic device during the acquisition of the at least two video frames. The determination module 12 can be used to determine a first object based on the affine transformation matrix acquired by the acquisition module 11. The encoding module 13 can be used to encode the at least two video frames based on the first object determined by the determination module 12. The first object includes at least one of the following: an encoding search range for each macroblock in the at least two video frames; an internal encoding frame in the at least two video frames.

[0184] In one possible implementation, the affine transformation matrix may include at least two sub-matrices, each sub-matrix corresponding to one of the at least two video frames, and each sub-matrix is ​​used to indicate the geometric position of a macroblock in the corresponding video frame.

[0185] In one possible implementation, the first object includes a coding search range for each macroblock in the at least two video frames. Determination module 12 may be specifically configured to determine, based on the at least two sub-matrices, a motion vector corresponding to each macroblock, each motion vector indicating a search range; and, for each macroblock, determine the search range indicated by the motion vector corresponding to the macroblock as the coding search range for the macroblock, thereby obtaining the coding search range for each macroblock.

[0186] In one possible implementation, the first object includes an intra-coded frame in the at least two video frames. Determination module 12 may be specifically configured to determine, based on the at least two submatrices, a first change rate corresponding to each of the at least two submatrices; and determine the intra-coded frame based on the first change rate. The first change rate corresponding to a submatrix includes the rate of change of a value in the submatrix compared to a value in a submatrix immediately preceding the submatrix.

[0187] In one possible implementation, the determination module 12 can be specifically used to: for at least one first video frame corresponding to the submatrix whose first change rate is greater than or equal to the first threshold, determine the at least one first video frame as the internal coding frame; or, for at least one second video frame corresponding to the submatrix whose first change rate is less than the first threshold, calculate the second change rate corresponding to each second video frame, the second change rate corresponding to a second video frame including: the rate of change of the pixel value of the pixel in the second video frame compared to the pixel value of the pixel in the previous video frame of the second video frame; and determine all second video frames whose second change rate is greater than or equal to the second threshold as the internal coding frame.

[0188] In one possible implementation, the acquisition module 11 can be specifically used to input the above-mentioned image data and the above-mentioned first data into an electronic anti-shake algorithm; and through the electronic anti-shake algorithm, obtain the pixel coordinate data of the feature points in the above-mentioned at least two video frames from the image data; and through the electronic anti-shake algorithm, calculate the above-mentioned affine transformation matrix based on the pixel coordinate data and the first data.

[0189] In the video encoding device provided in the embodiment of the present application, on the one hand, when determining the encoding search range for each macroblock in at least two captured video frames, the video encoding device can directly base its determination on an affine transformation matrix indicating the mapping relationship between corresponding macroblocks in adjacent video frames, without having to traverse multiple pixels around each pixel in each macroblock; on the other hand, when determining the internal encoding frame in the at least two video frames, the video encoding device can also directly base its determination on the affine transformation matrix, without having to compare pixel changes in the macroblocks in the at least two video frames. In this way, when encoding the at least two video frames based on at least one of the encoding search range and the internal encoding frame, computing power and power consumption can be greatly reduced.

[0190] The video encoding device in the embodiment of the present application can be an electronic device or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices other than a terminal. For example, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, an in-vehicle electronic device, a mobile Internet device (MID), an augmented reality (AR) / virtual reality (VR) device, a robot, a wearable device, an ultra-mobile personal computer (UMPC), a netbook or a personal digital assistant (PDA), etc. It can also be a server, a network attached storage (NAS), a personal computer (PC), a television (TV), a teller machine or a self-service machine, etc., and the embodiment of the present application does not specifically limit it.

[0191] The video encoding device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0192] The video encoding device provided in the embodiment of the present application can implement each process implemented in the above method embodiment. To avoid repetition, it will not be repeated here.

[0193] As shown in Figure 11, an embodiment of the present application also provides an electronic device 100, including a processor 101 and a memory 102, and the memory 102 stores a program or instruction that can be run on the processor 101. When the program or instruction is executed by the processor 101, the various steps of the above-mentioned video encoding method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0194] It should be noted that the electronic devices in the embodiments of the present application include mobile electronic devices and non-mobile electronic devices.

[0195] FIG12 is a schematic diagram of the hardware structure of an electronic device implementing an embodiment of the present application.

[0196] As shown in Figure 12, the electronic device 1000 includes but is not limited to: a radio frequency unit 1001, a network module 1002, an audio output unit 1003, an input unit 1004, a sensor 1005, a display unit 1006, a user input unit 1007, an interface unit 1008, a memory 1009, and a processor 1010 and other components.

[0197] Those skilled in the art will appreciate that the electronic device 1000 may further include a power source (such as a battery) to power various components. The power source may be logically connected to the processor 1010 via a power management system, thereby enabling the power management system to manage charging, discharging, and power consumption. The electronic device structure shown in FIG12 does not limit the electronic device. The electronic device may include more or fewer components than shown, or may combine certain components, or have different component arrangements, which will not be described in detail here.

[0198] In particular, processor 1010 can be configured to obtain an affine transformation matrix based on image data of at least two captured video frames and first data, the affine transformation matrix being used to indicate a mapping relationship between corresponding macroblocks between each two adjacent video frames in the at least two video frames, wherein the first data is acceleration data and angular velocity data of the electronic device during the process of capturing the at least two video frames. Processor 1010 can also be configured to determine a first object based on the acquired affine transformation matrix. Processor 1010 can also be configured to encode the at least two video frames based on the determined first object. In particular, the first object includes at least one of the following: a coding search range for each macroblock in the at least two video frames; or an intra-coded frame in the at least two video frames.

[0199] In one possible implementation, the affine transformation matrix may include at least two sub-matrices, each sub-matrix corresponding to one of the at least two video frames, and each sub-matrix is ​​used to indicate the geometric position of a macroblock in the corresponding video frame.

[0200] In one possible implementation, the first object includes a coding search range for each macroblock in the at least two video frames. Processor 1010 may be specifically configured to determine, based on the at least two sub-matrices, a motion vector corresponding to each macroblock, each motion vector indicating a search range; and, for each macroblock, determine the search range indicated by the motion vector corresponding to the macroblock as the coding search range for the macroblock, thereby obtaining the coding search range for the macroblock.

[0201] In one possible implementation, the first object includes an intra-coded frame in the at least two video frames. Processor 1010 may be specifically configured to determine, based on the at least two submatrices, a first change rate corresponding to each of the at least two submatrices; and determine the intra-coded frame based on the first change rate. The first change rate corresponding to a submatrix includes: a change rate of a value in the submatrix compared to a value in a submatrix immediately preceding the submatrix.

[0202] In one possible implementation, the processor 1010 can be specifically used to: for at least one first video frame corresponding to the submatrix whose first change rate is greater than or equal to the first threshold, determine the at least one first video frame as the internal coding frame; or, for at least one second video frame corresponding to the submatrix whose first change rate is less than the first threshold, calculate the second change rate corresponding to each second video frame, the second change rate corresponding to a second video frame including: the rate of change of the pixel value of the pixel in the second video frame compared to the pixel value of the pixel in the previous video frame of the second video frame; and determine all second video frames whose second change rate is greater than or equal to the second threshold as the internal coding frame.

[0203] In one possible implementation, the processor 1010 can be specifically used to input the above-mentioned image data and the above-mentioned first data into an electronic anti-shake algorithm; and through the electronic anti-shake algorithm, obtain the pixel coordinate data of the feature points in the above-mentioned at least two video frames from the image data; and through the electronic anti-shake algorithm, calculate the above-mentioned affine transformation matrix based on the pixel coordinate data and the first data.

[0204] In the electronic device provided in the embodiment of the present application, on the one hand, when determining the coding search range of each macroblock in at least two captured video frames, the electronic device can directly base its determination on an affine transformation matrix indicating the mapping relationship between corresponding macroblocks of adjacent video frames, without having to traverse multiple pixel points around each pixel point in each macroblock; on the other hand, when determining the internal coding frame in the at least two video frames, the electronic device can also directly base its determination on the affine transformation matrix, without having to compare pixel changes in the macroblocks in the at least two video frames. In this way, when encoding the at least two video frames based on at least one of the coding search range and the internal coding frame, the computing power of the electronic device can be greatly reduced, and the power consumption of the electronic device can be reduced.

[0205] It should be understood that in an embodiment of the present application, the input unit 1004 may include a graphics processing unit (GPU) 10041 and a microphone 10042, and the GPU 10041 processes the image data of a static picture or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 1006 may include a display panel 10061, and the display panel 10061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 1007 includes a touch panel 10071 and at least one of other input devices 10072. The touch panel 10071 is also called a touch screen. The touch panel 10071 may include two parts: a touch detection device and a touch controller. Other input devices 10072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and an operating stick, which will not be repeated here.

[0206] The memory 1009 can be used to store software programs and various data. The memory 1009 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, applications or instructions required for at least one function (such as a sound playback function, an image playback function, etc.). In addition, the memory 1009 may include a volatile memory or a non-volatile memory, or the memory 1009 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM), and a direct memory bus random access memory (DRRAM). The memory 1009 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0207] Processor 1010 may include one or more processing units. Optionally, processor 1010 integrates an application processor and a modem processor. The application processor primarily handles operations related to the operating system, user interface, and application programs, while the modem processor primarily processes wireless communication signals, such as a baseband processor. It is understood that the modem processor may not be integrated into processor 1010.

[0208] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned video encoding method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0209] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0210] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned video encoding method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0211] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0212] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned video encoding method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0213] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element. In addition, it should be noted that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in the opposite order according to the functions involved. For example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0214] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), including a number of instructions for enabling a terminal (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0215] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.

Claims

1. A video encoding method, comprising: Based on the image data of at least two video frames collected, and first data, an affine transformation matrix is ​​obtained, where the affine transformation matrix is ​​used to indicate a mapping relationship between corresponding macroblocks between every two adjacent video frames in the at least two video frames, and the first data is: acceleration data and angular velocity data of the electronic device during the process of collecting the at least two video frames; Based on the affine transformation matrix, determining a first object; Based on the first object, encoding the at least two video frames; Wherein, the first object includes at least one of the following: a coding search range for each macroblock in the at least two video frames; An intra-coded frame among the at least two video frames.

2. The method according to claim 1, wherein: The affine transformation matrix includes at least two sub-matrices, each of the sub-matrices corresponds to one of the at least two video frames, and each of the sub-matrices is used to indicate a geometric position of a macroblock in the corresponding video frame.

3. The method according to claim 2, wherein: The first object includes a coding search range for each macroblock; The determining the first object based on the affine transformation matrix comprises: Determine, according to the at least two sub-matrices, a motion vector corresponding to each macroblock, each motion vector indicating a search range; For each macroblock, a search range indicated by a motion vector corresponding to a macroblock is determined as a coding search range of the macroblock, thereby obtaining a coding search range of each macroblock.

4. The method according to claim 2, wherein: The first object comprises the intra-coded frame; The determining the first object based on the affine transformation matrix comprises: Determine, according to the at least two sub-matrices, a first change rate corresponding to each of the sub-matrices; determining the intra-coded frame based on the first change rate; The first change rate corresponding to a submatrix includes: a change rate of a value in the submatrix compared with a value in a previous submatrix of the submatrix.

5. The method according to claim 4, wherein: The determining the intra-coded frame based on the first change rate comprises: For at least one first video frame corresponding to the submatrix whose first change rate is greater than or equal to a first threshold, determining the at least one first video frame as the intra-coding frame; or, For at least one second video frame corresponding to the submatrix whose first change rate is less than the first threshold, a second change rate corresponding to each second video frame is calculated, and the second change rate corresponding to a second video frame includes: a rate of change of a pixel value of a pixel in the second video frame compared to a pixel value of a pixel in a video frame preceding the second video frame; All the second video frames whose second change rate is greater than or equal to a second threshold are determined as the intra-coded frames.

6. The method according to claim 1 or 2, wherein: The acquiring of an affine transformation matrix based on the collected image data of at least two video frames and the first data comprises: Inputting the image data and the first data into an electronic image stabilization algorithm; Obtaining pixel coordinate data of feature points in the at least two video frames from the image data by using the electronic anti-shake algorithm; The affine transformation matrix is ​​calculated based on the pixel coordinate data and the first data through the electronic image stabilization algorithm.

7. A video encoding device, comprising an acquisition module, a determination module and an encoding module; The acquisition module is used to acquire an affine transformation matrix based on the image data of at least two video frames acquired and first data, wherein the affine transformation matrix is ​​used to indicate a mapping relationship between corresponding macroblocks between every two adjacent video frames in the at least two video frames, and the first data is: acceleration data and angular velocity data of the electronic device during the process of acquiring the at least two video frames; The determining module is used to determine the first object based on the affine transformation matrix obtained by the obtaining module; The encoding module is configured to encode the at least two video frames based on the first object determined by the determination module; Wherein, the first object includes at least one of the following: a coding search range for each macroblock in the at least two video frames; An intra-coded frame among the at least two video frames.

8. The device according to claim 7, wherein: The affine transformation matrix includes at least two sub-matrices, each of the sub-matrices corresponds to one of the at least two video frames, and each of the sub-matrices is used to indicate a geometric position of a macroblock in the corresponding video frame.

9. The device according to claim 8, wherein: The first object includes a coding search range for each macroblock; The determination module is specifically used to determine the motion vector corresponding to each macroblock according to the at least two sub-matrices, each motion vector indicating a search range; and for each macroblock, determine the search range indicated by the motion vector corresponding to a macroblock as the encoding search range of the macroblock, to obtain the encoding search range of each macroblock.

10. The device according to claim 8, wherein: The first object comprises the intra-coded frame; The determination module is specifically configured to determine, according to the at least two sub-matrices, a first change rate corresponding to each of the sub-matrices; and determine the internal coding frame based on the first change rate; The first change rate corresponding to a submatrix includes: a change rate of a value in the submatrix compared with a value in a previous submatrix of the submatrix.

11. The device according to claim 7 or 8, wherein: The acquisition module is specifically used to input the image data and the first data into the electronic anti-shake algorithm; and through the electronic anti-shake algorithm, obtain the pixel coordinate data of the feature points in the at least two video frames from the image data; and through the electronic anti-shake algorithm, calculate the affine transformation matrix according to the pixel coordinate data and the first data.

12. An electronic device comprising a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the video encoding method according to any one of claims 1 to 6 are implemented.

13. A readable storage medium storing a program or an instruction, wherein the program or the instruction, when executed by a processor, implements the steps of the video encoding method according to any one of claims 1 to 6.

14. A chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or an instruction to implement the video encoding method according to any one of claims 1 to 6.

15. A computer program product, wherein the program product is stored in a storage medium and is executed by at least one processor to implement the video encoding method according to any one of claims 1 to 6.

Images