Video encoding method, device, storage medium and electronic device

Abstract

The present disclosure provides a video encoding method, apparatus, storage medium, and electronic device. The method includes: obtaining reference frame data for a current frame; determining inter-frame losses between each reference frame in the reference frame data and the current frame; selecting at least one reference frame from the reference frames based on the inter-frame losses; and encoding the current frame using the at least one reference frame. The present disclosure can better balance encoding speed and encoding quality, significantly improving encoding efficiency.

Description

Technical Field

[0001] The present disclosure relates to the field of video coding and decoding, and in particular to a video coding method, a video coding device, an electronic device, and a computer-readable storage medium for dynamically selecting a reference frame based on video features. Background Art

[0002] The image data of a video is encoded by a video encoder based on a specific data compression standard (e.g., Moving Picture Experts Group (MPEG) standard, High Efficiency Video Coding (HEVC), Versatile Video Coding (VVC)), and then stored in a recording medium or transmitted in the form of a bit stream through a communication channel.

[0003] In video coding standards, frames are typically categorized into I-frames (Intra-frames), P-frames (Predicted-frames), and B-frames (Bidirectional-frames) based on their functions and compression efficiency. In the latest video coding standards, to further improve the compression efficiency of P-frames, a special B-frame, GPB (Generalized P and B picture), has been introduced to replace P-frames. With the exception of I-frames, which are self-reconstructed frames, all other frame types reference other frames during encoding. To increase video compression efficiency, various video coding standards and encoders select multiple reference frames to obtain a larger reference area. When encoding a block, it is typically necessary to traverse each reference frame and find the block with the smallest difference from the current block as the best match. Therefore, too few reference frames can lead to a loss in coding compression efficiency, while too many reference frames can also lead to a loss in coding speed.

[0004] For example, standards like HEVC and VVC limit the number of reference frames to a specific frame position. Open-source software like x265 and x264 typically use parameters to control the maximum number of reference frames. In both implementations, the number of reference frames is determined through configuration or parameters before encoding begins, making it difficult for the encoder to strike a balance between encoding efficiency and speed. Summary of the Invention

[0005] The present disclosure provides a reference frame selection method, device, storage medium and electronic device for video coding to at least solve the problems mentioned above.

[0006] According to a first aspect of the present disclosure, a video encoding method is provided, which may include: obtaining reference frame data of a current frame; determining various inter-frame losses between each reference frame in the reference frame data and the current frame; selecting at least one reference frame from the various reference frames based on the various inter-frame losses; and encoding the current frame using the at least one reference frame.

[0007] Optionally, selecting at least one reference frame from the various reference frames based on the various inter-frame losses may include: filtering out inter-frame losses from the various inter-frame losses that are less than a result value of a minimum inter-frame loss multiplied by a threshold, wherein the minimum inter-frame loss is determined from the various inter-frame losses; and selecting a reference frame corresponding to the filtered inter-frame loss from the various reference frames as the at least one reference frame.

[0008] Alternatively, the threshold may be determined based on a distance between the current frame and a nearest reference frame of the current frame.

[0009] Optionally, when the current frame is a P frame or a GPB frame, the reference frame data may include forward reference frame data, wherein determining the inter-frame losses between each reference frame in the reference frame data and the current frame may include: dividing the current frame and the each reference frame into blocks of a predetermined size; for each block of the current frame, performing a motion search in each reference frame in the forward reference frame data to determine a reference block with minimum inter-block loss in each reference frame; for each reference frame, adding the minimum inter-block losses of the reference blocks determined in the reference frame for each block of the current frame to obtain the inter-frame loss between the current frame and the reference frame.

[0010] Optionally, in the case that the current frame is a B frame, the reference frame data may include forward reference frame data and backward reference frame data, wherein determining the inter-frame losses between each reference frame in the reference frame data and the current frame includes: dividing the current frame and the each reference frame into blocks of a predetermined size; performing a motion search in each forward reference frame in the forward reference frame data for each block of the current frame, so as to determine a forward reference block with minimum inter-block loss in each forward reference frame; performing a motion search in each backward reference frame in the backward reference frame data based on pixel values of each block of the current frame and the corresponding forward reference block, so as to determine a backward reference block with minimum inter-block loss in each backward reference frame; and for each backward reference frame, adding the minimum inter-block losses of the backward reference blocks determined in the backward reference frame for each block of the current frame to obtain the inter-frame loss between the current frame and the backward reference frame.

[0011] Optionally, when the current frame is a P frame or a GPB frame, the reference frame corresponding to an inter-frame loss may include a corresponding forward reference frame; when the current frame is a B frame, the reference frame corresponding to an inter-frame loss may include a corresponding forward reference frame and a corresponding backward reference frame.

[0012] Optionally, the inter-frame loss may include one of a frequency domain sum of absolute error (SATD), a squared error (SSE), and a time domain sum of absolute error (SAD).

[0013] According to a second aspect of the present disclosure, a video encoding device is provided, which may include: an acquisition module, configured to acquire reference frame data of a current frame; a selection module, configured to: determine each inter-frame loss between each reference frame in the reference frame data and the current frame, and select at least one reference frame from the each reference frame based on the each inter-frame loss; and an encoding module, configured to encode the current frame using the at least one reference frame.

[0014] Optionally, the selection module can be configured to: filter out inter-frame losses that are less than the result value of the minimum inter-frame loss multiplied by a threshold from the various inter-frame losses, wherein the minimum inter-frame loss is determined from the various inter-frame losses; and select a reference frame corresponding to the filtered inter-frame loss from the various reference frames as the at least one reference frame.

[0015] Alternatively, the threshold may be determined based on a distance between the current frame and a nearest reference frame of the current frame.

[0016] Optionally, when the current frame is a P frame or a GPB frame, the reference frame data may include forward reference frame data, wherein the selection module may be configured to: divide the current frame and the respective reference frames into blocks of predetermined sizes; for each block of the current frame, perform motion search in each reference frame in the forward reference frame data to determine a reference block with minimum inter-block loss in each reference frame; for each reference frame, add the minimum inter-block losses of the reference blocks determined in the reference frame for each block of the current frame to obtain the inter-frame loss between the current frame and the reference frame.

[0017] Optionally, in the case where the current frame is a B frame, the reference frame data may include forward reference frame data and backward reference frame data, wherein the selection module is configured to: divide the current frame and the respective reference frames into blocks of predetermined sizes; for each block of the current frame, perform motion search in each forward reference frame in the forward reference frame data to determine a forward reference block with minimum inter-block loss in each forward reference frame; based on the pixel values of each block of the current frame and the corresponding forward reference block, perform motion search in each backward reference frame in the backward reference frame data to determine a backward reference block with minimum inter-block loss in each backward reference frame; for each backward reference frame, add the minimum inter-block losses of the backward reference blocks determined in the backward reference frame for each block of the current frame to obtain the inter-frame loss between the current frame and the backward reference frame.

[0018] Optionally, when the current frame is a P frame or a GPB frame, the reference frame corresponding to an inter-frame loss may include a corresponding forward reference frame; when the current frame is a B frame, the reference frame corresponding to an inter-frame loss may include a corresponding forward reference frame and a corresponding backward reference frame.

[0019] Optionally, the inter-frame loss may include one of a frequency domain sum of absolute error (SATD), a squared error (SSE), and a time domain sum of absolute error (SAD).

[0020] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: at least one processor; and at least one memory storing computer-executable instructions, wherein the computer-executable instructions, when executed by the at least one processor, prompt the at least one processor to execute the video encoding method described above.

[0021] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, which, when the instructions in the computer-readable storage medium are executed by a processor of a video encoding device / electronic device / server, enables the video encoding device / electronic device / server to perform the video encoding method as described above.

[0022] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, wherein instructions in the computer program product are executed by at least one processor in an electronic device to perform the video encoding method as described above.

[0023] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects: the number of reference frames is dynamically selected based on the video features between frames, so that when encoding different video sequences, a balance between encoding speed and encoding quality can be achieved, thereby greatly improving the encoding efficiency.

[0024] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The accompanying drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute an improper limitation of the present disclosure.

[0026] Figure 1 is a flowchart illustrating a video encoding method for dynamically selecting a reference frame according to an exemplary embodiment of the present disclosure;

[0027] Figure 2 is a block diagram illustrating a video encoding apparatus for dynamically selecting a reference frame according to an exemplary embodiment of the present disclosure;

[0028] Figure 3 FIG. 4 is a structural block diagram illustrating an electronic device for video encoding according to an exemplary embodiment of the present disclosure.

[0029] Figure 4 FIG. 2 is a schematic diagram illustrating an electronic device according to another exemplary embodiment. DETAILED DESCRIPTION

[0030] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0031] It should be noted that the terms "first," "second," and the like in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the numbers used in this manner are interchangeable where appropriate so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The implementation methods described in the following examples do not represent all implementation methods consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with certain aspects of the present disclosure as detailed in the appended claims.

[0032] It should be noted that the phrase "at least one of the items" in this disclosure includes three types of parallel situations: "any one of the items", "a combination of any multiple items of the items", and "all of the items". For example, "including at least one of A and B" includes the following three parallel situations: (1) including A; (2) including B; (3) including A and B. For another example, "performing at least one of step 1 and step 2" includes the following three parallel situations: (1) performing step 1; (2) performing step 2; and (3) performing steps 1 and 2.

[0033] Before describing the embodiments of the present disclosure in detail, some terms or abbreviations that may be involved in the embodiments of the present disclosure are first described.

[0034] In the current video coding standards, the number of reference frames is determined by configuration or parameters before coding begins, and is not dynamically selected based on the characteristics of the video itself. This makes it difficult for the encoder to achieve a good balance between coding efficiency and coding speed. Based on this, the present disclosure proposes a dynamic reference frame selection method based on video characteristics, which dynamically selects reference frames based on the inter-frame loss results obtained through pre-analysis. A good balance between coding speed and coding quality can be achieved for different video sequences. Figure 1 The dynamic reference frame selection method disclosed in the present invention is described in detail.

[0035] Figure 1 is a flowchart illustrating a video encoding method for dynamically selecting a reference frame according to an exemplary embodiment of the present disclosure.

[0036] Reference Figure 1 In step S101, reference frame data of the current frame is obtained. The reference frame data can be stored in the form of a list, an array, etc. For example, the reference frame data can be a reference frame list consisting of multiple video frames.

[0037] For each video frame, if the currently encoded frame is a B-frame or a GPB-frame, the bidirectional reference frame list for that frame can be obtained. If the currently encoded frame is a P-frame, the forward reference frame list for that frame can be obtained. Because the forward reference frame list and the backward reference frame list in the bidirectional reference frame list of a GPB frame are completely consistent, the present disclosure can use the same method when calculating inter-frame loss for GPB frames and P frames.

[0038] In step S102, each inter-frame loss between each reference frame in the reference frame data of the current frame and the current frame is determined. The inter-frame loss can be an indicator such as the Sum of Absolute Transformed Difference (SATD) in the frequency domain, the Sum of Squared Error (SSE) in the error, or the Sum of Absolute Difference (SAD) in the time domain.

[0039] When the current frame is a P frame or a GPB frame, the reference frame corresponding to an inter-frame loss may include a corresponding forward reference frame; when the current frame is a B frame, the reference frame corresponding to an inter-frame loss may include a corresponding forward reference frame and a corresponding backward reference frame.

[0040] Specifically, when the current frame is a P-frame or a GPB-frame, the reference frame data may include forward reference frame data. The inter-frame loss of the current frame (i.e., the P-frame or GPB-frame) may be calculated based on the following operations: dividing the current frame and each reference frame in the reference frame data into blocks of a predetermined size; performing a motion search for each block of the current frame in each reference frame in the forward reference frame data to determine a reference block with minimum inter-block loss in each reference frame; and, for each reference frame, summing the minimum inter-block losses of the reference blocks determined for each block of the current frame in the reference frame to obtain the inter-frame loss between the current frame and the reference frame.

[0041] As an example, when the current frame is a P frame or a GPB frame, downsampling processing can be performed on the current frame and the forward reference frame in the reference frame data, and the downsampled current frame and forward reference frame are divided into multiple blocks. For each block in the current frame, the inter-block loss between the block and each block in the forward reference frame is calculated block by block in the forward reference frame, and the reference block with the minimum inter-block loss with the block is found. Then, for each forward reference frame, the minimum inter-block losses of all blocks of the current frame are added to obtain the inter-frame loss of the current frame relative to the forward reference frame.

[0042] That is to say, each block in the current frame determines a corresponding reference block in each reference frame, and then for each reference frame, the minimum inter-block loss of the determined reference blocks can be summed to obtain the inter-frame loss of the current frame relative to the corresponding reference frame.

[0043] When the current frame is a B frame, the reference frame data may include forward reference frame data and backward reference frame data. The inter-frame loss of the current frame (i.e., the B frame) can be calculated based on the following operations: divide the current frame and each reference frame in the forward reference frame data and the backward reference frame data into blocks of a predetermined size; for each block of the current frame, perform a motion search in each forward reference frame in the forward reference frame data to respectively determine a forward reference block with the minimum inter-block loss in each forward reference frame; based on the pixel values of each block of the current frame and the corresponding forward reference block, perform a motion search in each backward reference frame in the backward reference frame data to respectively determine a backward reference block with the minimum inter-block loss in each backward reference frame; for each backward reference frame, add up the minimum inter-block losses of the respective blocks of the current frame in the backward reference block determined in that backward reference frame to obtain the inter-frame loss between the current frame and that backward reference frame.

[0044] As an example, when the current frame is a B frame, a downsampling process can be performed on the current frame and the forward and backward reference frames in the reference frame data. The downsampled current frame and the forward and backward reference frames are divided into multiple blocks. For each block in the current frame, calculate the inter-block loss between this block and each block in the forward reference frame block by block in the forward reference frame, and find the forward reference block with the minimum inter-block loss with this block. Then, based on the forward reference block and the current block, calculate the inter-block loss between each block in the backward reference frame block by block in the backward reference frame, and find the backward reference block with the minimum inter-block loss. For each backward reference block, add up the minimum inter-block losses of all blocks of the current frame for the backward reference frame to obtain the inter-frame loss of the current frame relative to the backward reference frame. Finally, use the determined backward reference frame and the forward reference frame paired with this backward reference frame as the reference frame of the current frame.

[0045] First, the calculation method of the inter-frame loss of the present disclosure will be specifically described below.

[0046] Define Distortion(p0, p1, b) (where p0, b, and p1 respectively represent frame numbers, and p0 < b <= p1) to represent the inter-frame loss between the current frame b and the forward reference frame p0 and the backward reference frame p1. Here, the inter-frame loss can be an index such as SATD, SSE, or SAD. When b == p1, it can indicate that the current frame b is a GPB / P frame, that is, Distortion(p0, p1, b) represents the inter-frame loss between the current frame b and the forward reference frame p0.

[0047] Taking the inter-frame loss as SATD as an example below, the calculation method of Distortion(p0, p1, b) can be as follows:

[0048] 1. Downsample frames p0, p1, and b to a resolution of 1 / 2, and divide the downsampled frames into 8×8 blocks.

[0049] 2. For the current block I in frame b, perform motion search in frame p0 and find the block with the smallest SATD as the best matching block I'.

[0050] 3. If p1 == b, the SATD from operation 2 is used as the SATD for the current block I. Otherwise, if p1!= b, a motion search is performed in frame p1 using the value 2*I-I' as the search value. The block with the smallest SATD is found as the best match, and this SATD is used as the SATD for the current block I. Here, a*I represents a multiplication operation, multiplying each pixel in block I by a, where a is a constant. I-I' represents a subtraction operation, subtracting pixels in block I from those in block I', point by point.

[0051] 4. Perform motion search on each block in the current frame according to the method in 3 above to find the corresponding best matching block, and based on this, obtain the SATD of each block in the current frame.

[0052] 5. Sum the SATD of each block in the current frame to obtain the SATD of the final frame level (SATD of the current frame), that is, Distortion(p0, p1, b).

[0053] When the current frame is a P frame or a GPB frame, a forward reference frame can be determined according to Distortion (p0, p1, b). When the current frame is a B frame, a forward reference frame and a backward reference frame can be determined according to Distortion (p0, p1, b).

[0054] For each frame in the video to be encoded, assuming that the encoding sequence number of the current frame is A, according to the encoding configuration, the current frame A can obtain its forward reference frame list L0 = {P 0，0 , P 0，1 ,..,P 0，M} and backward reference frame list L1 = {P 1，0 , P 1，1 ,..,P 1，N Here, the parameters in the reference frame list represent the frame numbers of the reference frames.

[0055] For the current frame A, calculate Distortion(P 0，i , P 1，j , A), where 0≤i≤M, 0≤j≤N.

[0056] In step S103 , at least one reference frame is selected from the reference frames in the reference frame list based on the calculated inter-frame losses.

[0057] As an example, inter-frame losses obtained in step S102 may be screened out to obtain inter-frame losses that are less than a result of multiplying a minimum inter-frame loss by a threshold, where the minimum inter-frame loss is determined from the inter-frame losses. Then, a reference frame corresponding to the screened inter-frame loss is selected from the reference frames as at least one reference frame for encoding. Here, the threshold is determined based on the distance between the current frame and the nearest reference frame of the current frame.

[0058] For example, for the current frame A, Distortion(P 0，i , P 1，j , A), where 0≤i≤M, 0≤j≤N, and find the minimum value SATD min Then, from the above Distortion (P 0，i , P 1，j , A) find SATD is less than SATD min ·Th DT All values of , where DT represents the sum of the distances between the current frame and its nearest forward reference frame and backward reference frame,

[0059] Th DT represents the threshold value related to DT. That is, Th DT Th can be set differently according to the sum of the distances between the current frame and its nearest forward reference frame and / or backward reference frame. DT is a threshold greater than 1.

[0060] In the present disclosure, in order to distinguish between GPB type frames and B type frames, for B type frames, DT can be any number from 2 to the maximum number of consecutive B frames. For GPB / P frames, DT=0 can be considered. Different types of frames can obtain different thresholds Th according to different DTs. DT In this way, the GPB type frame can also obtain a specific threshold value separately. However, the above example is only exemplary, and the present disclosure can set a specific Th for the GPB type frame. DT .

[0061] After filtering out all the Distortion (P 0，i’ , P 1，j’ , B), P 0，i’ and P 1，j’ Construct a new forward reference frame list and backward reference frame list.

[0062] In step S104, the current frame is encoded using at least one selected reference frame. 0，i’ and P 1j’The constructed new forward reference frame list and / or new backward reference frame list is used to encode the current frame.

[0063] According to the embodiments of the present disclosure, it is possible to find the reference frame that best matches the current frame from the video frames based on the video features, so as to dynamically adjust the number of reference frames used for encoding. Compared with the existing technology, this avoids the problem of slow encoding speed caused by the need to refer to a fixed number of frames when encoding the current frame. At the same time, since encoding with reference to more effective frames also ensures the encoding quality of the current frame, a better balance between encoding speed and quality is achieved.

[0064] Figure 2 FIG2 is a block diagram illustrating a video encoding apparatus for dynamically selecting a reference frame according to an exemplary embodiment of the present disclosure. It should be understood that the video encoding apparatus can be implemented in any of software, hardware, or a combination of software and hardware. Figure 2 The device shown.

[0065] The video encoding device 200 may include an acquisition module 201, a selection module 202, and an encoding module 203. Each module in the video encoding device 200 may be implemented by one or more modules, and the name of the corresponding module may vary depending on the type of module. In various embodiments, some modules in the video encoding device 500 may be omitted, or additional modules may be included. In addition, the modules / elements according to various embodiments of the present disclosure may be combined to form a single entity, and thus may equivalently perform the functions of the corresponding modules / elements before the combination.

[0066] The acquisition module 201 can acquire reference frame data of the current frame. The reference frame data can be in the form of a list, an array, etc. For example, the reference frame data can be a reference frame list consisting of multiple video frames.

[0067] The selection module 202 may determine inter-frame losses between each reference frame in the reference frame data of the current frame and the current frame, and select at least one reference frame from the reference frames based on the inter-frame losses. The inter-frame losses may include one of a frequency domain sum of absolute error (SATD), a squared error (SSE), and a time domain sum of absolute error (SAD).

[0068] The selection module 202 may filter out inter-frame losses that are less than a result value of a minimum inter-frame loss multiplied by a threshold from among the determined inter-frame losses, wherein the minimum inter-frame loss is determined from among the inter-frame losses, and then select a reference frame corresponding to the filtered inter-frame loss from among the reference frames as at least one reference frame for encoding.

[0069] Here, the threshold may be determined based on a distance between the current frame and a nearest reference frame of the current frame.

[0070] When the current frame is a P frame or a GPB frame, the reference frame corresponding to an inter-frame loss may include a corresponding forward reference frame; when the current frame is a B frame, the reference frame corresponding to an inter-frame loss may include a corresponding forward reference frame and a corresponding backward reference frame.

[0071] When the current frame is a P frame or a GPB frame, the reference frame data of the current frame may include forward reference frame data based on the encoding configuration. The selection module 202 may divide the current frame and each reference frame into blocks of predetermined sizes, and for each block of the current frame, perform a motion search in each reference frame in the forward reference frame data to determine a reference block with minimum inter-block loss in each reference frame. For each reference frame, the minimum inter-block loss of the reference blocks determined for each block of the current frame in the reference frame is summed to obtain the inter-frame loss between the current frame and the reference frame.

[0072] As an example, when the current frame is a P frame or a GPB frame, the selection module 202 may perform downsampling processing on the current frame and the forward reference frame in the reference frame data, divide the downsampled current frame and the forward reference frame into multiple blocks, and for each block in the current frame, calculate the inter-block loss between the block and each block in the forward reference frame block by block in the forward reference frame, find the reference block with the minimum inter-block loss with the block, and then add the minimum inter-block losses of all blocks in the current frame to obtain the inter-frame loss of the current frame relative to the forward reference frame.

[0073] In the case where the current frame is a B frame, the reference frame data of the current frame may include forward reference frame data and backward reference frame data based on the encoding configuration. The selection module 202 may divide the current frame and each reference frame into blocks of a predetermined size, and for each block of the current frame, perform a motion search in each forward reference frame in the forward reference frame data to determine a forward reference block with minimum inter-block loss in each forward reference frame, and perform a motion search in each backward reference frame in the backward reference frame data based on the pixel values of each block of the current frame and the corresponding forward reference block to determine a backward reference block with minimum inter-block loss in each backward reference frame. Then, for each backward reference frame, the minimum inter-block losses of the backward reference blocks determined in the backward reference frame for each block of the current frame are added together to obtain the inter-frame loss between the current frame and the backward reference frame.

[0074] As an example, in the case where the current frame is a B frame, the selection module 202 may perform downsampling processing on the current frame and the forward and backward reference frames in the reference frame data, divide the downsampled current frame and the forward and backward reference frames into multiple blocks, and for each block in the current frame, calculate the inter-block loss between the block and each block in the forward reference frame block by block in the forward reference frame, and find the forward reference block with the minimum inter-block loss with the block, and then calculate the inter-block loss between the forward reference block and the current block and each block in the backward reference frame block by block in the backward reference frame, and find the backward reference block with the minimum inter-block loss, and add the minimum inter-block losses of all blocks of the current frame for the backward reference frame to obtain the inter-frame loss of the current frame relative to the backward reference frame.

[0075] The encoding module 203 may encode the current frame using the selected at least one reference frame.

[0076] The above has been referenced Figure 1 The operations and functions of the various modules of the video encoding apparatus 200 have been described in detail and will not be repeated here.

[0077] Figure 3 This is a block diagram illustrating the structure of an electronic device for video encoding according to an exemplary embodiment of the present disclosure. The electronic device 300 may be, for example, a smartphone, a tablet computer, an MP4 (Moving Picture Experts Group Audio Layer IV) player, a laptop computer, or a desktop computer. The electronic device 300 may also be referred to as a user device, a portable terminal, a laptop terminal, a desktop terminal, or other similar names.

[0078] Typically, the electronic device 300 includes a processor 301 and a memory 302 .

[0079] The processor 301 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 301 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field Programmable Gate Array), or PLA (Programmable Logic Array). The processor 301 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 301 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 301 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0080] The memory 302 may include one or more computer-readable storage media, which may be non-transitory. The memory 302 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices and flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 302 is used to store at least one instruction, which is used to be executed by the processor 301 to implement the present disclosure. Figure 1 The method embodiment shown provides a video encoding method.

[0081] In some embodiments, electronic device 300 may optionally include a peripheral device interface 303 and at least one peripheral device. Processor 301, memory 302, and peripheral device interface 303 may be connected via a bus or signal lines. Each peripheral device may be connected to peripheral device interface 303 via a bus, signal lines, or circuit boards. Specifically, the peripheral device may include at least one of a radio frequency circuit 304, a touchscreen display 305, a camera 306, an audio circuit 307, a positioning component 308, and a power supply 309.

[0082] The peripheral device interface 303 can be used to connect at least one I / O (Input / Output)-related peripheral device to the processor 301 and the memory 302. In some embodiments, the processor 301, the memory 302, and the peripheral device interface 303 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 301, the memory 302, and the peripheral device interface 303 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0083] The RF circuit 304 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 304 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 304 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. Optionally, the RF circuit 304 includes an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, and the like. The RF circuit 304 can communicate with other terminals via at least one wireless communication protocol. Such wireless communication protocols include, but are not limited to, metropolitan area networks (MANs), various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks (WLANs), and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF circuit 304 may also include circuits related to NFC (Near Field Communication), which is not limited in this disclosure.

[0084] Display screen 305 is used to display a user interface (UI). This UI may include graphics, text, icons, videos, or any combination thereof. When display screen 305 is a touchscreen display, it is also capable of collecting touch signals on or above the surface of display screen 305. These touch signals can be input as control signals to processor 301 for processing. In this case, display screen 305 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there can be one display screen 305, located on the front panel of electronic device 300. In other embodiments, there can be at least two display screens 305, located on different surfaces of terminal 300 or in a foldable design. In still other embodiments, display screen 305 can be a flexible display, located on a curved or foldable surface of terminal 300. Furthermore, display screen 305 can be configured as a non-rectangular, irregular shape, also known as a special-shaped screen. Display screen 305 can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0085] The camera assembly 306 is used to capture images or videos. Optionally, the camera assembly 306 includes a front camera and a rear camera. Typically, the front camera is set on the front panel of the terminal, and the rear camera is set on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions. In some embodiments, the camera assembly 306 may also include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.

[0086] The audio circuit 307 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals to be input into the processor 301 for processing, or input into the radio frequency circuit 304 to achieve voice communication. For the purpose of stereo sound collection or noise reduction, there can be multiple microphones, which are respectively arranged at different parts of the terminal 300. The microphone can also be an array microphone or an omnidirectional collection microphone. The speaker is used to convert the electrical signals from the processor 301 or the radio frequency circuit 304 into sound waves. The speaker can be a traditional thin film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert the electrical signals into sound waves audible to humans, but also convert the electrical signals into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 307 may also include a headphone jack.

[0087] The positioning component 308 is used to locate the current geographic location of the electronic device 300 to implement navigation or LBS (Location Based Service). The positioning component 308 can be a positioning component based on the US GPS (Global Positioning System), China's Beidou system, Russia's Greninja system, or the European Union's Galileo system.

[0088] Power supply 309 is used to power the various components of electronic device 300. Power supply 309 can be AC power, DC power, a disposable battery, or a rechargeable battery. When power supply 309 includes a rechargeable battery, the rechargeable battery can support wired charging or wireless charging. The rechargeable battery can also be used to support fast charging technology.

[0089] In some embodiments, the electronic device 300 further includes one or more sensors 310 , including but not limited to: an acceleration sensor 311 , a gyroscope sensor 312 , a pressure sensor 313 , a fingerprint sensor 314 , an optical sensor 315 , and a proximity sensor 316 .

[0090] The accelerometer 311 can detect the magnitude of acceleration along the three coordinate axes of the coordinate system established by the terminal 300. For example, the accelerometer 311 can be used to detect the components of gravity acceleration along the three coordinate axes. The processor 301 can control the touch screen display 305 to display the user interface in a landscape or portrait view based on the gravity acceleration signal collected by the accelerometer 311. The accelerometer 311 can also be used to collect game or user motion data.

[0091] The gyroscope sensor 312 can detect the orientation and rotation angle of the terminal 300. It can also work with the accelerometer 311 to collect the user's 3D movements on the terminal 300. Based on the data collected by the gyroscope sensor 312, the processor 301 can implement the following functions: motion sensing (for example, changing the UI based on the user's tilt operation), image stabilization during shooting, game control, and inertial navigation.

[0092] The pressure sensor 313 can be installed on the side frame of the terminal 300 and / or below the touch screen display 305. When the pressure sensor 313 is installed on the side frame of the terminal 300, it can detect the user's grip signal of the terminal 300. The processor 301 performs left and right hand recognition or shortcut operations based on the grip signal collected by the pressure sensor 313. When the pressure sensor 313 is installed below the touch screen display 305, the processor 301 controls the operable controls on the UI based on the user's pressure operation on the touch screen display 305. Operable controls include at least one of button controls, scroll bar controls, icon controls, and menu controls.

[0093] The fingerprint sensor 314 is used to collect the user's fingerprint, and the processor 301 identifies the user's identity based on the fingerprint collected by the fingerprint sensor 314, or the fingerprint sensor 314 identifies the user's identity based on the collected fingerprint. When the user's identity is identified as a trusted identity, the processor 301 authorizes the user to perform relevant sensitive operations, which include unlocking the screen, viewing encrypted information, downloading software, making payments, and changing settings. The fingerprint sensor 314 can be set on the front, back, or side of the electronic device 300. When a physical button or manufacturer logo is set on the electronic device 300, the fingerprint sensor 314 can be integrated with the physical button or manufacturer logo.

[0094] The optical sensor 315 is used to detect ambient light intensity. In one embodiment, the processor 301 can control the display brightness of the touchscreen display 305 based on the ambient light intensity detected by the optical sensor 315. Specifically, when the ambient light intensity is high, the display brightness of the touchscreen display 305 is increased; when the ambient light intensity is low, the display brightness of the touchscreen display 305 is decreased. In another embodiment, the processor 301 can also dynamically adjust the shooting parameters of the camera assembly 306 based on the ambient light intensity detected by the optical sensor 315.

[0095] Proximity sensor 316, also known as a distance sensor, is typically located on the front panel of electronic device 300. Proximity sensor 316 is used to detect the distance between the user and the front of electronic device 300. In one embodiment, when proximity sensor 316 detects that the distance between the user and the front of electronic device 300 is gradually decreasing, processor 301 controls touchscreen display 305 to switch from the screen-on state to the screen-off state. When proximity sensor 316 detects that the distance between the user and the front of electronic device 300 is gradually increasing, processor 301 controls touchscreen display 305 to switch from the screen-off state to the screen-on state.

[0096] Those skilled in the art will understand that Figure 3 The structure shown in the figure does not constitute a limitation on the electronic device 300, and the electronic device 300 may include more or fewer components than shown in the figure, or combine certain components, or adopt a different component arrangement.

[0097] Figure 4 FIG. 4 is a block diagram of another electronic device 400. For example, the electronic device 400 may be provided as a server. Figure 4 The electronic device 400 includes one or more processors 410 and a memory 420. The memory 420 may include one or more programs for executing the above video encoding method. The electronic device 400 may also include a power supply component 430 configured to perform power management of the electronic device 400, a wired or wireless network interface 440 configured to connect the electronic device 400 to a network, and an input / output (I / O) interface 450. The electronic device 400 may operate based on an operating system stored in the memory 420, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, or the like.

[0098] According to an embodiment of the present disclosure, a computer-readable storage medium storing instructions may also be provided, wherein when the instructions are executed by at least one processor, the at least one processor is prompted to perform the video encoding method according to the present disclosure. Examples of computer-readable storage media include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as, multimedia card, secure digital (SD) card or ultra-fast digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, any other device configured to store the computer program and any associated data, data files and data structures in a non-transitory manner and provide the computer program and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the above-mentioned computer-readable storage medium can be run in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. In addition, in one example, the computer program and any associated data, data files and data structures are distributed on a networked computer system so that the computer program and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers.

[0099] According to an embodiment of the present disclosure, a computer program product may be provided. Instructions in the computer program product may be executed by a processor of a computer device to implement the above-mentioned video encoding method.

[0100] The video encoding method, apparatus, electronic device, and computer-readable storage medium disclosed herein can dynamically adjust the number of reference frames based on video characteristics, achieving a better balance between encoding speed and quality. Using this method, the average loss in BD-Rate (an objective metric for measuring encoding performance) measured at high-quality online settings was only 0.1%, while simultaneously saving approximately 20% in encoding time, significantly improving encoding efficiency.

[0101] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the following claims.

[0102] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A video encoding method, comprising: Get the reference frame data of the current frame; determining respective inter-frame losses between respective reference frames in the reference frame data and the current frame; selecting at least one reference frame from the reference frames based on the respective inter-frame losses; encoding the current frame using the at least one reference frame, Wherein, when the current frame is a B frame, the reference frame data includes forward reference frame data and backward reference frame data, and determining each inter-frame loss between each reference frame in the reference frame data and the current frame includes: dividing the current frame and the reference frames into blocks of a second predetermined size; For each block of the current frame, perform motion search in each forward reference frame in the forward reference frame data to determine a forward reference block with minimum inter-block loss in each forward reference frame; performing a motion search in each backward reference frame in the backward reference frame data based on pixel values of each block of the current frame and a corresponding forward reference block, so as to determine a backward reference block with minimum inter-block loss in each backward reference frame; For each backward reference frame, the minimum inter-block losses of the backward reference blocks of each block of the current frame determined in the backward reference frame are added together to obtain the inter-frame loss between the current frame and the backward reference frame.

2. The video encoding method according to claim 1, wherein: Selecting at least one reference frame from the reference frames based on the inter-frame losses includes: Filtering out inter-frame losses that are smaller than a result value of a minimum inter-frame loss multiplied by a threshold value from the inter-frame losses, wherein the minimum inter-frame loss is determined from the inter-frame losses; A reference frame corresponding to the filtered inter-frame loss is selected from the reference frames as the at least one reference frame.

3. The video encoding method according to claim 2, wherein: The threshold is determined based on a distance between the current frame and a nearest reference frame of the current frame.

4. The video encoding method according to claim 1, wherein: In the case that the current frame is a P frame or a GPB frame, the reference frame data includes forward reference frame data, The determining of each inter-frame loss between each reference frame in the reference frame data and the current frame includes: dividing the current frame and the reference frames into blocks of a first predetermined size; For each block of the current frame, perform motion search in each reference frame in the forward reference frame data to determine a reference block with minimum inter-block loss in each reference frame; For each reference frame, the minimum inter-block losses of the reference blocks of each block of the current frame determined in the reference frame are added together to obtain the inter-frame loss between the current frame and the reference frame.

5. The video encoding method according to claim 1, wherein: When the current frame is a P frame or a GPB frame, the reference frames corresponding to an inter-frame loss include a corresponding forward reference frame; when the current frame is a B frame, the reference frames corresponding to an inter-frame loss include a corresponding forward reference frame and a corresponding backward reference frame.

6. The video encoding method according to claim 1, wherein: The inter-frame loss includes one of a frequency domain sum of absolute error (SATD), a square sum of error (SSE), and a time domain sum of absolute error (SAD).

7. A video encoding apparatus, comprising: an acquisition module, configured to acquire reference frame data of a current frame; a selection module configured to: determine respective inter-frame losses between respective reference frames in the reference frame data and the current frame, and select at least one reference frame from the respective reference frames based on the respective inter-frame losses; an encoding module configured to encode the current frame using the at least one reference frame, Wherein, when the current frame is a B frame, the reference frame data includes forward reference frame data and backward reference frame data, and the selection module is configured to: dividing the current frame and the respective reference frames into blocks of predetermined sizes; For each block of the current frame, perform motion search in each forward reference frame in the forward reference frame data to determine a forward reference block with minimum inter-block loss in each forward reference frame; performing a motion search in each backward reference frame in the backward reference frame data based on pixel values of each block of the current frame and a corresponding forward reference block, so as to determine a backward reference block with minimum inter-block loss in each backward reference frame; For each backward reference frame, the minimum inter-block losses of the backward reference blocks of each block of the current frame determined in the backward reference frame are added together to obtain the inter-frame loss between the current frame and the backward reference frame.

8. The video encoding device according to claim 7, wherein: The selection module is configured as follows: Filtering out inter-frame losses that are smaller than a result value of a minimum inter-frame loss multiplied by a threshold value from the inter-frame losses, wherein the minimum inter-frame loss is determined from the inter-frame losses; A reference frame corresponding to the filtered inter-frame loss is selected from the reference frames as the at least one reference frame.

9. The video encoding device according to claim 8, wherein: The threshold is determined based on a distance between the current frame and a nearest reference frame of the current frame.

10. The video encoding device according to claim 7, wherein: In the case that the current frame is a P frame or a GPB frame, the reference frame data includes forward reference frame data, Among them, the selection module is configured as follows: dividing the current frame and the respective reference frames into blocks of predetermined sizes; For each block of the current frame, perform motion search in each reference frame in the forward reference frame data to determine a reference block with minimum inter-block loss in each reference frame; For each reference frame, the minimum inter-block losses of the reference blocks of each block of the current frame determined in the reference frame are added together to obtain the inter-frame loss between the current frame and the reference frame.

11. The video encoding device according to claim 7, wherein: When the current frame is a P frame or a GPB frame, the reference frames corresponding to an inter-frame loss include a corresponding forward reference frame; when the current frame is a B frame, the reference frames corresponding to an inter-frame loss include a corresponding forward reference frame and a corresponding backward reference frame.

12. The video encoding device according to claim 7, wherein: The inter-frame loss includes one of a frequency domain sum of absolute error (SATD), a square sum of error (SSE), and a time domain sum of absolute error (SAD).

13. An electronic device, characterized in that: include: at least one processor; at least one memory storing computer-executable instructions, When the computer-executable instructions are executed by the at least one processor, the computer-executable instructions cause the at least one processor to perform the video encoding method according to any one of claims 1 to 6.

14. A computer-readable storage medium, when the instructions in the computer-readable storage medium are executed by a processor of a video encoding device / electronic device / server, enables the video encoding device / electronic device / server to perform the video encoding method according to any one of claims 1 to 6.

15. A computer program product, characterized in that The instructions in the computer program product are executed by at least one processor in an electronic device to perform the video encoding method according to any one of claims 1 to 6.

Images