Video coding optimization method and system based on reference frame thumbnail, encoder, medium and program product

By using reference frame thumbnails in video encoding for preliminary search and precise search in original frames, combined with coarse and fine strategies, the contradiction between video encoder in search range settings is solved, and the efficiency and accuracy of motion estimation is improved. Reduces the computational volume and improves coding efficiency.

CN118984389APending Publication Date: 2024-11-19VERISILICON MICROELECTRONICS (HAINAN) CO LTD +4
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202411108139.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-13
Publication Date
2024-11-19

AI Technical Summary

Technical Problem

When setting the search range, it is difficult for existing video encoders to find a balance between accuracy and calculation amount, resulting in the possibility of missing the most similar blocks or the calculation amount is too large, affecting the encoding efficiency.

Method used

By generating thumbnails of reference frames for preliminary search, the initial motion vector is determined, and then the accurate search in the original frame is carried out, combining coarse and fine search strategies to reduce the computational amount and improve coding quality.

Benefits of technology

It effectively improves the efficiency and accuracy of motion estimation, reduces storage space and computing costs, reduces unnecessary calculations, and improves coding efficiency and compression rate.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118984389A_ABST
    Figure CN118984389A_ABST
Patent Text Reader

Abstract

The invention provides a video coding optimization method and system based on a reference frame thumbnail, an encoder, a medium and a program product. The function of Inter prediction unit motion vector search in Inter frame video coding is decomposed into two steps of rough search and fine search. A path of 1 / N thumbnail is added when a reference frame is output, and an initial motion vector is provided by roughly searching the thumbnail when video coding begins, so that the accurate search range of a subsequent Inter prediction unit is greatly reduced, and the total data volume and the calculation amount which need to be read in for completing the motion vector search of the Inter prediction unit are greatly reduced. Besides, due to the fact that the pre-judging block is suitable for the Inter prediction unit or the Intra prediction unit, each block value only needs to go through one of the processes, each block does not need to go through the Inter prediction unit and the Intra prediction unit at the same time, unnecessary calculation on each block is avoided, the total calculation amount is reduced, and the coding efficiency is improved. Therefore, rough and fine search strategies are combined, and the efficiency and accuracy of motion estimation are improved. The calculation amount is reduced through quick search on a thumbnail, then the coding quality is ensured through fine search on an original frame, and the method is an effective means for improving the compression efficiency and reducing the coding time in video coding.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of video coding technology, and in particular to a video coding optimization method, system, encoder, medium and program product based on reference frame thumbnails. Background Art

[0002] Video compression technology involves dividing video frames into coding blocks and analyzing the correlation between blocks. These correlations and differences are encoded using coding technology to obtain output binary data. Through coding technology, the size of video data can be significantly reduced, thereby saving storage space and bandwidth. The video encoder converts a sequence of video frames composed of pixels into a binary data stream. The purpose of this process is to compress the data in order to save storage space and speed up the transmission speed in computers and networks.

[0003] The "correlation between blocks" involved here includes the following two types: Intra blocks that are similar only to adjacent coding blocks, and Inter blocks that are similar only to corresponding coding blocks in the reference frame. Intra blocks produce similar blocks by interpolation (Intra Prediction) in a given direction of adjacent boundaries, and Inter blocks produce similar blocks by searching for motion estimation (Inter Prediction) of the reference frame relative to the current block. The encoder needs to use one of these two methods to find the most similar block for each block in the current frame. This similarity is expressed by the SAD (Sum of Absolute Differences) function. The smaller the SAD, the smaller the amount of differential coding. If the encoder can always find the smallest and optimal SAD as quickly as possible, the better the performance of the encoder. Large blocks can continue to be divided into small blocks to find smaller SADs until the smallest block specified by the protocol (the number of layers and range limits of block division are specified by the specific protocol).

[0004] The general process of existing video encoders is as follows Figure 1 As shown in the figure, it can be clearly observed that the traditional motion estimation processing method usually follows the following steps: First, the encoder sets the maximum search range according to its own performance and storage. All data of the reference frame corresponding to the coordinates of the read-in original macroblock within this range are read in; then, the encoder starts from the upper left corner of the reference frame data (i.e., the negative maximum value of the x,y relative coordinates) and gradually searches to the lower right corner (i.e., the positive maximum value of the x,y relative coordinates) to find the block corresponding to the minimum absolute difference (SAD). Calculate the difference between the corresponding block coordinates found and the current block coordinates (i.e., the motion vector MV), record the corresponding reference frame number and generate a prediction value so that it can be passed to the subsequent processing stage.

[0005] However, this method has some problems: if the search range is not set large enough, the block most similar to the current block (i.e., the real similar block) may be missed, because the search range is limited and the real similar block is likely to be outside the search range. If the search range is large, for example, the search range covers the entire image, each block must be matched with the reference data one by one, and the amount of reference frame data to be read and calculated is large, resulting in a large amount of calculation, and the processing speed of the encoder will be slow. Summary of the invention

[0006] In view of the above-mentioned shortcomings of the prior art, the purpose of the present application is to provide a video encoding optimization method, system, encoder, medium and program product based on reference frame thumbnails, which are used to solve the technical problem of the contradiction between search accuracy and computational complexity when setting the search range.

[0007] To achieve the above-mentioned purpose and other related purposes, the first aspect of the present application provides a video encoding optimization method based on reference frame thumbnails, including: generating and outputting a corresponding reference frame thumbnail after the reference frame is downsampled and reduced in size when outputting a reference frame; when inter-frame coding starts, after dividing the frame to be encoded into macroblocks, the macroblocks are downsampled into corresponding original macroblocks based on the same thumbnail ratio as the reference frame thumbnail, and the reference frame thumbnail is read in at the same time; a preliminary search is performed in the reference frame thumbnail to search for the reference block with the highest similarity to the original macroblock in the reference frame thumbnail through similarity calculation, and the corresponding as the initial motion vector; determine whether the similarity value corresponding to the reference block with the highest similarity meets the preset similarity requirement; if the preset similarity requirement is met, input the original macroblock and the initial motion vector into the inter-frame prediction unit for processing, and perform an accurate search in the reference frame with the position indicated by the initial motion vector as the starting point to find the best matching prediction block in the reference frame; if the preset similarity requirement is not met, input the corresponding original macroblock into the intra-frame prediction unit for processing; or, further divide the original macroblock into sub-blocks and continue to perform a preliminary search in the reference frame thumbnail until the preset similarity requirement is met.

[0008] In some embodiments of the first aspect of the present application, the searching for the reference block having the highest similarity with the original macroblock in the reference frame thumbnail through similarity calculation includes: using the SAD function to calculate the similarity between each reference block in the reference frame thumbnail and the original macroblock; the reference block corresponding to the calculated minimum SAD value has the highest similarity with the original macroblock.

[0009] In some embodiments of the first aspect of the present application, the further division of the original macroblock into sub-blocks and the continued preliminary search in the reference frame thumbnail until the preset similarity requirement is met include: continuously dividing the original macroblock into sub-blocks of smaller sizes, repeating the preliminary search in the reference frame thumbnail and determining the initial motion vector until the SAD value of the sub-block is lower than a preset threshold.

[0010] In some embodiments of the first aspect of the present application, the size of the reference frame thumbnail is 1 / N of the original size of the reference frame; the size of the original macroblock after thumbnail is 1 / N of the macroblock size before thumbnail; and N is a reduction coefficient suitable for calculation.

[0011] In some embodiments of the first aspect of the present application, the range of the precise search includes a region from -N / 2 to N / 2 around the position indicated by the initial motion vector.

[0012] In some embodiments of the first aspect of the present application, the method includes: downsampling the reference frame through a pixel averaging method or a filtering method to generate the reference frame thumbnail, and downsampling the macroblocks obtained by dividing the frame to be encoded to generate the original macroblocks.

[0013] To achieve the above-mentioned purpose and other related purposes, the second aspect of the present application provides a video encoding optimization system based on reference frame thumbnails, including: a downsampling module, which is used to generate and output a corresponding reference frame thumbnail after the reference frame is downsampled and reduced in size when outputting a reference frame; after dividing the frame to be encoded into macroblocks, the macroblocks are thumbnailed into corresponding original macroblocks based on the same thumbnail ratio as the reference frame thumbnail, and the reference frame thumbnail is read in at the same time; a motion vector module is used to perform a preliminary search in the reference frame thumbnail to search for the reference block with the highest similarity to the original macroblock in the reference frame thumbnail through similarity calculation, and determine the corresponding motion vector as an initial motion vector; a judgment module, used to judge whether the similarity value corresponding to the reference block with the highest similarity meets the preset similarity requirement; a coding prediction module, used to input the original macroblock and the initial motion vector into the inter-frame prediction unit for processing when it is judged that the preset similarity requirement is met, and perform a precise search in the reference frame with the position indicated by the initial motion vector as the starting point to find the best matching prediction block in the reference frame; when it is judged that the preset similarity requirement is not met, input the corresponding original macroblock into the intra-frame prediction unit for processing; or, after further dividing the original macroblock into sub-blocks, continue to perform a preliminary search in the reference frame thumbnail until the preset similarity requirement is met.

[0014] To achieve the above-mentioned purpose and other related purposes, the third aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the video encoding optimization method based on reference frame thumbnails is implemented.

[0015] To achieve the above-mentioned purpose and other related purposes, the fourth aspect of the present application provides a computer program product, which includes a computer program code. When the computer program code runs on a computer, the computer implements the video encoding optimization method based on reference frame thumbnails.

[0016] To achieve the above-mentioned purpose and other related purposes, the fifth aspect of the present application provides a video encoder, including a memory, a processor and a computer program stored in the memory; the processor executes the computer program to implement the video encoding optimization method based on reference frame thumbnails.

[0017] As described above, the video coding optimization method, system, encoder, medium and program product based on reference frame thumbnails of the present application combine coarse and fine search strategies to improve the efficiency and accuracy of motion estimation. By reducing the amount of calculation by quickly searching on thumbnails, and then ensuring the encoding quality by fine searching on the original frames, this method is an effective means to improve compression efficiency and reduce encoding time in video encoding. The specific beneficial effects are as follows:

[0018] (1) By adding a 1 / N thumbnail when outputting the reference frame, it is a fixed 1 / N operation that can be achieved by simple addition and subtraction operations. The required storage space is 1 / N2 of the original space. The generation and storage cost of the thumbnail is relatively low. Therefore, in this embodiment, adding a 1 / N thumbnail is a very low-cost optimization method.

[0019] (2) By providing the initial motion vector, the search range of the Inter prediction unit is greatly reduced, and the amount of data and calculation required to be read are also greatly reduced.

[0020] (3) Since it is predicted that the block is suitable for the Inter prediction unit or the Intra prediction unit, each block only needs to go through one of the processes. It is not necessary for each block to go through the Inter prediction unit and the Intra prediction unit at the same time. However, the existing encoder performs full search and full calculation. Therefore, the encoder in this embodiment can avoid unnecessary calculation for each block, reduce the overall calculation amount and improve the coding efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 A schematic diagram showing a video encoding process in the prior art.

[0022] Figure 2A A schematic diagram showing the flow of a video encoding optimization method based on reference frame thumbnails in one embodiment of the present application.

[0023] Figure 2BA schematic diagram showing a flow chart of video encoding optimization processing with an additional reference frame thumbnail in one embodiment of the present application.

[0024] Figure 3A A partial schematic diagram of an original frame in an embodiment of the present application is shown.

[0025] Figure 3B A partial schematic diagram of a thumbnail frame in an embodiment of the present application is shown.

[0026] Figure 4 A distribution diagram showing the calculation of the initial SAD of the full thumbnail in one embodiment of the present application is shown.

[0027] Figure 5 A schematic diagram showing the distribution of motion vectors (MVs) in an image according to an embodiment of the present application is shown.

[0028] Figure 6 A schematic diagram showing the structure of a video encoding optimization system based on reference frame thumbnails in one embodiment of the present application.

[0029] Figure 7 A schematic diagram showing the structure of a video encoder in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0030] The following describes the embodiments of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0031] Before further describing the present invention in detail, the nouns and terms involved in the embodiments of the present invention are explained. The nouns and terms involved in the embodiments of the present invention are applicable to the following interpretations:

[0032] <1> Video compression: Video compression technology involves dividing video frames into coding blocks and analyzing the correlation between blocks. These correlations and differences are encoded using coding technology to obtain output binary data. Coding technology can significantly reduce the size of video data, thereby saving storage space and bandwidth. The video encoder converts a sequence of video frames composed of pixels into a binary data stream. The purpose of this process is to compress data in order to save storage space and speed up the transmission speed in computers and networks. The coding blocks referred to here include Intra blocks (intra-frame prediction blocks) and Inter blocks (inter-frame prediction blocks).

[0033] <2> Intra block (intra-frame prediction block): It has similarity only with adjacent coding blocks. For example, if the coding block in the image is a texture or gradient color, then this coding block is an Intra block. Generally speaking, the first frame of a video consists of all Intra blocks, which is called an I frame.

[0034] <3> Inter block (inter-frame prediction block): It only has similarities with the corresponding coding blocks in other frames, such as moving objects or moving backgrounds. The motion direction and distance between the current coding block and the reference coding block are determined by the motion vector (MV). The frame with this coding block is called a P frame or a B frame.

[0035] <4> P frame (Predictive-coded Frame): A frame that depends on an I frame or a previous P frame, mainly including Inter blocks. It uses motion compensation to predict and encode the difference between the frame and the reference frame. P frames achieve compression by predicting motion vectors to reduce temporal redundancy. Motion vectors indicate the movement of objects in the picture from one frame to the next, and P frames only store these moving parts instead of the entire frame. P frames are smaller than I frames because they store the changing parts instead of the complete frame.

[0036] <5> B frame (Bi-directional predictive-coded Frame): A bidirectional predictive frame, mainly including Inter blocks. It depends on the previous and next I frames or P frames for encoding. B frames use motion compensation in two directions to predict the frame content, that is, forward reference to the most recent I frame or P frame, and backward reference to a later I frame or P frame. Since B frames can utilize information in two directions, they can usually achieve higher compression rates, but decoding B frames requires more computing resources because they need to access the previous and next frames. B frames do not exist independently in the video sequence, and their existence is completely dependent on the reference frame.

[0037] <6> I-frame (Intra-coded Frame): can only be composed of Intra blocks. It is a key frame that is completely independent of other frames and does not rely on other frames for decoding. I-frame can only be predicted using adjacent coded blocks, and the predicted content is obtained by interpolation of adjacent points in one direction. Intra-frame prediction efficiency is generally lower than inter-frame prediction, but this redundancy of data allows I-frames to provide a reference point for the video sequence, and other types of frames (P-frames and B-frames) can rely on I-frames for prediction and encoding. Because I-frames do not rely on other frames, they are usually larger than other types of frames and require more storage space and bandwidth.

[0038] <7> Macroblock (MB, CTB...) coded frames always divide the frame into fixed-size macroblocks in the order of rast_scan, and then Intra blocks or Inter blocks can only be macroblocks or sub-blocks divided from macroblocks. The size of macroblocks is specified by the protocol.

[0039] Before explaining the video encoding optimization method based on reference frame thumbnails in the embodiment of the present application, it is necessary to first Figure 1 The Inter prediction process in the existing encoder is explained in detail. The Inter prediction process is generally as follows:

[0040] Step S101: The encoder reads the current video frame and divides it into raster-scanned macroblocks. These macroblocks are the basic units for motion estimation and compensation.

[0041] Step S102: The encoder selects one or more previously encoded frames as reference frames. These frames usually include the most recently encoded frames because they are closest to the current frame in time.

[0042] Step S103: The encoder searches for the area in the reference frame that is most similar to the current coding block. This process is called motion estimation, which aims to find the position of the best matching block, which involves calculating the motion vector (MV).

[0043] Step S104: The encoder determines the most accurate motion vector by minimizing SAD or minimizing PSNR (Peak Signal-to-Noise Ratio).

[0044] Step S105: The encoder extracts a corresponding block from the reference frame as a prediction block based on the motion vector.

[0045] Step S106: The encoder calculates the difference between the current coding block and the prediction block as a residual; the residual is new information contained in the current video frame and needs to be encoded and transmitted.

[0046] Step S107: The residual is usually transformed (such as DCT-Discrete Cosine Transform) and quantized to further reduce the amount of data.

[0047] Step S108: The transformed and processed residual data is further compressed by entropy coding (such as CABAC, Context-Adaptive Binary Arithmetic Coding), and the final coding block includes information such as motion vector, prediction block index, residual data, etc., which is encoded and output to the bit stream.

[0048] Step S109: After the encoding is completed, the current frame is used as a reference frame for the subsequent frame, and the reference frame queue is updated.

[0049] Figure 2A A flowchart of a video encoding optimization method based on reference frame thumbnails in an embodiment of the present invention is shown, comprising the following steps:

[0050] Step S21: when outputting a reference frame, generating and outputting a corresponding reference frame thumbnail after the reference frame is downsampled and reduced in size.

[0051] For example, in combination Figure 2B Taking the video encoding optimization process with one reference frame thumbnail added as an example, the reference frame can be downsampled by pixel averaging or filtering to reduce the width and height of the original reference frame to 1 / N of the original. The data volume of the generated reference frame thumbnail is 1 / N of the original reference frame. 2 , N is the reduction factor suitable for the operation. The reduction factor is used to reduce or adjust the size of a quantity. It can be an exponent of 2, a fraction (any positive number less than 1, such as 1 / 2, 1 / 3, 1 / 4, etc.), a reciprocal, a ratio, a percentage, a logarithm (in logarithmic operations, the reduction factor can be the base of the logarithm, such as the logarithm with base 10 or the natural logarithm with base e), a square root (such as etc.), functions (such as exponential function e -x or trigonometric function sin(x), etc.).

[0052] It should be understood that the pixel averaging method generates the pixel values ​​of the new image by averaging each pixel block in the original image; the filtering method uses specific filters when reducing the image resolution, such as Gaussian filters, Lanczos filters, or bilinear filters, which take into account the influence of surrounding pixels when calculating new pixel values ​​to smooth transitions and reduce aliasing effects. Regardless of the downsampling method used, the general steps of downsampling include: determining the downsampling ratio, for example, reducing the image size to 1 / 4 of the original, a 4x4 sampling grid may be required. According to the selected method (pixel averaging or filtering), each sampling area is processed. The processed value is assigned to the corresponding position of the new image. Repeat the above process until the entire image is downsampled.

[0053] In this embodiment, if Figure 2BAs shown, in addition to outputting the reference frame, the corresponding thumbnail needs to be output, that is, a 1 / N thumbnail output and input is added, and an additional data stream is required during encoding and decoding. Nevertheless, since it is a fixed 1 / N operation, it can be done with simple addition and subtraction operations. Since the width and height of the thumbnail are both 1 / N of the original size of the reference frame, the storage space required is 1 / N2 of the original space. For example, if the resolution of the reference frame is N times the resolution of the thumbnail, the storage space required for the thumbnail is only 1 / N2 of the reference frame. The generation and storage cost of thumbnails is relatively low, so adding a 1 / N thumbnail in this embodiment is a very cost-effective optimization method.

[0054] Step S22: When inter-frame coding starts, after dividing the frame to be coded into macroblocks, the macroblocks are downsampled into corresponding original macroblocks based on the same thumbnail ratio as the reference frame thumbnail, and the reference frame thumbnail is read in.

[0055] Specifically, in the first step of the encoder's P / B frame encoding, that is, after the original frame is divided into blocks, the original macroblock is also downsampled, and the width and height of the original macroblock are also reduced to 1 / N of the original, where N is an exponent of 2. The same downsampling operation is also performed on the frame to be encoded to obtain the reduced original macroblock. This step is to perform preliminary motion estimation at a lower resolution to reduce the amount of calculation. In addition, P / B frame encoding is two important frame types that play different roles in video compression. P frames (Predicted frames) and B frames (Bidirectional Predicted frames) are both motion-compensated frames, but they differ in terms of prediction and reference frames.

[0056] Step S23: Perform a preliminary search in the reference frame thumbnail to search for a reference block with the highest similarity to the original macroblock in the reference frame thumbnail through similarity calculation, and determine the corresponding motion vector as the initial motion vector.

[0057] In an embodiment of the present application, the search for the reference block with the highest similarity to the original macroblock in the reference frame thumbnail through similarity calculation includes: using the SAD function to calculate the similarity between each reference block in the reference frame thumbnail and the original macroblock; the reference block corresponding to the calculated minimum SAD value has the highest similarity with the original macroblock.

[0058] It should be noted that in order to find the block in the reference frame thumbnail that is most similar to the original macroblock, the SAD function is usually used to measure the similarity between the coding blocks. The smaller the SAD value, the higher the similarity between the blocks, and thus the smaller the coding amount of the difference. Therefore, the SAD values ​​of all reference blocks in the reference frame thumbnail and the reduced original macroblock are calculated, and the minimum SAD value and its corresponding motion vector are recorded. At this time, since the reference frame thumbnail and the original macroblock are at the same thumbnail ratio, the motion vector is equivalent to the original motion vector with 1 / N accuracy on the original frame.

[0059] In the calculation method of the SAD function, for two images A and B of the same size, the SAD function can be defined as:

[0060]

[0061] Where A and B are the two images to be compared; M and N are the number of rows and columns (i.e., height and width) of the image; A(i, j) and B(i, j) are the pixel values ​​of images A and B at position (i, j), respectively; |·| represents the absolute value.

[0062] However, it should be understood that the method used to calculate the similarity between coding blocks in the embodiment of the present application is not limited to the SAD function calculation, and other various methods can also be used to evaluate the similarity between coding blocks, including but not limited to: mean square error (MSE, Mean Squared Error), structural similarity index (SSIM, StructuralSimilarity Index), discrete cosine transform (DCT, Discrete Cosine Transform), etc. Among them, the mean square error (MSE, Mean Squared Error) is the average of the sum of squares of the deviations between the original data and the mean value, reflecting the degree of discreteness of the data. The smaller the mean square error value, the higher the similarity between the coding blocks. The structural similarity index comprehensively considers the brightness, contrast and structural information of the image, and is commonly used in image quality assessment. The larger the structural similarity index, the more similar it is. The discrete cosine transform converts the frame to the frequency domain and then compares the similarity of the coefficients.

[0063] Step S24: Determine whether the similarity value corresponding to the reference block with the highest similarity meets the preset similarity requirement. If it meets the preset similarity requirement, execute step S25; if it does not meet the preset similarity requirement, execute step S26.

[0064] Exemplarily, the calculated SAD value is compared with a preset threshold, where the setting of the preset threshold depends on the trade-off between the desired motion estimation accuracy and coding efficiency. If the SAD value is lower than the preset threshold, it is considered that a sufficiently similar reference block is indeed found in the reference frame thumbnail; if it is higher than the preset threshold, it means that a sufficiently similar reference block to the original macroblock is not found in the reference frame thumbnail, and further fine search is required to find a better match.

[0065] Step S25: input the original macroblock and the initial motion vector into the inter-frame prediction unit for processing, and perform an accurate search in the reference frame starting from the position indicated by the initial motion vector to find the best matching prediction block in the reference frame.

[0066] Exemplarily, if the SAD value is lower than the preset threshold, the corresponding original macroblock and motion vector are input into the Inter prediction unit, and the motion vector is used as the initial motion vector; a precise search is performed within a preset search range around the position indicated by the initial motion vector in the reference frame to find the best matching prediction block.

[0067] Specifically, the initial motion vector is determined based on the SAD value. The initial motion vector is an estimate at the scale of the reference frame thumbnail and can be used as the starting point for subsequent precise search. The determination of the initial motion vector is an important step in the motion estimation process and provides a good starting point for precise search. The initial motion vector is used to locate the data address in the reference frame. Since the accuracy of the initial motion vector is limited by the resolution of the reference frame thumbnail (1 / N), a more refined and precise search is performed in the reference frame, ranging from -N / 2 to N / 2 around the initial motion vector position. A buffer based on the accuracy of the initial motion vector is used to ensure that the best matching reference block can be found.

[0068] Step S26: input the corresponding original macroblock into the intra prediction unit for processing; or, further divide the original macroblock into sub-blocks and continue to perform preliminary search in the reference frame thumbnail until the preset similarity requirement is met.

[0069] Exemplarily, if the SAD value is not lower than the preset threshold, the corresponding original macroblock is input into the Intra prediction unit for processing, or the original macroblock is further divided into sub-blocks and then motion estimation is performed with the reference frame thumbnail.

[0070] The initial motion vector is a motion vector calculated based on the reference frame thumbnail (i.e., a reduced version of the reference frame), which provides a rough estimate indicating the approximate position of the reference block in the reference frame. With the initial motion vector, a good starting point for the search is provided, and the range of the precise search can be greatly reduced, and there is no need to search the entire reference frame in a large range. As the search range is reduced, the amount of data that needs to be read from the reference frame is also reduced, and the reduction in the search space leads to a corresponding reduction in the number of comparisons and calculations required. For example: the existing encoder search range is 256×64, so it needs to be searched more than 16,000 times. If 1 / 4 is used for thumbnail search, the search range becomes 1 / 16 of the original. This is because the 1 / 4 reduction is 4 times smaller in width and height, so only one thousand searches are needed to get the initial motion vector, and then 256 searches are performed on the basis of the initial motion vector to get the accurate motion vector.

[0071] The original macroblock is further divided into sub-blocks and then motion estimated with the reference frame thumbnail, and finally a SAD value lower than a preset threshold is calculated. Then the sub-block and the corresponding initial motion vector at this time can be input into the Inter prediction unit for processing.

[0072] The further division of the original macroblock into sub-blocks and then performing motion estimation with the reference frame thumbnail means that although no good match is found in the original macroblock size, an acceptable motion estimate may exist at the smaller sub-block level. Therefore, the encoder further subdivides the original macroblock into smaller sub-blocks according to the encoding protocol, and repeats the motion estimation process for these sub-blocks in the reference frame thumbnail until a reference block with a sufficiently small SAD value (below a preset threshold) is found for each sub-block. During the precise search process, the encoder no longer further reduces the thumbnail, but instead performs a more detailed search on smaller blocks at the existing thumbnail resolution to find the best motion vector. This can balance the search speed and accuracy while avoiding information loss due to excessive reduction.

[0073] In this embodiment, since it is predicted that the block is suitable for the Inter prediction unit or the Intra prediction unit, each block value only needs to go through one of the processes, and there is no need for each block to go through the Inter prediction unit and the Intra prediction unit at the same time. The existing encoder is a full search and full calculation. Therefore, in this embodiment, the encoder can avoid unnecessary calculations for each block, reduce the overall calculation amount and improve coding efficiency.

[0074] It should be noted that Inter prediction unit and Intra prediction unit are two basic prediction units in video coding. The Inter prediction unit uses previously encoded frames (reference frames) to predict the content of the current frame based on the temporal correlation between video frames. It is applicable to changes in the video caused by object motion or camera movement. The Intra prediction unit does not rely on other frames, but looks for similarities within the current frame. It is usually used to process areas that do not have obvious temporal redundancy in the current frame. Therefore, the Inter prediction unit is usually used for the encoding of P frames or B frames. These frame types rely on at least one reference frame to reduce the amount of data; the Intra prediction unit is used for the encoding of I frames. I frames are self-sufficient frames and are usually used for the encoding of key frames. The Inter prediction unit involves motion estimation, that is, determining the motion vector, which represents the difference between the position of the block in the reference frame and the position in the current frame; the Intra prediction unit does not perform motion estimation because it does not involve inter-frame prediction.

[0075] The advantage of the technical solution in the embodiment of the present application is that it combines coarse and fine search strategies to improve the efficiency and accuracy of motion estimation. By reducing the amount of calculation by fast search on thumbnails, and then ensuring the encoding quality by fine search on the original frame, this method is an effective means to improve compression efficiency and reduce encoding time in video encoding.

[0076] To help those skilled in the art better understand the technical principles and technical effects of the technical solutions in the embodiments of the present application, the standard test stream CrowdRun_1920x1080_60f_50mbps is taken as an example below. The video resolution is 1920×1080, the frame rate is 60 frames per second, and the video bit rate is 50 megabits per second (Mbps).

[0077] (1) The original frame is as follows Figure 3A As shown, the thumbnail frame is partially as follows Figure 3B As shown, it can be seen that the thumbnail frame loses some details, but it does not affect the result when searching for the motion vector (MV), because the purpose of this step is only to know a motion vector within an approximate range.

[0078] (2) Calculate the distribution of the initial SAD of the full thumbnail, as shown in the following figure: Figure 4 shown.

[0079] The left part corresponds to the upper part of the image. It can be seen that for the blue sky part, the SAD and MV are very small, indicating that the static background is very suitable for Inter prediction. The regular protrusion on the left side of the SAD is the calculation result of the tree in the middle of the upper part of the image. It can be seen that the complex texture and irregular leaves affect the calculation of SAD. This part may be more suitable for intra prediction. The MV in the right half begins to increase, corresponding to the densely running crowd. As the video frame is observed from top to bottom, the lower the crowd moves, the faster the MV increases.

[0080] (3) The distribution of motion vectors (MV) in the image is as follows: Figure 5 As shown in FIG. 1 , the motion vector (MV) of the thumbnail is consistent with the real motion vector (MV) obtained by the traditional encoder.

[0081] It should be noted that in the embodiments of the present application, words such as "exemplary" or "for example" represent examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0082] In the embodiments of the present application, "at least one" refers to one or more, and "plurality" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can represent: a, b, c, ab, ac, bc or abc, where a, b, c can be single or multiple.

[0083] Figure 6 is a schematic block diagram of a video coding optimization system based on reference frame thumbnails provided in an embodiment of the present application. Figure 6 As shown, the system includes a down-sampling module 61 , a motion vector module 62 , a determination module 63 and a coding prediction module 64 .

[0084] The downsampling module 61 is used to generate and output a corresponding reference frame thumbnail when outputting a reference frame; after dividing the frame to be encoded into macroblocks, the macroblocks are thumbnailed into corresponding original macroblocks based on the same thumbnail ratio as the reference frame thumbnail, and the reference frame thumbnail is read in.

[0085] The motion vector module 62 is used to perform a preliminary search in the reference frame thumbnail to search for a reference block with the highest similarity to the original macroblock in the reference frame thumbnail through similarity calculation, and determine the corresponding motion vector as the initial motion vector.

[0086] The judgment module 63 is used to judge whether the similarity value corresponding to the reference block with the highest similarity meets the preset similarity requirement.

[0087] The coding prediction module 64 is used to input the original macroblock and the initial motion vector into the inter-frame prediction unit for processing when it is determined that the preset similarity requirement is met, and perform an accurate search in the reference frame starting from the position indicated by the initial motion vector to find the best matching prediction block in the reference frame; when it is determined that the preset similarity requirement is not met, input the corresponding original macroblock into the intra-frame prediction unit for processing; or, further divide the original macroblock into sub-blocks and continue to perform a preliminary search in the reference frame thumbnail until the preset similarity requirement is met.

[0088] It should be understood that the specific process of each module executing the above corresponding steps has been described in detail in the above method embodiment, and for the sake of brevity, it will not be repeated here.

[0089] It should also be understood that the division of modules in the embodiments of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, each functional module in each embodiment of the present application may be integrated into a processor, or may exist physically separately, or two or more modules may be integrated into one module. The above-mentioned integrated modules may be implemented in the form of hardware or in the form of software functional modules.

[0090] Figure 7 is a schematic block diagram of a video encoder provided in an embodiment of the present application. Figure 7 As shown in FIG. 1 , the video encoder includes: at least one processor 701, a memory 702, at least one network interface 703, and a user interface 705. The various components in the device are coupled together via a bus system 704. It is understood that the bus system 704 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 704 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, the bus system 704 is not described in detail. Figure 7 In the specification, various buses are labeled as bus systems.

[0091] The user interface 705 may include a display, a keyboard, a mouse, a trackball, a click gun, keys, buttons, a touch pad or a touch screen.

[0092] It is understood that the memory 702 can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), which is used as an external cache. By way of exemplary but not limiting explanation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM). The memory described in the embodiments of the present invention is intended to include but is not limited to these and any other suitable categories of memory.

[0093] The memory 702 in the embodiment of the present invention is used to store various categories of data to support the operation of the electronic terminal 700. Examples of these data include: any executable program for operating on the electronic terminal 700, such as an operating system 7021 and an application 7022; the operating system 7021 includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks. The application 7022 can include various applications, such as a media player (Media Player), a browser (Browser), etc., for implementing various application services. The XX method provided by the embodiment of the present invention can be included in the application 7022.

[0094] The method disclosed in the above embodiment of the present invention can be applied to the processor 701, or implemented by the processor 701. The processor 701 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit in the processor 701 or the instruction in the form of software. The above processor 701 may be a general processor, a digital signal processor (DSP, Digital Signal Processor), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The processor 701 can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiment of the present invention. The general processor 701 may be a microprocessor or any conventional processor, etc. In combination with the steps of the accessory optimization method provided in the embodiment of the present invention, it can be directly embodied as a hardware decoding processor to execute, or it can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium, which is located in a memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0095] In an exemplary embodiment, the electronic terminal 700 may be implemented by one or more application specific integrated circuits (ASIC), DSP, programmable logic device (PLD), complex programmable logic device (CPLD) to execute the aforementioned method.

[0096] According to the method provided in the embodiment of the present application, the present application also provides a computer program product, which includes: a computer program code, when the computer program code is run on a computer, the computer executes Figures 1 to 3B A method according to any one of the embodiments shown.

[0097] According to the method provided in the embodiment of the present application, the present application also provides a computer-readable storage medium, which stores a program code, and when the program code is run on a computer, the computer executes Figures 1 to 3B A method according to any one of the embodiments shown.

[0098] The terms "component", "module", "system", etc. used in this specification are used to represent computer-related entities, hardware, firmware, a combination of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable file, an execution thread, a program and / or a computer. By way of illustration, both applications running on a computing device and a computing device can be components. One or more components may reside in a process and / or an execution thread, and a component may be located on a computer and / or distributed between two or more computers. In addition, these components may be executed from various computer-readable media having various data structures stored thereon. Components may, for example, communicate through local and / or remote processes according to signals having one or more data packets (e.g., data from two components interacting with another component between a local system, a distributed system and / or a network, such as the Internet interacting with other systems through signals).

[0099] Those skilled in the art will appreciate that the various illustrative logical blocks and steps described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or in a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0100] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0101] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0102] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0103] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0104] In the above embodiments, the functions of each functional unit can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions (programs). When loading and executing computer program instructions (programs) on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. Computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media integrations. Available media may be magnetic media (e.g., floppy disks, hard disks, tapes), optical media (e.g., high-density digital video discs (DVDs), or semiconductor media (e.g., solid state disks (SSDs), etc.).

[0105] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program codes.

[0106] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0107] In summary, the present application provides a video coding optimization method, system, encoder, medium and program product based on reference frame thumbnails, which combines coarse and fine search strategies to improve the efficiency and accuracy of motion estimation. By reducing the amount of calculation through a fast search on the thumbnail, and then ensuring the encoding quality through a fine search on the original frame, this method is an effective means to improve compression efficiency and reduce encoding time in video encoding. Therefore, the present application effectively overcomes the various shortcomings of the prior art and has a high industrial utilization value.

[0108] The above embodiments are merely illustrative of the principles and effects of the present application and are not intended to limit the present application. Anyone familiar with the technology may modify or change the above embodiments without violating the spirit and scope of the present application. Therefore, all equivalent modifications or changes made by a person of ordinary skill in the art without departing from the spirit and technical ideas disclosed in the present application shall still be covered by the claims of the present application.

Claims

1. A video coding optimization method based on reference frame thumbnails, characterized in that: include: When outputting a reference frame, generating and outputting a corresponding reference frame thumbnail after the reference frame is downsampled and reduced; When inter-frame coding starts, after dividing the frame to be coded into macroblocks of the frame, the macroblocks of the frame are downsampled into corresponding original frame reduction blocks based on the same thumbnail ratio as the reference frame thumbnail, and the reference frame thumbnail is read in at the same time; Performing a preliminary search in the reference frame thumbnail to search for a reference block with the highest similarity to the original macroblock in the reference frame thumbnail through similarity calculation, and determining a corresponding motion vector as an initial motion vector; Determine whether the similarity value corresponding to the reference block with the highest similarity meets the preset similarity requirement; If the preset similarity requirement is met, the original macroblock and the initial motion vector are input into the inter-frame prediction unit for processing, and an accurate search is performed in the reference frame starting from the position indicated by the initial motion vector to find the best matching prediction block in the reference frame; If the preset similarity requirement is not met, the corresponding original macroblock is input into the intra-frame prediction unit for processing; or, the original macroblock is further divided into sub-blocks and a preliminary search is continued in the reference frame thumbnail until the preset similarity requirement is met.

2. The video encoding optimization method based on reference frame thumbnails according to claim 1, characterized in that: The method of searching for the reference block with the highest similarity to the original macroblock in the reference frame thumbnail through similarity calculation includes: using the SAD function to calculate the similarity between each reference block in the reference frame thumbnail and the original macroblock; the reference block corresponding to the calculated minimum SAD value has the highest similarity to the original macroblock.

3. The video encoding optimization method based on reference frame thumbnails according to claim 2, characterized in that: After further dividing the original macroblock into sub-blocks, continuing to perform preliminary search in the reference frame thumbnail until the preset similarity requirement is met, including: continuously dividing the original macroblock into sub-blocks of smaller sizes, repeating preliminary search in the reference frame thumbnail and determining the initial motion vector until the SAD value of the sub-block is lower than a preset threshold.

4. The video encoding optimization method based on reference frame thumbnails according to claim 1, characterized in that: The size of the reference frame thumbnail is 1 / N of the original size of the reference frame; the size of the original macroblock after thumbnail is 1 / N of the size of the macroblock before thumbnail; and N is a reduction coefficient suitable for calculation.

5. The video encoding optimization method based on reference frame thumbnails according to claim 4, characterized in that: The range of the precise search includes a region from -N / 2 to N / 2 around the position indicated by the initial motion vector.

6. The video encoding optimization method based on reference frame thumbnails according to claim 1, characterized in that: The method comprises: performing down-sampling processing on a reference frame through a pixel averaging method or a filtering method to generate the reference frame thumbnail, and performing down-sampling on a macroblock obtained by dividing a frame to be encoded to generate the original macroblock.

7. A video coding optimization system based on reference frame thumbnails, characterized in that: include: A downsampling module, used for generating and outputting a corresponding reference frame thumbnail after the reference frame is downsampled when outputting the reference frame; When inter-frame coding starts, after dividing the frame to be coded into macroblocks, the macroblocks are reduced to corresponding original macroblocks based on the same reduction ratio as the thumbnail of the reference frame, and the thumbnail of the reference frame is read in at the same time; A motion vector module, configured to perform a preliminary search in the reference frame thumbnail to search for a reference block having the highest similarity to the original macroblock in the reference frame thumbnail by similarity calculation, and determine a corresponding motion vector as an initial motion vector; A judgment module, used to judge whether the similarity value corresponding to the reference block with the highest similarity meets the preset similarity requirement; The coding prediction module is used to input the original macroblock and the initial motion vector into the inter-frame prediction unit for processing when it is judged that the preset similarity requirement is met, and perform an accurate search in the reference frame starting from the position indicated by the initial motion vector to find the best matching prediction block in the reference frame; when it is judged that the preset similarity requirement is not met, input the corresponding original macroblock into the intra-frame prediction unit for processing; or, after further dividing the original macroblock into sub-blocks, continue to perform preliminary search in the reference frame thumbnail until the preset similarity requirement is met.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the video encoding optimization method based on reference frame thumbnails described in any one of claims 1 to 6 is implemented.

9. A computer program product, characterized in that The computer program product includes computer program code, and when the computer program code is executed on a computer, the computer implements the video encoding optimization method based on reference frame thumbnails as described in any one of claims 1 to 6.

10. A video encoder comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the video encoding optimization method based on reference frame thumbnails as described in any one of claims 1 to 6.

Citation Information

Cited By

  • Predictive coding method and device and storage medium

    CN119788877A

  • Method and electronic circuit for optical motion sensing

    EP4756713A1