Video processing method, device, electronic device and storage medium

Abstract

The present application relates to a video processing method, device, electronic device and storage medium. The method comprises: setting the reference frame according to the largest LCB coding block divided by the encoder as a processing unit for pixel interpolation; establishing multiple pixel groups for the pixel points in the LCB coding block; respectively calculating the pixel interpolation in the LCB coding block according to the dependency between the pixel groups; storing the pixel interpolation in a storage space having a corresponding relationship therewith, so as to use the pixel interpolation to complete video coding. Through the above scheme, when calculating the pixel interpolation in the LCB coding block, the pixel points are grouped, and the pixel interpolation calculation is performed in parallel according to the dependency between the groups, and the pixel interpolation calculation in each group is also performed synchronously, which means that multiple groups can be executed simultaneously, which can effectively improve the calculation efficiency of the pixel interpolation in the LCB coding block.

Description

Technical Field

[0001] The present invention relates to the technical field of video processing, and in particular to a video processing method, device, electronic equipment and storage medium. Background Art

[0002] In video application scenarios, video encoding is an essential operation link, and the operation process is relatively complicated when performing video encoding.

[0003] In the video encoding process, Rate Distortion Optimization (RDO) and Motion Estimation (ME) are two very important parts, which occupy more computing resources and have a great impact on the speed of the encoder.

[0004] The processing object of the video encoder is the sequence of video images (Sequence). The video sequence is a collection of multiple video images (Frame) in time order. The encoder encodes according to the Frame, and each Frame will go through various divisions, decisions, and selections, and will eventually start encoding with each coding unit CU. For the current common video encoders, the basic processing flow is as follows: the first step is to divide each frame into the largest coding block LCB (Largest Coding Block) of a fixed size, and each LCB is a coding entry; the second step is that each LCB will be divided into coding units CU (Coding Unit) of various sizes in a quadtree manner. An LCB will have a combination of CUs of various sizes to form a coding tree unit CTU (Coding Tree Unit); the third step is that each CU may be divided into prediction units PU (Prediction Unit) of different shapes.

[0005] When encoding a CU / PU, the image blocks in the adjacent reference frames are used as references. Based on the similarity, data compression is achieved through inter-frame prediction. There may be multiple reference relationships. For each reference relationship, the RDO decision is usually made by traversing and comparing various possible partitions, and selecting the one with the lowest cost as the final encoding mode. ME and its sub-pixel interpolation are used. For the sub-pixel interpolation in ME, the common practice of encoders is to perform sub-pixel interpolation every time RDO needs to use a sub-pixel as a reference. Although this is more flexible, it brings a lot of redundant calculations, which affects the speed of the encoder. Summary of the invention

[0006] The purpose of the present application is to provide a video processing method, device, electronic device and storage medium, which can realize a solution for improving the video encoding speed.

[0007] According to a first aspect of an embodiment of the present application, a video processing method is provided, including:

[0008] The reference frame is divided according to the largest LCB coding block of the encoder, and is set as a processing unit for pixel interpolation;

[0009] Establishing a plurality of sub-pixel groups for sub-pixel points in the LCB coding block;

[0010] Calculate the sub-pixel interpolation values ​​in the LCB coding block respectively according to the dependency relationship between the sub-pixel groups;

[0011] The pixel-by-pixel interpolation is stored in a storage space corresponding to the pixel-by-pixel interpolation so as to complete video encoding by using the pixel-by-pixel interpolation.

[0012] In one implementation, the establishing a plurality of sub-pixel groups for sub-pixel points in the LCB coding block includes:

[0013] Determine position information of sub-pixel points in the LCB coding block;

[0014] According to the position information, the sub-pixel points are grouped into a basic sub-pixel group and a plurality of parallel sub-pixel groups; wherein the basic sub-pixel group contains half-pixel points whose position information is located between adjacent whole pixels.

[0015] In one implementation, grouping the sub-pixel points into a basic sub-pixel group and a plurality of parallel sub-pixel groups according to the position information includes:

[0016] If the LCB coding block is a pixel matrix including 4 rows and 4 columns, wherein the pixel matrix includes 1 integer pixel and 15 sub-pixel points, the position information is to set corresponding position indexes 0 to 15 for each pixel point in the pixel matrix in order from left to right and from top to bottom;

[0017] A basic pixel group is formed based on the sub-pixel points whose position information is 2, 8 and 10;

[0018] The first parallel sub-pixel group is formed based on the sub-pixel points whose position information is 0, 5, 7, 13 and 15;

[0019] A second parallel sub-pixel group is formed based on the sub-pixel points whose position information is 1, 3, 9 and 11;

[0020] The sub-pixel points whose position information is 4, 6, 12, and 14 form a third parallel sub-pixel group.

[0021] In one implementation, the calculating the sub-pixel interpolation values ​​in the LCB coding block respectively according to the dependency relationship between the sub-pixel groups includes:

[0022] Based on the integer pixels adjacent to the sub-pixel points with position information of 2, 8 and 10, the corresponding basic sub-pixel interpolation values ​​are calculated using the first algorithm.

[0023] In one implementation, the calculating the sub-pixel interpolation values ​​in the LCB coding block respectively according to the dependency relationship between the sub-pixel groups includes:

[0024] Based on the dependency relationship between the basic pixel group and the first parallel sub-pixel group, the second parallel sub-pixel group, and the third parallel sub-pixel group, determine to simultaneously calculate sub-pixel interpolation of the first parallel sub-pixel group, the second parallel sub-pixel group, and the third parallel sub-pixel group;

[0025] Based on the basic sub-pixel interpolation in a diagonally adjacent relationship and the integer pixel value of the integer pixel point, first sub-pixel interpolation of each sub-pixel point in the first parallel sub-pixel group is calculated in parallel;

[0026] Based on the basic sub-pixel interpolation and the integer pixel value of the integer pixel in the same row, parallelly calculate the second sub-pixel interpolation of each sub-pixel point in the second parallel sub-pixel group;

[0027] Based on the basic sub-pixel interpolation and the integer pixel value of the integer pixel in the same column, the third sub-pixel interpolation of each sub-pixel point in the third parallel sub-pixel group is calculated in parallel.

[0028] In one implementation, the mean value of the region where the position information is 0 is calculated based on the integer pixel value corresponding to the 0 position information and the integer pixel values ​​of a plurality of integer pixel points adjacent thereto.

[0029] In one implementation, after storing the pixel interpolation in a storage space corresponding to the pixel interpolation, the method further includes:

[0030] Subsequently, when performing an ME operation with reference to the previously interpolated sub-pixels, that is, after performing a full pixel motion search to obtain the best integer pixel offset point, when performing a sub-pixel search based on the area around the best integer pixel position, a first area mean corresponding to all zero position information starting from the integer pixel position in the current LCB coding block, and a second area mean corresponding to all zero position information starting from multiple other integer pixel positions adjacent to the integer pixel position in the current LCB coding block;

[0031] Comparing pixel differences between the first region mean and a plurality of second region mean values;

[0032] The sub-pixel area at the lower right corner of the integer pixel position corresponding to the second area mean value with the smallest pixel difference is used as the area for preferential sub-pixel motion search. (The block size for motion search is not necessarily LCB, but is based on the actual coding block size, and LCB is the largest coding block unit).

[0033] In one implementation, storing the pixel interpolation values ​​into a storage space corresponding to the pixel interpolation values ​​includes:

[0034] The pixel interpolation corresponding to the LCB coding block in the reference frame is stored in a storage area; each storage area has its corresponding storage area index;

[0035] According to the correspondence between the position index of the sub-pixel point contained in the LCB coding block and the storage space index, each sub-pixel is interpolated and stored in each sub-storage area in the storage area.

[0036] In one implementation, the method further includes: when offset calculation is required, determining pixel interpolation using the offset and the storage space index.

[0037] According to a second aspect of an embodiment of the present application, a video processing device is provided, including:

[0038] A division module, used for setting the reference frame according to the maximum LCB coding block divided by the encoder as a processing unit for pixel interpolation;

[0039] A grouping module, used for establishing a plurality of sub-pixel groups for sub-pixel points in the LCB coding block;

[0040] A calculation module, used for respectively calculating sub-pixel interpolation values ​​in the LCB coding block according to the dependency relationship between the sub-pixel groups;

[0041] The storage module is used to store the pixel-by-pixel interpolation into a storage space corresponding to the pixel-by-pixel interpolation so as to complete video encoding by using the pixel-by-pixel interpolation.

[0042] According to a third aspect of an embodiment of the present application, an electronic device is provided, comprising a memory and a processor, wherein the memory is used to store a computer program executable by the processor; and the processor is used to execute the computer program in the memory to implement the above method.

[0043] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the executable computer program in the storage medium is executed by a processor, the above method can be implemented.

[0044] Compared with the prior art, the beneficial effects of the present application are: using the reference frame according to the largest coding unit block, i.e., the LCB block, into which it is divided when it is encoded, as a processing unit for pixel interpolation; establishing multiple pixel groups for the pixel points in the LCB coding block; respectively calculating the pixel interpolation in the LCB coding block according to the dependency between the pixel groups; storing the pixel interpolation in a storage space having a corresponding relationship therewith, so as to use the pixel interpolation to complete video encoding. Through the above scheme, when calculating the pixel interpolation in the LCB coding block, the pixel points are grouped, and the pixel interpolation calculation is performed in parallel according to the dependency between the groups, and the pixel interpolation calculation in each group is also performed synchronously, which means that multiple groups can be executed simultaneously, which can effectively improve the calculation efficiency of the pixel interpolation in the LCB coding block.

[0045] Each LCB block will be divided into CUs of different sizes by the encoder during encoding. In this patent, the pixel interpolation of the reference frame is processed in units of LCB blocks, and there is no need to consider the division situation within the LCB block. Subsequently, under each possible reference relationship, only the LCB coding block that has previously completed the pixel interpolation needs to be used for ME operation, which can effectively improve the efficiency of pixel interpolation. In addition, the pixel interpolation of each LCB block is only performed once, and repeated interpolation calculations can be avoided when each LCB coding block calculates the pixel interpolation. In addition, the pixel interpolation calculated by the LCB coding block is stored in the storage space. Even if the reference relationship is flexible and changeable, that is, when a reference frame can be referenced by one or more frames, the pixel value corresponding to any offset position of the reference frame can be directly obtained from the storage space without repeated calculations, which can avoid or reduce redundant calculations. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 A schematic diagram of motion estimation provided in an embodiment of the present application.

[0047] Figure 2 A schematic diagram of sub-pixels provided in an embodiment of the present application.

[0048] Figure 3 A schematic diagram of the coding block division provided in an embodiment of the present application.

[0049] Figure 4 A schematic diagram of the reference relationship provided for the embodiments of the present application.

[0050] Figure 5 A flowchart of a video processing method provided in an embodiment of the present application.

[0051] Figure 6 A schematic diagram of pixel point distribution provided in an embodiment of the present application.

[0052] Figure 7 A schematic diagram of the LCB coding block processing sequence illustrated in an embodiment of the present application.

[0053] Figure 8 A schematic diagram of the correspondence between pixel values ​​and storage areas provided in an embodiment of the present application.

[0054] Fig. 9 is a block diagram of a video processing device according to an exemplary embodiment.

[0055] Fig.10 It is a block diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0056] Unless otherwise defined, the technical terms or scientific terms used in this specification and claims shall have the usual meanings understood by persons with ordinary skills in the technical field to which the invention belongs. The specific embodiments of the present invention will be described below in conjunction with the accompanying drawings. It should be noted that in the specific description of these embodiments, in order to provide a concise description, it is impossible for this specification to provide a detailed description of all the features of the actual embodiments. Without departing from the spirit and scope of the present invention, those skilled in the art may modify and replace the embodiments of the present invention, and the resulting embodiments are also within the scope of protection of the present invention.

[0057] In modern video compression standards, inter-frame prediction coding is usually used, that is, the correlation between frames at different times is used to achieve the purpose of video compression through predictive coding. ME is the basis of inter-frame prediction. Inter-frame prediction finds the best match by performing complex ME operations in the reference frame, that is, finding an image block in the reference image that is most similar to the current coding block in the current frame, as shown below Figure 1 A schematic diagram of motion estimation provided in an embodiment of the present application.

[0058] ME performs motion search based on blocks in the reference frame, that is, within a certain search range, it moves to different positions to obtain the corresponding reference blocks, and by comparing the pixel difference between the reference block and the current coding block, finds a position with the smallest difference as the best position. The block corresponding to this position is used as the best reference block (equivalent to the CU or PU mentioned above). The offset corresponding to this position is the motion vector, such as Figure 1The solid arrow in the figure. Usually the best position is not located at the integer pixel position of the original image, but most of the time it is located somewhere between two integer pixels. Therefore, before motion search, it is necessary to divide the blank area between the integer positions into equally spaced fractional pixel positions, and perform interpolation calculations on the fractional pixel positions so that the fractional pixel positions obtain the corresponding pixel values. Then, based on this, motion search can be performed, which can greatly improve the search accuracy. Figure 2 Schematic diagram of pixel division provided in the embodiment of the present application. Figure 2 The "o" shown here represents a pixel point at an integer pixel position, and the "x" represents a pixel at a fractional pixel position.

[0059] The processing object of the video encoder is a sequence of video images (Sequence). A video sequence is a collection of multiple video images (Frame) in time order. The encoder encodes by Frame, and each Frame will go through various divisions, decisions, and selections, and will eventually start encoding with each coding unit CU. Figure 3 Schematic diagram of coding block division provided in the embodiment of the present application. For the currently common video encoder, the basic processing flow is as follows Figure 3 As shown,

[0060] (1) The first step is to divide each frame into LCBs (Largest Coding Blocks) of a fixed size. Each LCB is a coding entry. There are many LCBs in a reference frame. Generally speaking, the number of LCBs in a reference frame is the same as the number of integer pixels, that is, one LCB is divided into one integer pixel.

[0061] (2) In the second step, each LCB is divided into coding units (CUs) of various sizes using a quadtree. An LCB can have a combination of CUs of various sizes, forming a coding tree unit (CTU).

[0062] (3) In the third step, each CU may be divided into prediction units (PUs) of different shapes.

[0063] exist Figure 3In the coding block division shown in the figure, except for step (1), the other two steps require RDO to decide how to divide, the size of the CU, and which PU mode to select for each CU. When encoding a CU / PU, the image blocks in the adjacent reference frame are used as references. According to the similarity, inter-frame prediction is used to achieve data compression. There may be multiple reference relationships here. For each reference relationship, the RDO decision is usually made by traversing and comparing various possible divisions, and selecting the one with the lowest cost as the final coding mode. ME and its pixel-by-pixel interpolation are used. Moreover, different reference frames correspond to different reference relationships and different pixel-by-pixel interpolations. The selection of reference frames is also decided by RDO. Figure 4 Schematic diagram of the reference relationship provided in the embodiment of the present application. Figure 4 As shown in the figure (the arrow in the figure indicates a possible reference relationship when CU / PU is encoded). For sub-pixel interpolation in ME, the common practice of encoders is to perform sub-pixel interpolation every time RDO needs to use sub-pixels as references. Although this is more flexible, it brings a lot of redundant calculations, which affects the speed of the encoder.

[0064] like Figure 5 A schematic flow chart of a video processing method provided in an embodiment of the present application. The execution subject of the method may be a local computer, a cloud computing server, etc. The method specifically comprises the following steps:

[0065] Step 501: Set the reference frame as a processing unit for pixel-by-pixel interpolation according to the largest LCB coding block divided by the encoder.

[0066] Step 502: Create multiple sub-pixel groups for sub-pixel points in the LCB coding block.

[0067] Step 503: Calculate the sub-pixel interpolation values ​​in the LCB coding block respectively according to the dependency relationship between the sub-pixel groups.

[0068] Step 504: Store the sub-pixel interpolation into a storage space corresponding thereto, so as to complete video encoding by using the sub-pixel interpolation.

[0069] The LCB coding block division mentioned here is determined by the encoder, and the pixel interpolation is only based on the largest coding block divided by the encoder in practice. It is the largest coding block obtained by dividing the reference frame, and is also the coding operation unit of this application, that is, pixel interpolation is performed according to the LCB coding block as the processing unit. There are multiple CU basic units in an LCB coding block. Generally speaking, an LCB coding block contains 128x128 integer pixels (the maximum coding block size used in the AV1 video compression standard) or 64×64 integer pixels (the maximum block size used in the HEVC video compression standard). Of course, it can also be divided into 32×32 integer pixel LCBs and 16x16 integer pixel LCBs (the maximum block size used in the AVC video compression standard) as needed.

[0070] In this solution, pixel-by-pixel interpolation calculations can be performed while encoding each LCB in the reference frame. Specifically, when encoding, each LCB coding block is encoded in order from left to right and from top to bottom. After encoding the LCB coding module in the second row and second column, pixel-by-pixel interpolation calculations can be performed on the previously completed LCB coding block in its upper left corner, that is, the pixel-by-pixel calculations of other LCB coding blocks in the reference frame are calculated according to the coding process within the frame. The pixel-by-pixel calculations are all based on the reconstructed image pixels of each LCB coding block. Finally, the pixel-by-pixel interpolation calculations of the entire frame are completed while the entire reference frame coding is completed, thereby effectively improving the coding efficiency.

[0071] Figure 6 This is a schematic diagram of pixel point distribution provided in the embodiment of the present application. Figure 6 As shown, the establishment of multiple sub-pixel groups for the sub-pixel points in the LCB coding block includes: determining the position information of the sub-pixel points in the LCB coding block; grouping the sub-pixel points into a basic sub-pixel group and multiple parallel sub-pixel groups according to the position information; wherein the basic sub-pixel group contains half-pixel points whose position information is located between adjacent whole pixels.

[0072] Figure 6 There is one integer pixel E in the image, and there are 15 sub-pixels to the lower right of the integer pixel (of course, in actual applications, considering the different sub-pixel precisions, there can also be 35 (corresponding to a sub-pixel precision of 1 / 6 pixel and a sub-pixel matrix of 6×6) or 63 (corresponding to a sub-pixel precision of 1 / 8 and a sub-pixel matrix of 8×8)).

[0073] The parallel pixel grouping mentioned here means that when each pixel grouping is executed, multiple groups can be executed in parallel, which can effectively improve the calculation efficiency of pixel interpolation.

[0074] When creating groups, they are divided into basic pixel groups and parallel pixel groups. In an LCB coding block, there is one basic pixel group and multiple parallel pixel groups. The pixels in the basic pixel group are half-pixels, that is, the pixel positions located between two whole pixels. This half-pixel does not really exist.

[0075] In one or more embodiments of the present application, grouping the sub-pixel points into a basic sub-pixel group and a plurality of parallel sub-pixel groups according to the position information includes:

[0076] If the pixel precision of the LCB coding block is 1 / 4 pixel, then the lower right corner of each integer pixel is a pixel matrix containing 4 rows and 4 columns (the pixel value of the pixel position needs to be obtained by interpolation). An LCB coding block contains N (LCB width x LCB height) integer pixels. The value of N depends on the width and height of the largest coding block LCB divided by the encoder. The same interpolation process is used for each integer pixel in the LCB coding block. It contains 1 integer pixel (that is, Figure 6 Point E in (that is, position index 0)) and 15 sub-pixel points ( Figure 6 The position information is position indexes 1-15 in the LCB coding block, and the position information is position indexes (0 to 15) set in sequence from left to right and from top to bottom for each sub-pixel point corresponding to each integer pixel point of the LCB coding block.

[0077] A basic pixel group is formed based on the sub-pixel points whose position information is 2, 8 and 10;

[0078] The first parallel sub-pixel group is formed based on the sub-pixel points whose position information is 0, 5, 7, 13 and 15;

[0079] A second parallel sub-pixel group is formed based on the sub-pixel points whose position information is 1, 3, 9 and 11;

[0080] The sub-pixel points whose position information is 4, 6, 12, and 14 form a third parallel sub-pixel group.

[0081] In one or more embodiments of the present application, respectively calculating the sub-pixel interpolation in the LCB coding block according to the dependency relationship between the sub-pixel groups includes:

[0082] Based on the integer pixels adjacent to the sub-pixel points with position information of 2, 8 and 10, the corresponding basic sub-pixel interpolation values ​​are calculated using the first algorithm.

[0083] In one or more embodiments of the present application, respectively calculating the sub-pixel interpolation in the LCB coding block according to the dependency relationship between the sub-pixel groups includes:

[0084] Based on the dependency relationship between the basic pixel group and the first parallel sub-pixel group, the second parallel sub-pixel group, and the third parallel sub-pixel group, determine to simultaneously calculate sub-pixel interpolation of the first parallel sub-pixel group, the second parallel sub-pixel group, and the third parallel sub-pixel group;

[0085] Based on the basic sub-pixel interpolation in a diagonally adjacent relationship and the integer pixel value of the integer pixel point, first sub-pixel interpolation of each sub-pixel point in the first parallel sub-pixel group is calculated in parallel;

[0086] Based on the basic sub-pixel interpolation and the integer pixel value of the integer pixel in the same row, parallelly calculate the second sub-pixel interpolation of each sub-pixel point in the second parallel sub-pixel group;

[0087] Based on the basic sub-pixel interpolation and the integer pixel value of the integer pixel in the same column, the third sub-pixel interpolation of each sub-pixel point in the third parallel sub-pixel group is calculated in parallel.

[0088] In one or more embodiments of the present application, the mean value of the region where the position information is 0 is calculated based on the integer pixel value corresponding to the 0 position information and the integer pixel values ​​of multiple integer pixel points adjacent to the 0 position information.

[0089] The pixel grouping implementation scheme will be described below in conjunction with specific embodiments.

[0090] Combine the following Figure 7 Expand the description. Figure 7 A schematic diagram of the LCB coding block processing sequence provided in an embodiment of the present application.

[0091] Each integer pixel position in an LCB coding block, such as Figure 6 and Figure 7 The position represented by the capital letter in has 15 sub-pixel positions (including its own position, a total of 16 positions) in its lower right corner, distributed in the area between other adjacent whole pixel positions, such as Figure 7 As shown in the dotted circle, these 16 sub-pixel positions are a group and correspond to one integer pixel. In an LCB (64 pixels x 64 pixels) coding block, there are 64x64 integer pixel positions, and the same set of operations is performed on each integer pixel position. That is, 15 sub-pixel interpolations are performed on 15 sub-pixel positions at each integer position, and 1 interpolation is performed at the integer position (position 0 in the figure) for the subsequent encoder's fast algorithm. And these sub-pixel interpolations are stored in the specified storage space. In the subsequent use process, there is no need to repeat the calculation, which can improve coding efficiency and reduce the burden of redundant calculations.

[0092] Design 16 sub-pixel storage areas corresponding to all sub-pixel positions in the reference frame, and map the sub-pixel position index (that is, the sub-pixel position identifier) ​​with the storage area index, such as Figure 6 and Figure 8 As shown. The sub-pixel storage area is obtained according to the sub-pixel position index, and the specific storage position in the sub-pixel storage area is pointed to according to the integer pixel position index corresponding to the sub-pixel. And these 16 sub-pixel storage areas are used as a large storage area set for storing all sub-pixels of each reference frame. This is very friendly and efficient for subsequent ME, because for each sub-pixel offset, the interpolated pixel value will be continuously stored in the memory.

[0093] For each integer pixel position in each LCB, there are 16 sub-pixel interpolations. Each sub-pixel interpolation can independently use a 4-tap or 6-tap interpolation filter to interpolate and filter the integer pixels in its vertical and horizontal directions. However, due to the large amount of calculation, the encoder speed will be slowed down. An optimized interpolation filter kernel is designed in the present invention to reduce the amount of calculation and the dependency between the 16 interpolations, which is very friendly to the assembly acceleration optimization of the encoder and is convenient for software implementation.

[0094] like Figure 7 As shown, positions (2, 8, 10) are the most important half-pixel positions, and a 6-tap interpolation filter is used. Positions (2) and (8) need to be interpolated with 6 taps before position (10). After position (2) is completed, all interpolated pixels at the horizontal half-pixel position of the LCB can be obtained. After position (8) is completed, all interpolated pixels at the vertical half-pixel position of the LCB can be obtained. Then, the pixel values ​​of the interpolated half-pixel positions are filtered using a 6-tap interpolation filter to obtain the interpolated pixels of all the middle half-pixel positions, position (10). Positions (0, 5, 7, 13, 15) all use a cross 4-value mean filter, positions (1, 3, 9, 11) use a horizontal 4-tap interpolation filter, and positions (4, 6, 12, 14) use a vertical 4-tap interpolation filter. The numbers here represent the serial numbers of all sub-pixel positions at the lower right corner of each integer pixel, and also represent the serial numbers of the corresponding storage areas. The filtering specifications for each position are described in the specific interpolation steps that follow.

[0095] 16 sub-pixel position interpolation steps:

[0096] In the first step, position (0) = (position E + position F + position H + position I) / 4. The sub-pixel value at this position (that is, the regional mean) is used for the subsequent encoder fast algorithm (which can be used as the subsequent selection priority motion search). Position (2) uses a 6-tap filter to perform interpolation filtering on the integer pixels in the horizontal direction (that is, the first algorithm mentioned above). The specific coefficients of the 6-tap filter are not limited here.

[0097] In the second step, position (8) uses a 6-tap filter to perform interpolation filtering on the whole pixels in the vertical direction. The specific coefficients of the 6-tap filter are not limited here; position (10) uses a 6-tap filter to perform interpolation filtering on the pixel values ​​of the half-pixel positions that have been generated. The specific coefficients of the 6-tap filter are not limited here, and there are three direction options, using only horizontal half pixels, or only vertical half pixels, or weighted average after vertical and horizontal. Using a 6-tap filter is a method with high calculation accuracy and accurate pixel interpolation, because the basic pixel interpolation obtained by the algorithm is used for the calculation of pixel interpolation in other subsequent parallel pixel groups. However, the calculation is complex, the computing power consumption is high, and the calculation time is long. When performing pixel interpolation calculation on the parallel pixel groups in the subsequent calculation, the second algorithm is used. The calculation accuracy of the second algorithm is lower than that of the first algorithm, but the algorithm is simple and the calculation efficiency is high.

[0098] The third step is to perform pixel interpolation on the position (5, 7, 13, 15) using a cross 4-value mean filter (that is, the second algorithm mentioned above).

[0099] Position(5) =<Position(E) + Position(2) + Position(8) + Position(10)> / 4;

[0100] Position(7) =<position(2) + position(F) + position(10) + position(F8)> / 4;

[0101] Position(13) =<position(8) + position(10) + position(H) + position(H2)> / 4;

[0102] Position(15) =<position(10) + position(F8) + position(H2) + position(I)> / 4;

[0103] Perform pixel interpolation on the position (1, 3, 9, 11) and use a 4-tap filter (another second algorithm) to filter the half-pixel points in the horizontal direction as follows:

[0104] Position(1) = 4-tap-filter<position(D2), position(E), position(2), position(F)>,

[0105] Position(3) = 4-tap-filter<position(E), position(2), position(F), position(F2)>,

[0106] Position (9) = 4-tap-filter <position (D10), position (8), position (10), position (F8)>,

[0107] Position (11) = 4-tap-filter <position (8), position (10), position (F8), position (F10)>

[0108] Here, 4-tap-filter represents a 4-tap filter, and the specific coefficients of the 4-tap filter are not specified.

[0109] The fourth step is to perform pixel interpolation on the positions (4, 6, 12, 14) and use a 4-tap filter to filter the half-pixel points in the vertical direction. Position (4) = 4-tap-filter<position (B8), position (E), position (8), position (H)>,

[0110] Position(12) = 4-tap-filter<position(E), position(8), position(H), position(H8)>,

[0111] Position (6) = 4-tap-filter <position (B10), position (2), position (10), position (H2)>,

[0112] Position (14) = 4-tap-filter <position (2), position (10), position (H2), position (H10)>,

[0113] Here, 4-tap-filter represents a 4-tap filter, and does not limit the specific coefficients of the 4-tap filter.

[0114] By the above method, the sub-pixels are grouped, and when calculating the sub-pixel interpolation, the basic pixel group is calculated first, and the first algorithm is used. The calculation accuracy is high, and the obtained pixel value is more accurate, which can meet the calculation requirements of subsequent calculations of other sub-pixel interpolation. When calculating multiple groups of parallel sub-pixel groups, the second algorithm is used for calculation. The algorithm is simple and the calculation efficiency is high. In addition, multiple parallel sub-pixel groups are calculated in parallel at the same time, and multiple sub-pixel points in the group are also calculated in parallel. While meeting the accuracy requirements of calculating the sub-pixel interpolation results, it can also effectively improve the processing speed and processing capacity within the LCB coding block.

[0115] In one or more embodiments of the present application, after storing the pixel interpolation in a storage space having a corresponding relationship therewith, the method further includes:

[0116] When the subsequent encoder performs the ME operation, it needs to use the sub-pixel value of the reference frame, and can directly use the sub-pixel value of the reference frame that has been interpolated previously, that is, after performing the full pixel motion search to obtain the best integer pixel offset point, when performing the sub-pixel search based on the surrounding of the best integer pixel position, determine the first area mean corresponding to all 0 position information starting from the integer pixel position in the current LCB coding block, and the second area mean corresponding to all 0 position information starting from multiple other integer pixel positions adjacent to the integer pixel position in the current LCB coding block;

[0117] Comparing pixel differences between the first region mean and a plurality of second region mean values;

[0118] The sub-pixel region at the lower right corner of the integer pixel position corresponding to the second region mean value with the smallest pixel difference is used as the region for preferential sub-pixel motion search.

[0119] like Figure 6 The LCB coding block includes multiple integer pixels and sub-pixel positions in the area corresponding to the integer pixels. For example, Figure 7 At the integer pixel position E, the sub-pixel area in the lower right corner has 15 sub-pixel positions (intervals are 1 / 4 integer pixels), and the corresponding sub-pixel interpolation calculations are performed as described above. For the position where the integer pixel is located (itself does not need sub-pixel interpolation), the integer pixel position is defined as position 0, plus the sub-pixel position, a total of 16 interpolation positions, and the interpolation with position index as position 0 (that is, the first area mean) can be expressed as:

[0120] Point E position (0) = <position (E) + position (F) + position (H) + position (I)> / 4.

[0121] Similarly, we can get the mean values ​​of other second regions:

[0122] Point A position (0) = <position (A) + position (B) + position (D) + position (E)> / 4;

[0123] Point B position (0) = <position (B) + position (C) + position (E) + position (F)> / 4;

[0124] Point D position (0) = <position (D) + position (E) + position (G) + position (H)> / 4.

[0125] The pixel values ​​obtained by setting all positions to 0 are used in the subsequent ME to speed up the sub-pixel motion search using a fast algorithm. Figure 6For example, assuming that point E is the best position after the integer pixel motion search in ME, the next step is the sub-pixel motion search. The conventional practice is to search and check all sub-pixel positions around the best integer pixel point E. In order to find the best sub-pixel position, many RD comparisons are required. If all sub-pixel positions around the integer pixel are traversed, some redundant calculations and comparisons will be introduced. The specific strategy in this solution is as follows: the special interpolation pixels of the previously obtained integer pixel positions are used to make predictions and accelerate the sub-pixel motion search process. The specific steps are as follows:

[0126] Assuming that point E is the best integer pixel position after the integer pixel motion search in ME, combined with the 8 candidate integer pixel positions ABCDFGHI around E, the sub-pixel motion search area around point E is formed. This search area is divided into 4 offset areas with point E as the center, namely the area surrounded by ABED in the upper left corner, the area surrounded by BCFE in the upper right corner, the area surrounded by DEHG in the lower left corner, and the area surrounded by EFIH in the lower right corner. First, through simple comparison, a sub-region with the maximum probability of sub-pixel search is selected from the four regions around the best integer pixel point (as mentioned above, upper left, upper right, lower left, and lower right), and then a sub-pixel search is performed in the sub-region, thereby avoiding sub-pixel search in all the above four regions, thereby accelerating the processing speed of the encoder. This is one of the key points of this patent, which is not limited to using only the special interpolated pixel value corresponding to the 0 point position.

[0127] The first step is to shift to the four integer pixel positions of ABDE respectively, and take out the first area mean (interpolated pixel value) of each corresponding position 0. For example, shift to point A, take out the pixel value of position 0 corresponding to all integer pixel positions of each LCB coding block in the reference frame, record it as pixel block Pixels(A0), and similarly obtain Pixels(B0) corresponding to point B, Pixels(D0) corresponding to point D, and Pixels(E0) corresponding to point E. When offset calculation is required, use the offset and the storage space index to determine the pixel interpolation, that is, assuming that the offset is X, the original storage space indexes are ABDE respectively, and calculate to obtain Pixels(A0)=A+X, Pixels(B0)=B+X, Pixels(D0)=D+X, Pixels(E0)=E+X. Since the storage order in the storage space is continuous and the offset is exactly the same, it is convenient to quickly read the corresponding pixel values.

[0128] In the second step, the pixel blocks in the above four cases are compared with the coding block of the current ME respectively, and the integer pixel points corresponding to the pixel block with the smallest difference are obtained. The area for the next sub-pixel search is determined according to the position of point E, that is, an optimal area is selected from the four areas, and a smaller area is used to continue the next search. For example, when the pixel value difference between the pixel block Pixels (A0) and the LCB coding block of the current ME is the smallest, the area surrounded by ABED in the upper left corner is the best area for sub-pixel search around point E. The sub-pixel search is only performed in this area, avoiding sub-pixel search in 4 areas, which greatly saves resources and improves the speed of motion search.

[0129] In the third step, in the best area determined in the second step, the differences between the interpolated pixel blocks under all pixel position offsets and the coding block of the current ME are compared, and the one with the smallest difference is selected as the best motion search match. For example, when offsetting to position 1, the pixel values ​​of position 1 corresponding to all integer pixels in the search block are taken out, that is, the values ​​corresponding to storage space 1 are directly taken out, buffers(1), and so on. When offsetting to position n, the corresponding storage space is found, buffers(n).

[0130] In one or more embodiments of the present application, storing the pixel interpolation in a storage space having a corresponding relationship therewith includes:

[0131] The pixel interpolation corresponding to the LCB coding block in the reference frame is stored in a storage area; each storage area has its corresponding storage area index;

[0132] According to the correspondence between the position index of the sub-pixel point contained in the LCB coding block and the storage space index, each sub-pixel is interpolated and stored in each sub-storage area in the storage area.

[0133] Among them, all LCB coding blocks in the reference frame share the 16 sub-storage areas of the reference frame, and the sub-pixel values ​​of each LCB coding block have different offsets in each sub-storage area. The correspondence between the position index and the storage space index is determined by the integer pixel position offset in the LCB coding block.

[0134] like Figure 8 A schematic diagram of the corresponding relationship between pixel values ​​and storage areas provided in an embodiment of the present application. Figure 8 As can be seen in

[0135] In the frame-level pixel-by-pixel interpolation architecture, each reference frame (I frame or P frame) corresponds to a large storage area, and the large storage area is divided into 16 small sub-storage areas, which are used to store the interpolated pixel values ​​of all 16 fractional pixel positions in a frame, as shown in 8.

[0136] Figure 8 Here, padded-width indicates the width of the original width of the frame image after it is expanded outward according to the motion search needs. Similarly, padded-height indicates the height of the original height of the frame image after it is expanded outward according to the motion search needs. The width and height after expansion here are related to the design of the actual encoder and are determined by the encoder. The specific size is not specified in the present application scheme. The present application scheme discloses allocating and managing the storage space of the sub-pixel frame corresponding to each reference frame based on the width and height after expansion, that is, allocating the size according to 4 times the expansion width and 4 times the expansion height (16 times the original frame image), and then dividing this large storage area into 16 small sub-storage areas, corresponding to the 16 sub-pixel interpolation positions of each integer pixel, as described in the sub-pixel interpolation kernel of the LCB coding block.

[0137] Inside the video encoder, the I frames and P frames used as reference frames are managed through the reference frame list. According to the solution of the present application, each reference frame corresponds to a sub-pixel interpolated frame, and each sub-pixel interpolated frame is also managed by a list. For example, two tags ref_index and buf_index are designed to manage the storage space of the sub-pixel frame, ref_index represents the index of the current reference frame, and buf_index represents the index of the current storage space.

[0138] For example, in the reference queue {I0, P1, P2, P3, P4, P5, P6, P7, ..., Pn, ...}, the subscript number in the queue represents the position index ref_index. The index of the corresponding storage space can be obtained using the formula buf_index = ref_index % total_buf_num. Here, total_buf_num represents the total amount of storage space, which depends on the reference structure and multi-threaded structure used by the software encoder. buf_index is used to obtain the interpolation buffer of each reference frame.

[0139] For example, when a P1 frame is encoded, a large storage space is obtained through buf_index to store all pixel values ​​of the frame for interpolation; when a P1 frame is referenced, the storage location of the pixel value of the frame is obtained through buf_index, and the ME directly takes the pixel value from the corresponding storage space for use. This avoids the redundant work of repeatedly calculating the pixel interpolation and improves the encoding efficiency.

[0140] Fig. 9 FIG. 1 is a block diagram of a video processing device according to an exemplary embodiment. Fig. 9 As shown, in this embodiment, the video processing device includes:

[0141] The division module 901 is used to set the reference frame according to the largest LCB coding block divided by the encoder as a processing unit for pixel-by-pixel interpolation.

[0142] The grouping module 902 is used to establish multiple sub-pixel groups for the sub-pixel points in the LCB coding block.

[0143] The calculation module 903 is used to calculate the sub-pixel interpolation values ​​in the LCB coding block respectively according to the dependency relationship between the sub-pixel groups.

[0144] The storage module 904 is used to store the sub-pixel interpolation into a storage space corresponding to it, so as to complete video encoding by using the sub-pixel interpolation.

[0145] A grouping module 902 is used to determine the position information of the sub-pixel points in the LCB coding block;

[0146] According to the position information, the sub-pixel points are grouped into a basic sub-pixel group and a plurality of parallel sub-pixel groups; wherein the basic sub-pixel group contains half-pixel points whose position information is located between adjacent whole pixels.

[0147] A grouping module 902 is configured to, if the LCB coding block is a pixel matrix including 4 rows and 4 columns, wherein the pixel matrix includes 1 integer pixel and 15 sub-pixel points, the position information is to set corresponding position indexes 0 to 15 for each pixel point in the pixel matrix in order from left to right and from top to bottom;

[0148] A basic pixel group is formed based on the sub-pixel points whose position information is 2, 8 and 10;

[0149] The first parallel sub-pixel group is formed based on the sub-pixel points whose position information is 0, 5, 7, 13 and 15;

[0150] A second parallel sub-pixel group is formed based on the sub-pixel points whose position information is 1, 3, 9 and 11;

[0151] The sub-pixel points whose position information is 4, 6, 12, and 14 form a third parallel sub-pixel group.

[0152] The grouping module 902 is configured to calculate corresponding basic sub-pixel interpolation values ​​based on integer pixels adjacent to the sub-pixel points whose position information is 2, 8 and 10 by using a first algorithm.

[0153] A grouping module 902 is used to determine to simultaneously calculate sub-pixel interpolation of the first parallel sub-pixel group, the second parallel sub-pixel group, and the third parallel sub-pixel group based on a dependency relationship between the basic pixel group and the first parallel sub-pixel group, the second parallel sub-pixel group, and the third parallel sub-pixel group;

[0154] Based on the basic sub-pixel interpolation in a diagonally adjacent relationship and the integer pixel value of the integer pixel point, first sub-pixel interpolation of each sub-pixel point in the first parallel sub-pixel group is calculated in parallel;

[0155] Based on the basic sub-pixel interpolation and the integer pixel value of the integer pixel in the same row, parallelly calculate the second sub-pixel interpolation of each sub-pixel point in the second parallel sub-pixel group;

[0156] Based on the basic sub-pixel interpolation and the integer pixel value of the integer pixel in the same column, the third sub-pixel interpolation of each sub-pixel point in the third parallel sub-pixel group is calculated in parallel.

[0157] The grouping module 902 is used to calculate the mean value of the area where the position information is 0 based on the integer pixel value corresponding to the 0 position information and the integer pixel values ​​of multiple integer pixel points adjacent to it.

[0158] A calculation module 903 is used to determine a first area mean corresponding to all zero position information starting from an integer pixel position in a current LCB coding block, and a second area mean corresponding to all zero position information starting from a plurality of other integer pixel positions adjacent to the integer pixel position in the current LCB coding block, when performing an ME operation with reference to the previously interpolated sub-pixels, that is, after performing a full pixel motion search to obtain an optimal integer pixel offset point, and performing a sub-pixel search based on the surrounding of the optimal integer pixel position;

[0159] Comparing pixel differences between the first region mean and a plurality of second region mean values;

[0160] The sub-pixel region at the lower right corner of the integer pixel position corresponding to the second region mean value with the smallest pixel difference is used as the region for preferential sub-pixel motion search.

[0161] Storage space 904, used to store the pixel interpolation corresponding to the LCB coding block in the reference frame into a storage area; each storage area has its corresponding storage area index;

[0162] According to the correspondence between the position index of the sub-pixel point contained in the LCB coding block and the storage space index, each sub-pixel is interpolated and stored in each sub-storage area in the storage area.

[0163] The calculation module 903 is used to determine the pixel interpolation using the offset and the storage space index when offset calculation is required.

[0164] An embodiment of the present application also proposes an electronic device, comprising a processor and a memory; the memory is used to store a computer program executable by the processor; the processor is used to execute the computer program in the memory to implement the video processing method described in any of the above embodiments.

[0165] The embodiments of the present application further provide a computer-readable storage medium. When the executable computer program in the storage medium is executed by a processor, the video processing method described in any of the above embodiments can be implemented.

[0166] Regarding the device in the above embodiment, the specific manner in which the processor performs the operation has been described in detail in the embodiment of the method, and will not be elaborated here.

[0167] Fig.10 900 is a block diagram of an electronic device according to an exemplary embodiment. For example, the electronic device 900 may be provided as a server. Fig.10 , the device 900 includes a processing component 922, which further includes one or more processors, and a memory resource represented by a memory 932 for storing instructions, such as applications, that can be executed by the processing component 922. The application stored in the memory 932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 922 is configured to execute instructions to perform the above-mentioned video processing method.

[0168] The device 900 may also include a power component 926 configured to perform power management of the device 900, a wired or wireless network interface 950 configured to connect the device 900 to a network, and an input / output (I / O) interface 958. The device 900 may operate based on an operating system stored in the memory 932, such as Windows Server™, MacOS X™, Unix™, Linux™, FreeBSD™, or the like.

[0169] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 932 including instructions, which can be executed by the processing component 922 of the device 900 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0170] In the present invention, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance. The term "plurality" refers to two or more than two, unless otherwise clearly defined.

[0171] The above description of the embodiments is to facilitate those of ordinary skill in the art to understand and apply the present application. It is obvious that those familiar with the art can easily make various modifications to these embodiments and apply the general principles described herein to other embodiments without having to pay creative labor. Therefore, the present application is not limited to the embodiments herein, and improvements and modifications made by those skilled in the art based on the contents disclosed in the present application without departing from the scope and spirit of the present application are within the scope of the present application.

Claims

1. A video processing method, characterized in that: The method comprises: The reference frame is divided according to the largest LCB coding block of the encoder, and is set as a processing unit for pixel interpolation; Establishing a plurality of sub-pixel groups for sub-pixel points in the LCB coding block; Calculate the sub-pixel interpolation values ​​in the LCB coding block respectively according to the dependency relationship between the sub-pixel groups; The sub-pixel interpolation is stored in a storage space having a corresponding relationship therewith, so as to use the sub-pixel interpolation to complete video encoding; the best integer pixel position after the integer pixel motion search in the motion estimation ME is divided into multiple sub-pixel search areas at integer pixel points around the best integer pixel position, a sub-pixel search sub-area with a maximum probability is selected from the multiple sub-pixel search areas, and then the sub-pixel search is performed in the sub-area, thereby avoiding performing sub-pixel searches in the above multiple areas.

2. The method according to claim 1, characterized in that The step of establishing a plurality of sub-pixel groups for the sub-pixel points in the LCB coding block comprises: Determine position information of sub-pixel points in the LCB coding block; According to the position information, the sub-pixel points are grouped into a basic sub-pixel group and a plurality of parallel sub-pixel groups; wherein the basic sub-pixel group contains half-pixel points whose position information is located between adjacent whole pixels.

3. The method according to claim 2, characterized in that The step of grouping the sub-pixel points into a basic sub-pixel group and a plurality of parallel sub-pixel groups according to the position information includes: If the LCB coding block is a pixel matrix including 4 rows and 4 columns, wherein the pixel matrix includes 1 integer pixel and 15 sub-pixel points, the position information is to set corresponding position indexes 0 to 15 for each pixel point in the pixel matrix in order from left to right and from top to bottom; A basic pixel group is formed based on the sub-pixel points whose position information is 2, 8 and 10; The first parallel sub-pixel group is formed based on the sub-pixel points whose position information is 0, 5, 7, 13 and 15; A second parallel sub-pixel group is formed based on the sub-pixel points whose position information is 1, 3, 9 and 11; The sub-pixel points whose position information is 4, 6, 12, and 14 form a third parallel sub-pixel group.

4. The method according to claim 3, characterized in that The calculating the sub-pixel interpolation values ​​in the LCB coding block respectively according to the dependency relationship between the sub-pixel groups includes: Based on the integer pixels adjacent to the sub-pixel points with position information of 2, 8 and 10, the corresponding basic sub-pixel interpolation values ​​are calculated using the first algorithm.

5. The method according to claim 4, characterized in that The calculating the sub-pixel interpolation values ​​in the LCB coding block respectively according to the dependency relationship between the sub-pixel groups includes: Based on the dependency relationship between the basic pixel group and the first parallel sub-pixel group, the second parallel sub-pixel group, and the third parallel sub-pixel group, determine to simultaneously calculate sub-pixel interpolation of the first parallel sub-pixel group, the second parallel sub-pixel group, and the third parallel sub-pixel group; Based on the basic sub-pixel interpolation in a diagonally adjacent relationship and the integer pixel value of the integer pixel point, first sub-pixel interpolation of each sub-pixel point in the first parallel sub-pixel group is calculated in parallel; Based on the basic sub-pixel interpolation and the integer pixel value of the integer pixel in the same row, parallelly calculate the second sub-pixel interpolation of each sub-pixel point in the second parallel sub-pixel group; Based on the basic sub-pixel interpolation and the integer pixel value of the integer pixel in the same column, the third sub-pixel interpolation of each sub-pixel point in the third parallel sub-pixel group is calculated in parallel.

6. The method according to claim 3, characterized in that: The mean value of the region where the position information is 0 is calculated based on the integer pixel value corresponding to the 0 position information and the integer pixel values ​​of multiple integer pixel points adjacent to the 0 position information.

7. The method according to claim 3, characterized in that The step of storing the pixel interpolation values ​​into a storage space having a corresponding relationship therewith comprises: The pixel interpolation corresponding to the LCB coding block in the reference frame is stored in a storage area; each storage area has its corresponding storage area index; According to the correspondence between the position index of the sub-pixel point contained in the LCB coding block and the storage space index, each sub-pixel is interpolated and stored in each sub-storage area in the storage area.

8. The method according to claim 7, characterized in that Also includes: When offset calculation is required, the sub-pixel interpolation is determined using the offset and the storage space index.

9. A video processing device, characterized in that: The device comprises: A division module, used for setting the reference frame according to the maximum LCB coding block divided by the encoder as a processing unit for pixel interpolation; A grouping module, used for establishing a plurality of sub-pixel groups for sub-pixel points in the LCB coding block; A calculation module, used for respectively calculating sub-pixel interpolation values ​​in the LCB coding block according to the dependency relationship between the sub-pixel groups; A storage module stores the sub-pixel interpolation in a storage space having a corresponding relationship therewith, so as to use the sub-pixel interpolation to complete video encoding; the best integer pixel position after the integer pixel motion search in the motion estimation ME, divides a plurality of sub-pixel search areas at integer pixel points around the best integer pixel position, selects a sub-pixel search sub-area with a maximum probability from the plurality of sub-pixel search areas, and then performs sub-pixel search in the sub-area, thereby avoiding performing sub-pixel search in the above plurality of areas.

10. An electronic device comprising a processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, at least one program, a code set or an instruction set is loaded and executed by the processor to implement the method according to any one of claims 1 to 8.

11. A computer-readable medium having stored thereon at least one instruction, at least one program, code set or instruction set, wherein the at least one instruction, at least one program, code set or instruction set is loaded and executed by a processor to implement the method according to any one of claims 1 to 8.

Images