A video residual data processing method and system

Abstract

The invention discloses a video residual data processing method and system, belonging to the field of H.264 video compression implementation and hardware technology, comprising: applying for residual storage space; completing the big-endian conversion of the residual according to the high-low bit switching mode of four bytes; calculating the offset position according to the small block sequence number, and storing the residual in the specified memory index; setting the address and length of the storage space and notifying the VCA register; releasing the occupied mark lock, and notifying the VCA that the data preparation is complete; the VCA processing thread obtains the storage space use mark lock, and reads the residual for encoding; the video residual data processing method and system, by using a shared memory mode, before the residual data calculation result is sent to the VCA, it is placed in the memory in advance according to the layout order and end order requirements required by the VCA, and is directly set to the input end of the VCA, avoiding the data copying process twice, thereby reducing the data processing time.

Description

Technical Field

[0001] The invention belongs to the field of H.264 video compression implementation and hardware technology, and in particular relates to a video residual data processing method and system. Background Art

[0002] In video compression technologies represented by H.264, macroblocks are usually used as the basic unit of data processing. After multiple macroblocks are compressed, a compressed bit stream in units of video frames is formed. In the compression operation of macroblock data, the DCT (discrete cosine transform) transformation and quantization operation experienced by the residual part are the calculation steps with the largest data volume and the most frequent memory access in the entire compression encoding stage. Therefore, in the ASIC (application-specific integrated circuit) design planning of video codecs, how to efficiently handle the interaction between software and hardware, or the residual data between functional modules, is a key link in the design and optimization of the hardware of the codec part.

[0003] However, in previous VCA designs, the format conversion and endian arrangement issues of residual data are often mentioned: the front end of VCA, that is, the external software or hardware, calculates the original residual of the image and usually stores it in the memory in the form of a two-dimensional array with a length and width of 16x16, and in little-endian byte order. After the data is processed by DCT Fourier transform and quantization, it is output to the entropy encoder to further generate a compressed bit stream; on the other hand, after dequantization and inverse discrete cosine transform, it needs to be converted again in the process of combining with the external motion compensation module, because the reconstruction data function module of the external software and hardware usually requires the data to be provided in little-endian byte order and 16x16 two-dimensional array format. Existing technical processing usually involves one conversion and copy operation, which is inefficient. Therefore, it is necessary to develop a new video residual data processing method and system to solve the existing problems. Summary of the invention

[0004] The object of the present invention is to provide a video residual data processing method and system to solve the problem of low efficiency in video residual data processing.

[0005] To achieve the above object, the present invention provides the following technical solution: a video residual data processing method, comprising the following steps:

[0006] Apply for residual storage space;

[0007] The residue is converted into big-endian and small-endian by switching the high and low bits of four bytes;

[0008] Calculate the offset position according to the small block number and store the residual at the specified memory index position;

[0009] Set the address and length of the storage space and notify the VCA register;

[0010] Release the occupied mark lock and notify VCA that data preparation is complete;

[0011] The VCA processing thread obtains the storage space using the marker lock and reads the residual data for encoding.

[0012] Preferably, the storage space for applying for residuals includes a storage space in the memory, and the storage space is set as the storage location of the residual data.

[0013] Preferably, after applying for the residual storage space, if the macroblock for data processing runs in a multi-threaded mode, a mutex lock for marking the exclusive use of the storage space is set.

[0014] Preferably, the residual calculation method comprises: performing a secondary loop subtraction operation in the length and width directions on the two-dimensional array of original pixels and predicted pixels respectively.

[0015] Preferably, the switching method to complete the big-endian and little-endian conversion includes: in bytes, for the residual result of each pixel, the most important byte is stored in the low order of the memory in units of 4 bytes, and the least important byte is stored in the high order of the memory in units of 4 bytes, thereby realizing the conversion from little end to big end.

[0016] Preferably, the method for calculating the offset position includes: according to the position of the current residual in the 16x16 two-dimensional array, calculating the position sequence number of the position in the memory composed of 16 4x4 small blocks.

[0017] Preferably, the deduction method includes: dividing 16 pixel blocks according to 4x4 pixels, obtaining the sequence value in the pixel block, and multiplying the coordinates of the pixel block by 4 through the block_idx_xy_fenc array to obtain the actual position in the 16x16 two-dimensional array.

[0018] Preferably, setting the address and length of the storage space and notifying the VCA register includes: notifying the VCA of the address and length information of the residual data involved in the macroblock operation, and calling the VCA compression command to start calculation.

[0019] Preferably, when the calculation of the residual and / or the big-endian and small-endian conversion is performed by a hardware computing unit, a cache invalidate operation is performed after the output.

[0020] Preferably, before notifying the VCA that data preparation is complete, a cache flush operation is performed on the storage space being used.

[0021] The present invention further provides a video residual data processing system, comprising:

[0022] The space application module is used to apply for residual storage space;

[0023] The big-endian conversion module is used to convert the residual into a big-endian sequence by switching the high and low bits of four bytes;

[0024] The offset position module is used to calculate the offset position according to the pixel block number and store the residual in the specified memory index position;

[0025] The storage space setting module is used to set the address and length of the storage space and notify the VCA register;

[0026] The marker lock release module is used to release the occupied marker lock and notify the VCA that the data preparation is complete;

[0027] The encoding module is used by the VCA processing thread to obtain the storage space using the mark lock and read the residual data for encoding.

[0028] The technical effects and advantages of the present invention are as follows: the video residual data processing method and system of the present invention, by using a shared memory method, under the technical conditions of satisfying the inter-core contention requirements and cache consistency, before the residual data calculation results are sent to the VCA, they are placed in the memory in advance according to the layout order and end order requirements required by the VCA. After the residual data calculation is completed, the memory address of the stored residual is directly set to the input end of the VCA, and then the operation resources are released to notify the VCA that the memory data can be directly used for the next step of processing, avoiding the two data copying processes, thereby reducing the data processing time. In the input and output interaction of each macroblock residual data, one data copying process is reduced; it can be directly used by the VCA, reducing the memory copying process. At the same time, the reconstructed residual reversely filled according to this rule when the VCA is output can also be directly used again by the external module, which is equivalent to reducing the data copying frequency once more. After experimental testing, the operation speed of each macroblock can save about 14% of the clock cycle consumption on average, greatly improving the data processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 Design block diagram for the hardware module of the present invention;

[0030] Figure 2 This is an example diagram of the conventional initial residual memory layout of the present invention;

[0031] Figure 3 The 16 4x4 small block residual index maps of the present invention;

[0032] Figure 4 This is an example diagram of the residual and 4x4 small block internal data layout of the present invention;

[0033] Figure 5An example diagram of the memory layout and endian conversion effect required by the VCA of the present invention;

[0034] Figure 6 This is a schematic diagram of the little-endian to big-endian conversion result of the present invention;

[0035] Figure 7 It is a schematic diagram of the conversion result of the present invention. DETAILED DESCRIPTION

[0036] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0037] The present invention provides a video residual data processing method. In the ASIC design based on the H.264 standard, the four functional links of discrete cosine transform (DCT), quantization, inverse quantization and inverse discrete cosine transform in the video compression step are designed as a hardware acceleration module (VCA). VCA is the abbreviation of video code accelerator. Figure 1 The dotted part, Figure 1 This is a functional module diagram of video data compression in the H.264 standard. The four functional modules in the dotted area, namely, discrete cosine transform (DCT), quantization, inverse quantization and inverse discrete cosine transform, are the functional modules designed as hardware in this solution. After the original image data is subtracted from the predicted data to generate the original residual, it is converted into frequency domain data through discrete cosine transform and the correlation is removed, and then sent to the quantization module for compression (after DCT transformation, the image signal has more high-frequency information data, but the amplitude is relatively small, and it can be easily quantized to 0 during the high-frequency information quantization process, thus achieving the purpose of compression); on the one hand, a shorter lossless bit stream is obtained through entropy coding, and on the other hand, reconstructed residual data is generated through inverse quantization and inverse discrete cosine transform, which is used as a reference image for the backward compressed frame.

[0038] Apply for a storage space, which is memory, and set the resource exclusive flag;

[0039] The calculated residual is converted into big-endian and small-endian by switching every 4 bytes; the residual is calculated for every 4 pixels (bytes) in each 4x4 pixel block, and the endianness is directly converted when stored;

[0040] The offset position is calculated according to the 4x4 small block sequence number, and the residual data is directly stored in the specified memory index, and copied one by one in bytes; the memory index is specifically included in the residual calculation of a 16x16 pixel, and each residual value occupies 1 byte. The first 4x4 residual block is stored in the 16-byte position of the memory index 0, the second 4x4 residual block is stored in the 16-byte position of the memory index 1, and so on. The small block sequence number refers to the sequence number of each small block with a length and width of 4 pixels in each macroblock with a length and width of 16 pixels respectively. The values ​​of the sequence numbers are 0 to 15, a total of 16 numbers, 16 pixels x 16 pixels = 4 pixels x 4 pixels x 16 (such 4 pixel x 4 pixel small blocks) = 256 pixels;

[0041] Set the address and length of the memory to the VCA register, and set the memory address to the register of the hardware acceleration module VCA;

[0042] The thread of the original residual calculation unit releases the occupied mark lock and notifies the VCA that the data preparation is complete; the original residual calculation unit refers to the step of calculating the residual and performing memory layout conversion and endian conversion;

[0043] The processing thread obtains the resource usage mark lock and directly fetches data for DCT calculation.

[0044] The specific method in this embodiment is as follows:

[0045] Step 1: Set a fixed memory or dynamic memory as the initial residual storage location. If it is designed as a macro block under multi-threading, a mutex lock needs to be allocated to mark the exclusive use of the memory resource; the mutex lock is a software system call, the purpose of which is to prevent conflicting access between multiple processes;

[0046] Step 2: The residual is the difference between the original video pixel data and the predicted value. Its implementation depends on the design plan. If it is a pure software implementation, the difference can be calculated by operating the two-dimensional array in a secondary iteration to perform a subtraction operation; the secondary iteration refers to the calculation method for obtaining the residual: the two-dimensional array of the original pixels and the predicted pixels is subtracted twice in the length and width directions respectively. In each 4x4 pixel block, according to the size of 16 pixels, the residual calculation and memory layout and endian conversion are performed in the length and width directions respectively. The converted memory address pVca_mdp is the first address that VCA can directly perform compression processing.

[0047] Step 3: The residual result of each pixel in bytes needs to be stored in the low memory of 4 bytes, and the least significant byte is stored in the high memory of 4 bytes to achieve the conversion from little end to big end; the most significant byte refers to the highest byte in the four-byte number of an integer type; the least significant byte refers to the lowest byte in the four-byte number of an integer type; little end to big end: the end order refers to whether it is currently in big end order or little end order. The big end order is that the highest byte in the four bytes of the integer type is placed in the lowest byte position in the memory, and the little end order is that the lowest byte in the four bytes of the integer type is placed in the lowest byte position in the memory.

[0048] Step 4: Based on the 16x16 two-dimensional array position of the current residual, calculate the memory position numbers of the 16 4x4 small blocks required in VCA, such as Figure 3 Hardware Accelerator (VCA) 16 4x4 small block index maps and Figure 4 An example diagram of the internal data layout of VCA, with skip order placement. The calculation method is that each 16x16 pixel block has a total of 256 pixels, which is a combination of 16 4x4 small pixel blocks; Figure 2 and Figure 4 The above are the original arrangement of 16x16 pixel residual data calculated by the system (referring to software or other hardware devices) in memory, where each pixel residual data occupies 1 byte of memory unit. However, when the system generates these residual data, it is only stored in memory according to the size of each small block of 16 pixels (bytes), and a total of 16 small blocks are stored in memory, which is not conducive to fast processing by VCA. Therefore, before sending them to VCA for compression processing, they are converted into memory layout and endian requirements suitable for fast compression by VCA, that is, Figure 5 The memory layout is shown in Figure 1. The conversion effect is as follows: Figure 6 As shown, it can be seen that in Figure 4 The initial residual memory layout is converted to Figure 5 In the process of residual memory layout required by VCA, the data arrangement method has changed, and the endianness has changed every 4 bytes. For example, the integer byte before conversion is little endian, that is, 0x63626160, and after conversion, it becomes big endian 0x60616263;

[0049] idx_b4 is a serial number with an input value range of 0 to 15. It is used as the array subscript of the block_idx_xy_fenc variable to query the 16x16 two-dimensional array position, and then converted into a 4x4 small block memory position serial number; Figure 7As shown, both sides are 16x16 pixel data. Due to calculation needs, they are first divided into 16 small blocks of 4x4 pixels, so idx_b4 represents any serial number value of these 16 4x4 pixel blocks, ranging from 0 to 15. The left side is the original memory layout for calculating pixel residuals, which needs to be converted into a memory layout that is convenient for VCA processing, through the block_idx_xy_fenc array; the block_idx_xy_fenc array is customized as:

[0050] #define FENC_STRIDE 16

[0051] static const uint8_t block_idx_xy_fenc

[16] =

[0052] {

[0053] 0*4 + 0*4*FENC_STRIDE, 1*4 + 0*4*FENC_STRIDE,

[0054] 0*4 + 1*4*FENC_STRIDE, 1*4 + 1*4*FENC_STRIDE,

[0055] 2*4 + 0*4*FENC_STRIDE, 3*4 + 0*4*FENC_STRIDE,

[0056] 2*4 + 1*4*FENC_STRIDE, 3*4 + 1*4*FENC_STRIDE,

[0057] 0*4 + 2*4*FENC_STRIDE, 1*4 + 2*4*FENC_STRIDE,

[0058] 0*4 + 3*4*FENC_STRIDE, 1*4 + 3*4*FENC_STRIDE,

[0059] 2*4 + 2*4*FENC_STRIDE, 3*4 + 2*4*FENC_STRIDE,

[0060] 2*4 + 3*4*FENC_STRIDE, 3*4 + 3*4*FENC_STRIDE

[0061] };

[0062] For example, when idx_b4=2 on the left (row 0, column 2 according to the 4x4 pixel block, multiplied by 4 according to the pixel coordinates, so it is row 0, column 8), block_idx_xy_fenc

[12] =0*4+1*4*16, so the residual data of the 4x4 block should be saved at the memory offset of 1*4*16, or 64 bytes, after conversion. In fact, it is 2 of the VCA memory layout (row 1, column 0 according to the 4x4 pixel block, multiplied by 4 according to the pixel coordinates, so it is row 4, column 0):

[0063] The purpose of the above memory conversion is to convert the 16x16 pixel residual calculated by the system in the memory into a residual format stored in another memory block, which is convenient for VCA to process, in units of 16 4x4 pixel blocks.

[0064] Step 5: If other hardware computing units calculate the residual and endian conversion, the system interface needs to be called to perform a cache invalidate operation in advance after the output. In this embodiment, steps 1-4 are all software methods; but as an implementation option, the hardware computing module referred to in step 5 can be used, and VDSP is also the name of one of the hardware modules;

[0065] Step 6: After each thread macroblock completes residual calculation, it starts thread unlocking operation; the data conversion effect is changed from the initial Figure 2 The example diagram of the general initial residual memory layout becomes Figure 6 Example diagram of layout and endian conversion effect; thread unlocking operation is to unlock the mutex lock of multiple threads, which is implemented by executing software system calls;

[0066] Step 7: For the fixed memory or dynamic memory used, before notifying VCA to fetch data, it is necessary to call a system interface to perform a cache flush operation in advance to ensure that the cache data has been updated to the memory;

[0067] Step 8: Notify VCA of the residual address and length of this macroblock operation, and call the VCA compression command to start calculation.

[0068] The present invention further provides a video residual data processing system, comprising:

[0069] The space application module is used to apply for residual storage space;

[0070] The big-endian conversion module is used to convert the residual into a big-endian sequence by switching the high and low bits of four bytes;

[0071] The offset position module is used to calculate the offset position according to the pixel block number and store the residual in the specified memory index position;

[0072] The storage space setting module is used to set the address and length of the storage space and notify the VCA register;

[0073] The marker lock release module is used to release the occupied marker lock and notify the VCA that the data preparation is complete;

[0074] The encoding module is used by the VCA processing thread to obtain the storage space using the mark lock and read the residual data for encoding.

[0075] The video residual data processing method and system, by using a shared memory method, under the technical conditions of meeting the requirements of inter-core contention and cache consistency, before sending the residual data calculation results to VCA, they are placed in the memory in advance according to the layout order and end order requirements required by VCA. After the residual data calculation is completed, the memory address of the stored residual is directly set to the input end of VCA, and then the operation resources are released to notify VCA that the memory data can be directly used for the next step of processing, avoiding the two data copying processes, thereby reducing the data processing time. In the input and output interaction of each macroblock residual data, one data copying process is reduced; it can be directly used by VCA, reducing the memory copying process. At the same time, the reconstructed residual reversely filled according to this rule when VCA outputs can also be directly used again by the external module, which is equivalent to reducing the frequency of data copying once more. After experimental testing, the operation speed of each macroblock can save about 14% of the clock cycle consumption on average, greatly improving the data processing efficiency.

[0076] Finally, it should be noted that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A video residual data processing method, characterized by: include: Apply for residual storage space; The residue is converted into big-endian and small-endian by switching the high and low bits of four bytes; Calculate the offset position according to the pixel block number and store the residual at the specified memory index position; Set the address and length of the storage space and notify the VCA register; Release the occupied mark lock and notify VCA that data preparation is complete; The VCA processing thread obtains the storage space usage mark lock and reads the residual data for encoding; The method for calculating the residual in the big-endian conversion by switching the high and low bits of the four bytes includes: performing a subtraction operation of two cycles in the length and width directions on the two-dimensional arrays of the original pixels and the predicted pixels respectively; The method for converting the big-endian sequence includes: taking bytes as units, for the residual result of each pixel, storing the most important byte in the low order of the memory in units of 4 bytes, and storing the least important byte in the high order of the memory in units of 4 bytes, so as to realize the conversion from little end to big end; The method for calculating the offset position includes: according to the position of the current residual in the 16x16 two-dimensional array, calculating the position sequence number of the position in the memory composed of 16 4x4 small blocks.

2. A video residual data processing method according to claim 1, characterized in that: The applying for the residual storage space includes applying for a storage space in the memory and setting it as the storage location of the residual data.

3. The video residual data processing method according to claim 1, characterized in that: After applying for the residual storage space, if the macroblock for data processing runs in a multi-threaded mode, a mutex lock for marking the exclusive use of the storage space is set.

4. The video residual data processing method according to claim 1, characterized in that: The calculation method includes: dividing 16 pixel blocks according to 4x4 pixels, obtaining the sequence number value in the pixel block, multiplying the coordinates of the pixel block by 4 through the block_idx_xy_fenc array, and obtaining the actual position in the 16x16 two-dimensional array.

5. The video residual data processing method according to claim 1, characterized in that: The setting of the address and length of the storage space and notifying the VCA register includes: notifying the VCA of the address and length information of the residual data involved in the macroblock operation, and calling the compression command of the VCA to start calculation.

6. A video residual data processing method according to claim 1, characterized in that: When the calculation of the residual and / or the big-endian and small-endian conversion is performed by a hardware computing unit, a cache invalidation operation is performed after the output; Before notifying the VCA that data preparation is complete, a cache refresh operation is performed on the storage space being used.

7. A system for implementing the video residual data processing method according to any one of claims 1 to 6, characterized in that: include: The space application module is used to apply for residual storage space; The big-endian conversion module is used to convert the residual into a big-endian sequence by switching the high and low bits of four bytes; The offset position module is used to calculate the offset position according to the pixel block number and store the residual in the specified memory index position; The storage space setting module is used to set the address and length of the storage space and notify the VCA register; The marker lock release module is used to release the occupied marker lock and notify the VCA that the data preparation is complete; The encoding module is used by the VCA processing thread to obtain the storage space using the mark lock and read the residual data for encoding.

Images