Method, device, medium and system for image processing in video encoding device

Abstract

The present application discloses a method, device, storage medium, electronic device and system for image processing in a video encoding device. The method includes: determining a block to be encoded from a current frame image; determining a first area that needs to be read repeatedly from a reconstructed frame image of the current frame image, and storing the image data of the first area in a preset memory, wherein the power consumption of the preset memory is less than a preset power consumption threshold; reading the image data of the first area from the preset memory; determining a matching block that matches the block to be encoded from the first area according to the image data of the first area; encoding the block to be encoded according to the relative relationship between the matching block and the block to be encoded. The present application can reduce the power consumption of a video encoding device.

Description

Technical Field

[0001] The present application belongs to the technical field of electronic equipment, and in particular, relates to a method, device, storage medium, electronic equipment and system for performing image processing in a video encoding device. Background Art

[0002] With the continuous development of technology, the functions of video encoders (VENC) are becoming more and more powerful. Video encoders can encode video images. When encoding a frame of video images, it is usually necessary to read the data of multiple frames of encoded video images. However, in the related art, when reading the data of the encoded video images, the power consumption of the video encoder is relatively large. Summary of the invention

[0003] Embodiments of the present application provide a method, device, storage medium, electronic device, and system for performing image processing in a video encoding device, which can reduce the power consumption of the video encoding device.

[0004] In a first aspect, an embodiment of the present application provides a method for performing image processing in a video encoding device, the method comprising:

[0005] Determine the block to be encoded from the current frame image;

[0006] Determine a first area that needs to be repeatedly read from the reconstructed frame image of the current frame image, and store image data of the first area in a preset memory, wherein power consumption of the preset memory is less than a preset power consumption threshold;

[0007] Reading the image data of the first area from the preset memory;

[0008] Determining a matching block that matches the block to be encoded from the first area according to the image data of the first area;

[0009] The block to be encoded is encoded according to the relative relationship between the matching block and the block to be encoded.

[0010] In a second aspect, an embodiment of the present application provides a device for performing image processing in a video encoding device, the device comprising:

[0011] A first determination module, used to determine a block to be encoded from a current frame image;

[0012] A second determination module is used to determine a first area that needs to be repeatedly read from the current frame image, and store the image data of the first area in a preset memory, wherein the power consumption of the preset memory is less than a preset power consumption threshold;

[0013] A reading module, used for reading the image data of the first area from the preset memory;

[0014] A third determination module, configured to determine a matching block matching the block to be encoded from the first area according to the image data of the first area;

[0015] The encoding module is used to encode the block to be encoded according to the relative relationship between the matching block and the block to be encoded.

[0016] In a third aspect, an embodiment of the present application provides a storage medium on which a computer program is stored. When the computer program is executed on a computer, the computer executes the method for performing image processing in a video encoding device provided in an embodiment of the present application.

[0017] In a fourth aspect, an embodiment of the present application further provides an electronic device, comprising a memory, a processor and a video encoding device, wherein the processor is used to execute the method for performing image processing in the video encoding device provided in an embodiment of the present application by calling a computer program stored in the memory.

[0018] In a fifth aspect, an embodiment of the present application further provides an image processing system, comprising a video encoding device, a first memory and a second memory, wherein the power consumption of the second memory is greater than a first preset multiple of the power consumption of the first memory, the video encoding device comprises a third memory, the reading speed of the third memory is greater than a second preset multiple of the reading speed of the first memory, the first memory and the second memory respectively store image data that needs to be repeatedly read multiple times in a reconstructed frame image of a current frame image, and when encoding, the video encoding device reads the image data that needs to be repeatedly read multiple times from the first memory according to a first preset number of times and from the second memory according to a second preset number of times, and determines the image data within a search window (Search Window, SWin) from the image data that needs to be repeatedly read multiple times, and stores the image data within the search window in the third memory, the video encoding device reads the image data within the search window from the third memory, and determines a matching block that matches the block to be encoded, and encodes according to the block vector (Block Vector) and residual of the matching block and the block to be encoded.

[0019] In an embodiment of the present application, the video encoding device can determine the block to be encoded from the current frame image, determine the first area that needs to be repeatedly read from the reconstructed frame image of the current frame image, and store the image data of the first area in a preset memory, the power consumption of the preset memory is less than the preset power consumption threshold. Then, the image data of the first area is read from the preset memory, and a matching block that matches the block to be encoded is determined from the first area based on the image data of the first area. Afterwards, the block to be encoded is encoded based on the relative relationship between the matching block and the block to be encoded. That is, in an embodiment of the present application, the purpose of reducing the power consumption of the video encoding device is achieved by storing the image data of the first area in a preset memory with lower power consumption. Therefore, the embodiment of the present application can reduce the power consumption of the video encoding device. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The technical solution and beneficial effects of the present application will be made apparent by describing in detail the specific implementation methods of the present application in conjunction with the accompanying drawings.

[0021] Figure 1 It is a flowchart of a method for performing image processing in a video encoding device provided in an embodiment of the present application.

[0022] Figure 2 It is a structural diagram of a video compression system in the related art.

[0023] Figure 3 It is a schematic diagram of data storage in a video encoding device in the related art.

[0024] Figure 4 It is a schematic diagram of increasing the number of channels of dynamic random access memory (DRAM) for data access in the related art.

[0025] Figure 5 It is a schematic diagram of the comparison of square blocks in the related art.

[0026] Figure 6 It is a schematic diagram of the hierarchical search provided in the embodiment of the present application.

[0027] Figure 7 It is a schematic diagram of a non-hierarchical search provided in an embodiment of the present application.

[0028] Figure 8 It is a schematic diagram comparing the search objects of inter-frame motion estimation (Inter-frame ME) and intra-block copy (Intra block copy, IntraBC) provided in an embodiment of the present application.

[0029] Fig. 9This is another flowchart of a method for performing image processing in a video encoding device provided in an embodiment of the present application.

[0030] Fig.10 It is a schematic diagram of a scene for searching in a reconstructed frame image of a current frame image provided by an embodiment of the present application.

[0031] Fig.11 It is a schematic diagram comparing the power consumption of a static random access memory (SRAM) and a dynamic random access memory when reading data provided by an embodiment of the present application.

[0032] Fig.12 It is a schematic diagram of the architecture of a video compression system using a system cache (Sys$) provided in an embodiment of the present application.

[0033] Fig.13 It is another architectural diagram of a video compression system using a system cache (Sys$) provided in an embodiment of the present application.

[0034] Fig.14 It is a schematic diagram of the architecture of a video compression system using a system buffer memory (System Buffer, SysBuf) provided in an embodiment of the present application.

[0035] Fig.15 This is a schematic diagram of a scenario when moving down one coding block row provided in an embodiment of the present application.

[0036] Fig.16 This is a schematic diagram of a power consumption curve when reading data from a multi-channel DRAM provided in an embodiment of the present application.

[0037] Fig.17 It is a schematic diagram of power consumption curves when reading data from Sys$ or SysBuf and DRAM respectively, provided in an embodiment of the present application.

[0038] Fig.18 It is a scene schematic diagram of the search range of the search window in the reconstructed frame image of the current frame image provided by an embodiment of the present application.

[0039] Fig.19 It is a schematic diagram of a scene encoded by a video encoding device provided in an embodiment of the present application.

[0040] Fig. 20 It is a structural schematic diagram of a device for performing image processing in a video encoding device provided in an embodiment of the present application.

[0041] Fig.21 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0042] Fig. 22 It is another structural schematic diagram of the electronic device provided in an embodiment of the present application.

[0043] Fig.23 It is a structural diagram of the image processing system provided in an embodiment of the present application.

[0044] Fig.24 It is another structural schematic diagram of the image processing system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0045] Please refer to the drawings, where the same component symbols represent the same components. The principle of the present application is illustrated by implementing it in an appropriate computing environment. The following description is based on the illustrated specific embodiments of the present application, which should not be considered as limiting other specific embodiments of the present application that are not described in detail herein.

[0046] See also Figure 1 , Figure 1 1 is a flow chart of a method for performing image processing in a video encoding device provided by an embodiment of the present application. The method for performing image processing in a video encoding device can be applied to a video encoding device. The flow of the method for performing image processing in a video encoding device may include:

[0047] 101. Determine a block to be encoded from the current frame image.

[0048] With the continuous development of technology, the functions of video encoding devices are becoming more and more powerful. Video encoding devices can encode video images. When encoding a frame of video image, it is usually necessary to read the amount of data of multiple frames of encoded video images. However, in the related art, when reading the data of the encoded video image, the power consumption of the video encoding device is relatively large.

[0049] See also Figure 2 , Figure 2The figure is a schematic diagram of the structure of the video compression system in the related art. In the video compression system, the central processing unit (CPU), video encoding device, display processor (DISP) and neural network processor (NPU) read and write data from DRAM through the bus and dynamic random access memory controller (DRAMC). The central processing unit, video encoding device, display processor and neural network processor share the bandwidth in a time-sharing manner. The priority of the central processing unit, display processor and neural network processor is higher than that of the video encoding device. The video encoding device needs to perform a search action when encoding, which will occupy a large bandwidth.

[0050] Video encoding devices attach great importance to cost. When buffering frame images, in order to achieve the lowest cost and the highest production yield, DRAM is usually used as the main storage space. Figure 3 , Figure 3 The figure is a schematic diagram of data storage in a video encoding device in the related art. Among them, the current frame image, the reference frame image, the reconstructed frame image, the bitstreams and the temporary data are all stored in the DRAM in the video encoding device. The video encoding device has a screen content coding (Screen Content Coding, SCC) function.

[0051] It should be noted that after encoding the current frame image, it becomes a reconstructed frame image, and the reconstructed frame image of the current frame image can be used as a reference frame image for the next frame image. Similarly, after encoding the previous frame image, it becomes a reconstructed image of the previous frame image, and the reconstructed image of the previous frame image can be used as a reference frame image for the current frame image. Temporal data can be temporal motion vectors (TMV), scaled frames, or other data. However, the bandwidth provided by DRAM is relatively small.

[0052] With the emergence of new video standards, such as High Efficiency Video Coding (H.265 / HEVC), Versatile Video Coding (H.266 / VVC), Alliance for Open Media Video 1 (AV1), Essential Video Coding (MPEG-5 / EVC), etc., which are targeting larger and larger screen sizes and higher and higher frame rates, it is common to increase DRAM bandwidth or increase dynamic random access memory frequency to achieve accelerated data throughput.

[0053] Even though the hierarchical search reduces the problem of multiple reconstruction frame image reading by copying blocks within the screen, in the case of large size and high frame rate, such as 4096×2160 resolution (i.e. 4K resolution) or 7680x4320 resolution (i.e. 8K resolution), a higher DRAM throughput is required. The increase in throughput is usually achieved by increasing the number of DRAM channels, which will cause excessive energy consumption.

[0054] See also Figure 4 , Figure 4 This is a schematic diagram of increasing the number of DRAM channels for data access in the related art. By increasing the number of DRAM channels, the bandwidth can be increased and the frequency can be increased to increase the speed at which DRAM can handle data, but this will cause greater power consumption. For example, in order to meet the requirements of the video encoding device to achieve the reading speed, the bandwidth of the system DRAM consumes a lot of energy. However, regardless of whether the video encoding device is performing real-time or non-real-time operations, it is very important to maintain the highest efficiency. The method in the related art will cause great power consumption of the DRAM when the video encoding device is completed within the expected time.

[0055] Video encoding devices generally use blocks (i.e., squares) as basic units. The blocks can be rectangular, square, trapezoidal, or triangular. Under such conditions, comparison algorithms based on blocks appear. Please refer to Figure 5 , Figure 5 It is a schematic diagram of the comparison of square blocks in the related art. The square blocks are used to compare the blocks to be compressed in the current frame image with the blocks of the reference frame image, and the reference frame image is a reconstructed image of the previous frame image, wherein the blocks to be compressed and the blocks of the reference frame image are N×N blocks, where N is an integer greater than or equal to 4. By comparing the blocks, the information redundancy in the time domain can be minimized to achieve the effect of compressing data. Figure 5This is an example of alignment based on square blocks, but the same alignment method can also be used for blocks made up of rectangles, trapezoids or triangles.

[0056] In the embodiment of the present application, when the block is copied within the picture, the image is divided into a plurality of non-overlapping blocks, which form a rectangular array, wherein each block is a block of N×N size, for example, it can be a 4×4 block, a 32×32 block, a 128×128 block, etc., wherein 4×4, 32×32, 128×128 refer to the number of pixels. For each coded block, the reconstructed frame image of the current frame image is searched for the block that best matches it around the block to be coded, and the relative displacement of the best matching block relative to the block to be coded can be considered as a block vector.

[0057] In the embodiment of the present application, a block to be encoded is determined from the current frame image, and the block to be encoded is a block to be compressed in the current frame image, that is, a block to be encoded in the current frame image. The block to be encoded may be a block of size N×N. When encoding the block to be encoded, it is usually necessary to compare it with a block in a reconstructed frame image of the current frame image, so it is necessary to search for the block to be compared in the reconstructed frame image of the current frame image.

[0058] 102. Determine a first region that needs to be repeatedly read from a reconstructed frame image of a current frame image, and store image data of the first region in a preset memory, wherein power consumption of the preset memory is less than a preset power consumption threshold.

[0059] For example, only after searching for the block that needs to be compared from the reconstructed frame image of the current frame image can the block be compared with the block to be encoded. It should be noted that when searching for blocks in the reconstructed frame image of the current frame image, it is necessary to know the search range (search range, SRng) in the reconstructed frame image of the current frame image, that is, the first area. When searching for blocks in the first area, it is necessary to repeatedly read the image data of the first area. Therefore, in the embodiment of the present application, after determining the first area that needs to be read repeatedly from the reconstructed frame image of the current frame image, the image data of the first area is stored in a preset memory to facilitate the reading of image data during subsequent encoding. In addition, the power consumption of the preset memory is less than the preset power consumption threshold. By using a preset memory with low power consumption to access image data, the power consumption of the video encoding device can be reduced.

[0060] 103. Read image data of the first area from a preset memory.

[0061] For example, after the image data of the first area is stored in the preset memory, when searching for blocks in the first area, the image data of the first area needs to be read from the preset memory to find a block that best matches the block to be encoded.

[0062] 104. Determine a matching block that matches the block to be encoded from the first area according to the image data of the first area.

[0063] For example, by reading the image data of the first area stored in the preset memory, the search for the first area is realized. During the search process, each block in the first area is compared with the block to be encoded in the current frame image, and the block that best matches the block to be encoded is found from the first area. The best matching block is the matching block.

[0064] As screen content coding becomes more and more important, new video standards such as H.265 / HEVC, AV1, and Versatile Video Coding (VVC) have incorporated screen content coding tools to optimize compression of common screen graphics. A common coding option in screen content coding is intra-screen block replication. This technology uses a common block search and matching algorithm, and the implementation method can be the same as motion estimation (ME). It can use hierarchical search or non-hierarchical search. After the search, the block vector is obtained for further compression coding.

[0065] See also Figure 6 , Figure 6 Schematic diagram of hierarchical search provided by an embodiment of the present application. Hierarchical search is to reduce the block to be searched and the area to be searched (such as the first area) by the same factor, such as 1 / 2, 1 / 4 or 1 / 8, etc. On the reduced image, the approximate range of the block to be searched is first determined, and then the unreduced image is returned to perform a more detailed block search. In the hierarchical search, the reduction factor of each level can be the same or different, for example, the reduction factor of each level can be 1 / 2, 1 / 4, 1 / 8 and 1 / 16. Figure 6 Taking a three-level search as an example, the 1 / 4 reduced image is searched first, then the block vector is obtained from the 1 / 4 reduced range image, a more detailed and smaller search is performed in the 1 / 2 reduced range image, and then the block vector obtained from the 1 / 2 reduced range image is searched in the original size image to obtain the final block vector.

[0066] See also Figure 7 , Figure 7 Schematic diagram of non-hierarchical search provided in the embodiment of the present application. Non-hierarchical search refers to performing block comparison tasks directly in the unreduced image, and common methods such as full search and n-step search can be seen in public documents. Figure 7It is to directly perform intra-frame motion search on the original size image, that is, to perform intra-frame motion estimation block by block in the reconstructed frame image of the current frame image, and to use a full search method to find which block has the smallest residual between the current block and the reconstructed frame pixels in the intra-frame search region. That is, after searching the reconstructed frame image of the current frame image, when encoding the current block in the current frame image, the block with the smallest residual with the current block in the reconstructed frame image of the current frame image is used as the matching block, and the matching block is the reference block.

[0067] See also Figure 8 , Figure 8 It is a comparative schematic diagram of the inter-frame motion estimation and intra-screen block copy search objects provided by the embodiment of the present application. Motion estimation is a mechanism for estimating the motion parameters between a coded image (coding picture) and a reference image (reference picture). Motion estimation is generally only performed on the coding side, and the basic unit of its estimation is a block. The residual between the block obtained by estimating the reference image and the block to be encoded is then converted and encoded. The search methods for intra-screen block copy and inter-frame motion estimation can be the same, but the objects are different. The object of the intra-screen block copy search is the reconstructed pixels that have been encoded in the current frame image, and the block vector is obtained after the search. However, inter-frame motion estimation searches for reconstructed pixels that have been encoded at different previous time points, that is, reconstructed pixels that have been encoded in historical frame images, or pixels in reconstructed frame images of historical frame images.

[0068] It should be noted that in the embodiment of the present application, the intra-frame motion vector is called a block vector. The block vector is the object of coding. When encoding the screen content, in the intra-frame block copy mode, the reconstructed frame image of the current frame image can be searched. After the matching block is searched, the relative displacement between the matching block and the block to be coded is the block vector, that is, Figure 8 The relative displacement between the predictor and the current prediction unit.

[0069] It can be understood that the position of the matching block in the reconstructed frame image of the current frame image can be determined according to the block vector.

[0070] 105. Encode the block to be encoded according to the relative relationship between the matching block and the block to be encoded.

[0071] For example, based on the relative displacement and relative error relationship between the matching block and the block to be encoded, such as subtracting the two-dimensional pixels of the block to be encoded from the two-dimensional pixels of the corresponding position of the matching block to obtain the relative error relationship between the matching block and the block to be encoded, the block to be encoded can be encoded based on the relative displacement and relative error relationship between the matching block and the block to be encoded.

[0072] It can be understood that in the embodiment of the present application, the video encoding device can determine the block to be encoded from the current frame image, determine the first area that needs to be read repeatedly from the reconstructed frame image of the current frame image, and store the image data of the first area in a preset memory, the energy consumption of the preset memory is less than the preset energy consumption threshold. Then, the image data of the first area is read from the preset memory, and a matching block that matches the block to be encoded is determined from the first area based on the image data of the first area. Afterwards, the block to be encoded is encoded based on the relative relationship between the matching block and the block to be encoded. That is, in the embodiment of the present application, the image data of the first area is stored in a preset memory with lower energy consumption, so as to achieve the purpose of reducing the power consumption of the video encoding device. Therefore, the embodiment of the present application can reduce the power consumption of the video encoding device.

[0073] See also Fig. 9 , Fig. 9 Another flowchart of a method for performing image processing in a video encoding device provided in an embodiment of the present application is provided. The method for performing image processing in a video encoding device can be applied to a video encoding device. The flow of the method for performing image processing in a video encoding device may include:

[0074] 201. Determine a block line to be encoded from the current frame image.

[0075] For example, each frame image can be divided into multiple block rows, and each block row can be divided into multiple blocks. Before determining the block to be encoded of the current frame image, it is necessary to determine the block row to be encoded from the current frame image. The block row to be encoded refers to the block row where the block to be encoded is located.

[0076] 202. Determine a block to be encoded from the row of blocks to be encoded.

[0077] For example, after determining the row of blocks to be encoded, it is necessary to determine the block to be encoded from the multiple blocks in the row of blocks to be encoded. In the row of blocks to be encoded, the blocks before the block to be encoded are all encoded blocks. Fig.10 , Fig.10 It is a schematic diagram of a scene for searching in a reconstructed frame image of a current frame image provided by an embodiment of the present application. Fig.10 In the block row to be encoded, the block located on the left side of the block to be encoded is the encoded block, and the multiple block rows located above the block row to be encoded are the encoded block rows.

[0078] 203. Determine a first region that needs to be repeatedly read from a reconstructed frame image of the current frame image.

[0079] For example, after determining the block to be encoded, it is necessary to determine a first region that needs to be read repeatedly from the reconstructed frame image of the current frame image, and the first region may include multiple block rows, wherein each block row includes multiple blocks, and the multiple blocks are arranged in a row.

[0080] 204. Store the image data of the first area in the first memory and the second memory respectively.

[0081] For example, the preset memory includes a first memory and a second memory. When a first area that needs to be read repeatedly is determined from the reconstructed frame image of the current frame image, the image data of the determined first area is stored in the first memory and the second memory respectively, wherein the power consumption of the second memory is greater than a first preset multiple of the power consumption of the first memory, and the sum of the power consumption of the first memory and the power consumption of the second memory is less than a preset power consumption threshold, so that the power consumption when reading and writing data can be reduced. The preset power consumption threshold can be considered as the power consumption when all the image data of the first area is read and written by the second memory.

[0082] For example, both the first memory and the second memory are memories outside the video encoding device. For example, the first memory may include a system cache or a system buffer memory arranged outside the video encoding device, that is, the first memory may include a Sys$ or a SysBuf arranged outside the video encoding device, and the second memory may include a dynamic random access memory arranged outside the video encoding device, that is, the second memory may include a DRAM arranged outside the video encoding device. Of course, the first memory may also be other low-power memory outside the video encoding device, etc. The embodiment of the present application takes Sys$ or SysBuf as an example for explanation. Sys$ or SysBuf is composed of a plurality of SRAMs, and the second memory may be a DRAM. The power consumption of the DRAM is greater than a first preset multiple of the power consumption of the Sys$ or SysBuf outside the video encoding device, and the sum of the power consumption of the Sys$ or SysBuf and the power consumption of the DRAM is less than a preset power consumption threshold, so that the power consumption when reading and writing data can be reduced. The preset power consumption threshold can be considered as the power consumption when all the image data of the first area is read and written by the DRAM.

[0083] See also Fig.11 , Fig.11This is a schematic diagram comparing the energy consumption of static random access memory and dynamic random access memory when reading data provided by the embodiment of the present application. The energy consumed by reading data from SRAM differs by about 100 times from that consumed by reading data from DRAM, that is, the power consumption of reading data from SRAM is much less than that of reading data from DRAM. By storing the image data of multiple blocks in Sys$ or SysBuf, and DRAM, respectively, when reading the image data in Sys$ or SysBuf, the power consumption when reading data can be reduced.

[0084] The video encoding device uses the method of intra-frame block search. The intra-frame block copy search step requires a large bandwidth of DRAM, because during the search process, some related areas (first areas) in the reconstructed frame image of the current frame image are read for block search comparison. Due to cost considerations, the block rows covered by the search range (i.e., the first area) are usually not completely stored inside the video encoding device. Usually, only the required size within the search range, such as within the search window range, is stored to meet the high-speed data access requirements when copying intra-frame blocks.

[0085] If the image data of the first area is stored in the hardware of the video encoding device, that is, the cache or buffer, and the cache or buffer includes multiple SRAMs, then the SRAM inside the video encoding device needs to be divided into more units, each unit is an area (bank), which will cause the area of ​​a single bank to increase. As the area of ​​a single bank increases, the area of ​​the SRAM also increases, while the storage capacity of the SRAM remains unchanged, which results in higher costs. For example, taking a width of 8192 pixels and a vertical search range of -64 as an example, the 8-bit brightness (luma) part requires at least 512 kilobytes (KB) of storage space. If the chroma part is added, a larger storage space is required. In addition, due to the use of the intra-screen block copy algorithm, the SRAM needs to be divided into more units to meet the data in and out requirements, resulting in a larger area of ​​the SRAM.

[0086] It should be noted that using a search window to perform block copy search within the screen and process the compression of a block row will require data of multiple block rows in the reconstructed frame image of the current frame image, which means that processing one frame of data will require reading multiple frames of data.

[0087] For example, when a video encoding device performs a block vector search, see Fig.10, usually because the bandwidth required in the search window is very large, the larger the search range in the vertical direction, the larger the bandwidth occupied. The image data of the search window is stored in the cache or buffer inside the video encoding device. The cache or buffer includes a finer SRAM group. The finer the division, the larger the area of ​​the same storage unit. For example, the average area of ​​1 bit in the bank is larger than the average area of ​​1 bit in the SRAM. This can provide sufficient data bandwidth for the block vector search circuit for block replication within the screen. This not only causes the SRAM area to be large, but also makes the layout routing more difficult because it is divided into more banks. Therefore, the block rows of the entire first area will not be implemented using this method with a high storage unit (such as 1 bit) area. It should be noted that a block search circuit is used when performing a block replication search within the screen, and it can use a hierarchical or non-hierarchical search.

[0088] That is to say, when the block rows covered by the first area move down one block row during each encoding process, the first area will be captured again. Usually the vertical search range will be multiple times the height of the block to be encoded, which means that the bandwidth for reading the image data in the first area will be multiple times the bandwidth for writing the image data in the first area. This situation is even more serious when the picture to be encoded reaches 4K or 8K. The resolution of a 4K picture is 3840×2160 pixels, and the resolution of a 4K picture is 7680×4320 pixels. When encoding 4K and 8K pictures, the vertical search range must be larger than the 1080P resolution to a certain extent, otherwise the degree of picture compression will be greatly reduced.

[0089] In the embodiment of the present application, the first area includes a plurality of block rows, and storing the image data of the first area in the first memory and the second memory respectively in 204 may include:

[0090] If the first area is shifted down by one block line, the image data of the shifted block line is stored in the first memory and the second memory respectively;

[0091] Remove the block rows that are not used in encoding the next block row to be encoded from the first memory. For example, if the block row area (i.e., the first area) in the reconstructed frame image of the current frame image that needs to be read repeatedly is stored in Sys$ or SysBuf outside the video encoding device in advance, please refer to Figure 12 to Figure 14 , Fig.12 It is a schematic diagram of the architecture of a video compression system using a system cache provided in an embodiment of the present application. Fig.13 This is another architectural diagram of a video compression system using a system cache provided in an embodiment of the present application. Fig.14The schematic diagram of the architecture of the video compression system using the system buffer memory provided by the embodiment of the present application is shown in Figure 1. The image data of the first area is stored in Sys$ or SysBuf, and the cache or buffer inside the video encoding device stores the image data of the search window. Figure 12 to Figure 14 Where n is a number, indicating the size of the storage capacity. For example, in one embodiment, the speed of DRAM reading and writing data is 0.5 GB / s to 2 GB / s, the speed of Sys$ or SysBuf reading and writing data is 3 GB / s to 8 GB / s, and the speed of cache or buffer reading and writing data is 10 GB / s to 50 GB / s.

[0092] It should be noted that, in other embodiments, the speed at which DRAM reads and writes data, the speed at which Sys$ or SysBuf reads and writes data, and the speed at which cache or buffer reads and writes data may also be other values, but the speed at which cache or buffer reads and writes data is greater than the speed at which Sys$ or SysBuf reads and writes data and the speed at which DRAM reads and writes data, and the speed at which Sys$ or SysBuf reads and writes data is greater than the speed at which DRAM reads and writes data.

[0093] by Fig.12 For example, Sys$ can read data from DRAM through DramC, and the data read by Sys$ from DRAM through DramC can be read by the central processing unit, the video encoding device, the display processor and the neural network processor. When the first area moves down one block row, Sys$ and DRAM both store the new block row, and Sys$ removes the block row that is not used when encoding the next block row to be encoded, that is, the image data of the top block row of the first area is removed from Sys$. When the video encoding device needs to encode, it can directly read the data of the first area stored in Sys$. In addition, Sys$ also reads the image data of the first area from DRAM through DramC, and then is read by the video encoding device.

[0094] During storage, Sys$ or SysBuf outside the video encoding device will discard the block rows that will not be used when encoding the next row of block rows to be encoded, so that the number of times the cache or buffer inside the video encoding device reads the image data of the first area from DRAM can be changed from multiple times to 1 times. At the same time, since DRAM consumes 100 times more energy than SRAM for access, this can greatly reduce power consumption.

[0095] For example, since the position (i.e., the first area) and behavior (repeated reading) of the reconstructed frame image of the current frame image are predictable, and reading the reconstructed frame image of the current frame image will be multiple times the bandwidth required to read the current frame image. If the image data of the first area to be read multiple times is stored in a low-power storage space such as Sys$ or SysBuf, while effectively maintaining the operation of the video encoding device, the power consumption of the entire system can be greatly reduced, thereby improving the user experience. According to the structure of the reconstructed frame image of the current frame image compressed by the video encoding device, it can be determined how many related block rows of the reconstructed frame image of the current frame image to be stored in such a low-power storage space. Whenever the block row to be encoded moves down one row, the top block row of the first area stored in Sys$ or SysBuf is removed, and then the block rows newly added when moving down in the first area are re-read, and the image data of the newly added block rows are stored.

[0096] See also Fig.15 , Fig.15 It is a schematic diagram of the scene when the reconstructed frame image of the current frame image provided by the embodiment of the present application moves down one block row. Each time the video encoding device encodes a block row downward, it expels the irrelevant block rows originally stored above Sys$ or SysBuf, and then sends the image data of the newly required block rows into Sys$ or SysBuf. That is, when the first area covered by the search range can move down with the block to be encoded, the unused block rows are expelled from Sys$ or SysBuf, and the block rows to be used are stored in Sys$ or SysBuf. That is, each time the video encoding device encodes a block row downward, it removes the irrelevant block rows above the first area stored in Sys$ or SysBuf, and then stores the latest encoded block rows in Sys$ or SysBuf outside the video encoding device.

[0097] In another implementation, the first region includes a plurality of block rows, each block row includes a plurality of blocks, and storing the image data of the first region in the first memory and the second memory respectively in 204 may include:

[0098] If a block is added to the bottom block row of the first area, image data of the added block is stored in the first memory and the second memory respectively;

[0099] Blocks that are not used when encoding the next block to be encoded are removed from the first memory.

[0100] During storage, when a block is encoded, the image data of the encoded block is stored in Sys$ or SysBuf and DRAM outside the video encoding device. At the same time, Sys$ or SysBuf outside the video encoding device will discard the blocks that are not used when encoding the next block to be encoded, so the number of times the cache or buffer inside the video encoding device reads the image data of the first area from DRAM can be changed from multiple times to 1 times. At the same time, since DRAM consumes 100 times more energy than SRAM, this can greatly reduce power consumption. In this case, when encoding, the video encoding device reads data from Sys$ or SysBuf block by block without waiting for a block row to be encoded before reading, so this can improve the overall operating speed of the video encoding device.

[0101] 205. Read the image data of the first area from the first memory block by block for a first preset number of times, and read the image data of the unread blocks in the first area from the second memory block by block for a second preset number of times.

[0102] For example, when the video encoding device needs to encode, the image data of the first area can be read from the first memory block by block, such as reading the image data of the first area from Sys$ or SysBuf. When reading, the reading is performed block by block, that is, in a top-down order.

[0103] Taking H.265 / HEVC as an example, assuming Fig.15 The size of the Coding Tree Unit (CTU) in H.265 is 16×16 pixels, that is, 16 pixels horizontally multiplied by 16 pixels vertically. Of course, the size of the Coding Tree Unit can also be 32×32 pixels, 64×64 pixels, etc. The Coding Tree Unit is the processing unit of H.265. The Coding Tree Unit is the block to be encoded in the current frame image. This processing unit is similar to the Macroblock in Advanced Video Coding (H.264 / AVC). The vertical search range is -96, and the number of times the reconstructed frame image after the current frame image is encoded is 7 times the number of times the current frame image is read, that is, (96+16) / 16=7.

[0104] For example, when reading the image data of the first area, the number of times originally read from the second memory can be divided into a first preset number of times read from the first memory and a second preset number of times read from the second memory, and the sum of the first preset number of times and the second preset number of times is the number of times originally read from the second memory. It should be noted that the first memory and the second memory are read in no particular order, as long as the number of times read from the first memory reaches the first preset number of times and the number of times read from the second memory reaches the second preset number of times.

[0105] For example, the original number of times all data are read from the second memory is 7 times. When the first preset number is set to 6 times, the second preset number is set to 1 time. For example, under the condition of meeting the minimum demand of Sys$ or SysBuf, the 7 copies of data read from DRAM are split into 1 read from DRAM and 6 reads from Sys$ or SysBuf. The power consumption of reading data with the assistance of Sys$ or SysBuf is reduced to 14.96% of that without assistance, that is, (1×640+6×5) / (7×640)=14.96%, and the power consumption in this case is relatively low. For example, according to specific needs, the 7 copies of data read from DRAM can be split into 2 reads from DRAM and 5 reads from Sys$ or SysBuf, and so on. When the power consumption is strict, the 7 copies of data read from DRAM can also be read from Sys$ or SysBuf. At this time, the power consumption is the lowest, but the cost will increase.

[0106] Since the cost of SRAM is high and the cost of DRAM is low, SRAM is generally not made too large in consideration of cost, while DRAM can be made relatively large. Therefore, in order to reduce the power consumption when reading data, the embodiment of the present application can split the original number of times of reading from DRAM into several times of reading from SRAM and several times of reading from DRAM, which can reduce the power consumption of reading data as a whole. Moreover, the number of times of reading from SRAM and the number of times of reading from DRAM can be adjusted to meet the requirements of different power consumption.

[0107] For example, when reading the image data of the first area, it can be read from Sys$ or SysBuf first, and when the number of reads reaches a first preset number, it is switched to reading the image data of the unread block rows in the first area from DRAM. For another example, when reading the image data of the first area, it can be read from DRAM first, and when the number of reads reaches a second preset number, it is switched to reading the image data of the unread block rows in the first area from Sys$ or SysBuf. When reading the same image data, the energy consumed by DRAM is 100 times greater than the energy consumed by SRAM. Therefore, by reading part of the image data in the first area from Sys$ or SysBuf and the other part of the data from DRAM, the power consumption of reading data can be reduced.

[0108] See also Fig.16 , Fig.16 This is a schematic diagram of a power consumption curve when reading data from a multi-channel DRAM provided in an embodiment of the present application. Fig.16In the figure, the horizontal axis is the position of the reconstructed frame image of the current frame image, such as the top position of the image, the middle position of the image, and the bottom position of the image, and the vertical axis is the power consumption of reading and writing data during video encoding. In the case where the video encoding device relies too much on DRAM or other cheap but power-consuming storage and high bandwidth, the power consumption limit provided by the video compression system is limited, which may make the video encoding device unable to meet the speed requirements or cause the video compression system to overheat. If the power consumption limit is considered, the speed of reading data is limited and cannot reach the reading speed when the power consumption limit is not considered.

[0109] See also Fig.17 , Fig.17 The embodiment of the present application provides a power consumption curve diagram when reading and writing data from Sys$ or SysBuf and DRAM. The video encoding device replaces a large amount of DRAM power consumption with the power consumption of Sys$ or SysBuf, greatly reducing power consumption.

[0110] 206. Determine image data of a search window from the read image data of the first region, where the search window is located in the first region, and a left side of the block to be encoded is adjacent to a right side of a bottom block row of the search window.

[0111] For example, the search window in the first area is the searched range, and the matching block can be searched from the search window. In order to further narrow the search range, a search window can be determined from the first area, so that the search range can be narrowed, thereby further reducing power consumption. For the intra-screen block copy mode, when a non-hierarchical search is adopted, it is a reconstructed frame image of the non-reduced current frame image, and when a hierarchical search is adopted, it is a reconstructed frame image of the reduced or non-reduced current frame image. As long as the search window can predict the vertical position, it can be applied.

[0112] See also Fig.18 , Fig.18 Schematic diagram of the search range of the search window in the reconstructed frame image of the current frame image provided by the embodiment of the present application. Fig.18 It can be seen from the figure that the search window is located in the first area, the area between adjacent dotted lines in the first area is the block row, the left side of the block to be encoded is adjacent to the right side of the bottom block row of the search window, and the searched block vector can point to any place in the search window. Among them, L, R and H are the left search range, right search range and vertical search range (Vertical SRng) of the search window respectively, R is a positive number, L and H are negative numbers, and L is not necessarily equal to R.

[0113] 207. Store the image data of the search window in a third memory, wherein the read and write speed of the third memory is greater than a second preset multiple of the read and write speed of the first memory.

[0114] For example, after the image data of the search window is determined, the image data of the search window is stored in a third memory, which may be a memory inside the video encoding device, and the third memory may include a cache or buffer set inside the video encoding device. Since the blocks within the search window are searched when the block is copied within the picture, the bandwidth requirement is high when the block search is performed, so the read and write speeds of the third memory are greater than the read and write speeds of the first memory and the read and write speeds of the second memory. To meet the requirements of search speed and high bandwidth. Among them, the read and write speeds of the third memory are greater than the second preset multiple of the read and write speeds of the first memory.

[0115] 208. Read the image data of the search window from the third memory, and determine, based on the image data of the search window, a block having the smallest coding cost with the block to be coded from the search window.

[0116] For example, when searching, the image data of the search window is read from the third memory, and the search can be performed in a hierarchical or non-hierarchical manner. According to the image data of the search window, the blocks in the search window read are compared with the blocks to be encoded, and the blocks with the smallest coding cost between the blocks in the search window read and the blocks to be encoded can be determined. For example, in one embodiment, the coding cost may include a residual, for example, in another embodiment, the coding cost may include a block vector and a residual, and so on. It can be seen that the block with the smallest coding cost may be the block with the smallest residual with the block to be encoded, or the block with the smallest coding cost after comprehensively considering the block vector and the residual with the block to be encoded.

[0117] For example, for intra-screen block copying, the search window is scanned line by line, and the searched block is compared with the block to be encoded, so that the block with the smallest residual with the block to be encoded can be found from the search window. Among them, the block vector can be the relative displacement between the searched block and the block to be encoded. The residual can be the difference obtained by subtracting the two-dimensional pixel of the searched block from the two-dimensional pixel of the block to be encoded.

[0118] For example, in one implementation, reading the image data of the search window from the third memory in 208, and determining the block with the smallest coding cost with the block to be coded from the search window according to the image data of the search window, may include:

[0119] Reading the image data of the search window from the third memory;

[0120] Reducing the search window according to a preset number of levels to obtain a reduced search window;

[0121] According to the image data in the reduced search window, determining from the reduced search window a reduced block having the minimum coding cost with the block to be coded;

[0122] According to the position of the reduced block in the reduced search window, a block having the smallest coding cost with the block to be encoded is determined from the search window.

[0123] For example, when searching, you can use a hierarchical search method. Depending on the number of levels, the search level will be different. For example, if a 2-level search is used, a 2-level search is performed. If a 3-level search is used, a 3-level search is performed. Of course, the more levels there are, the more accurate the search results will be, but it will also increase the consumption of system computing resources. In actual applications, the appropriate number of levels can be set according to specific needs. It should be noted that the reduction ratio of each level can be the same or different.

[0124] For example, after reading the image data of the search window from the third memory, the search window is reduced according to a preset number of levels, such as reducing the search window according to 2 levels, to obtain a reduced search window, which is 1 / 2 of the original search window size. Then, based on the image data of the reduced search window, a reduced block with the smallest coding cost with the block to be encoded is determined from the reduced search window, and the reduction ratio of the reduced block is the same as that of the reduced search window. On the image of the reduced search window, the approximate range of the reduced block to be searched is first determined, and then the image of the unreduced search window is returned to perform a more detailed block search, that is, based on the approximate range of the reduced block in the reduced search window, a more detailed search is performed on the original search window, so that the block with the smallest coding cost with the block to be encoded can be determined from the unreduced search window.

[0125] For another example, after reading the image data of the search window from the third memory, the search window is reduced according to a preset number of levels, such as reducing the search window according to 3 levels, to obtain a reduced search window, which is 1 / 4 of the original search window size. Then, based on the image data of the search window with a reduced range of 1 / 4, the reduced block with the smallest coding cost for the block to be encoded is determined from the search window with a reduced range of 1 / 4, and the corresponding block vector under the search window with a reduced range of 1 / 4 is obtained. After that, a more refined search with a smaller range is performed in the search window with a reduced range of 1 / 2, and finally, the range of the search window of the original size is searched again according to the block vector obtained from the search window with a reduced range of 1 / 2 to obtain the final block vector, so that the block with the smallest coding cost for the block to be encoded can be determined.

[0126] For another example, after reading the image data of the search window from the third memory, the search window is reduced according to a preset number of levels, such as reducing the search window according to 3 levels, to obtain a reduced search window, which is 1 / 6 of the original search window size. Then, based on the image data of the search window with a reduced range of 1 / 6, the reduced block with the smallest coding cost for the block to be encoded is determined from the search window with a reduced range of 1 / 6, and the corresponding block vector under the search window with a reduced range of 1 / 6 is obtained. After that, a more refined search with a smaller range is performed in the search window with a reduced range of 1 / 3, and finally, the range of the search window of the original size is searched again based on the block vector obtained from the search window with a reduced range of 1 / 3 to obtain the final block vector, so that the block with the smallest coding cost for the block to be encoded can be determined.

[0127] From this, it can be seen that on the image of the reduced search window, first determine the approximate range of the reduced block to be searched, and then return to the image of the unreduced search window to perform a more detailed block search, that is, based on the approximate range of the reduced block in the reduced search window, perform a more detailed search on the original search window, and determine the block with the smallest encoding cost with the block to be encoded from the unreduced search window.

[0128] 209. The block with the smallest coding cost with the block to be coded is taken as a matching block.

[0129] For example, after a block with the smallest coding cost with the block to be coded is found in the search window, the block with the smallest coding cost with the block to be coded is used as a matching block.

[0130] 210. Encode the block to be encoded according to the block vectors and residuals between the matching block and the block to be encoded.

[0131] For example, the relative relationship between the matching block and the block to be encoded may be a block vector and a residual. After the matching block is found, the block to be encoded may be encoded according to the block vectors and the residual of the matching block and the block to be encoded.

[0132] In one implementation, encoding the block to be encoded according to the block vectors and the residuals of the matching block and the block to be encoded in 210 may include:

[0133] Performing forward transform and quantization (FTQ) on the residual between the matching block and the block to be encoded;

[0134] Performing entropy coding (EC) on the block vectors of the matching block and the block to be coded and the first residual data after forward transformation and quantization to obtain video stream coded data; or

[0135] Performing inverse quantization and transformation (De-Quantization & Inv. Transform, DQIT) on the first residual data after the forward transformation and quantization to obtain second residual data;

[0136] The block to be encoded is reconstructed according to the second residual data.

[0137] See also Fig.19 , Fig.19 Schematic diagram of a scene encoded by a video encoding device provided in an embodiment of the present application. Fig.19 It can be seen from the figure that the intra-frame block replication is located in the data flow relationship between the video encoding device and other modules. The intra-frame block replication will compare the similarity between the block to be encoded in the current frame image and the block of the reconstructed frame image of the current frame image.

[0138] For example, see Fig.19 , the block copy in the picture (either hierarchical search or non-hierarchical search can be used) searches for the reconstructed frame image of the current frame image, and a matching block will be found. The relative displacement between the matching block and the current block (i.e., the block to be encoded) is the block vector, and the residual is obtained according to the error between the current block and the matching block. The residual is forward transformed and quantized, wherein the forward transformation uses Fast Fourier Transformation (FFT) to obtain the spectrum. The horizontal axis on the spectrum curve is the frequency, and the vertical axis is the energy. After the forward transformation, the pixels in the space are converted into unrelated and energy-concentrated spectrum coefficients. The forward transformed data is only converted to the frequency domain, and the amount of data does not change, which can reduce distortion. The matrix after the forward transformation is divided by the value of the corresponding position in the quantization matrix to achieve quantization. The spectrum coefficients are further compressed by quantization and entropy coding to obtain a compressed video stream. When entropy coding is performed, entropy coding is performed on the block vector and the first residual data after quantization. Among them, the quantization process removes some unimportant high-frequency information, which can compress the amount of image data, so quantization is the key to compression. After forward transformation and quantization, the first residual data is obtained.

[0139] The first residual data obtained after forward transformation and quantization is inversely quantized and transformed to the spatial domain, that is, the second residual data of the matching block and the block to be encoded is obtained, and the block to be encoded of the current frame image is reconstructed through the picture block area (BlockReconstruction, BlkRec) as the neighbor of the next block to be encoded. The in-loop filter (In-loop Filter, InF) is used to deal with the continuity problem between blocks to make the image smoother. The commonly used loop filter is a linear low-pass filter that can filter out high-frequency components and noise. Forward transformation and quantization can eliminate the spatial redundancy of video images, and entropy coding can eliminate coding redundancy.

[0140] It can be understood that the embodiment of the present application is based on the predictable data access behavior (i.e., the behavior of repeated reading) when encoding the screen content, so as to achieve an efficient selection of data storage methods, which can reduce the power consumption of the video encoding device. According to the frame reference reading strategy during encoding, it can be changed whether the data to be read (such as the first filtered area in the reconstructed frame image of the current frame image) should be first stored in the low-power Sys$ or SysBuf, so that the number of repeated readings of the reconstructed frame image of part or all of the current frame image stored in Sys$ or SysBuf is the highest, so as to minimize power consumption and ensure that the video encoding device can always maintain the lowest power consumption state when entering and exiting data. If the Sys$ or SysBuf also has a high-speed bandwidth, since the Sys$ or SysBuf can meet the bandwidth required for repeated reading of data, the bandwidth of the DRAM can be further reduced.

[0141] The embodiments of the present application can ensure that the processing power consumption of the video encoding device is controllable, and can enable the hardware or software of the video encoding device to complete the encoding work as soon as possible, and make full use of the predictable behavior of the video encoding device to repeatedly read the image data of the first area to change the storage characteristics of the read data, because accessing data saves power, and the video encoding device can maintain the operating speed while reducing power consumption. The speed of reading data will not be limited by power consumption, so the video encoding device will not overheat and will not trigger frequency reduction. In addition, the delay of the SRAM in Sys$ or SysBuf itself is low when reading and writing, which can increase the processing frame rate and reduce the response delay. Since the power consumption can be greatly reduced, the battery life of the video encoding device can be increased, and the user experience can be improved.

[0142] It is understandable that the embodiments of the present application can select the target location or attribute of data reading according to the screen content related to the video such as screen sharing and recording requirements, as well as the low heat dissipation cost requirements and the large power consumption caused by predictable behavior. For example, the data that needs to be read repeatedly is read from Sys$ or SysBuf, and DRAM respectively, instead of all from DRAM. Since the power consumption of SRAM is much less than that of DRAM when reading the same data, the embodiments of the present application can greatly reduce the power consumption when reading data.

[0143] The embodiment of the present application uses screen content encoding as an example to explain in detail how to reduce the power consumption of reading data. In other embodiments, it can also be applied to all modules and applications that require high bandwidth but have predictable data access behaviors, such as video decoders, frame rate up conversion devices, etc. The behaviors of these modules and applications are usually predictable, such as the number of repeated reads. Through these predictable behaviors, the corresponding storage characteristics can be pre-allocated, that is, the repeatedly read data is stored in a low-power memory, for example, according to the number of access requirements of the image data of all frames or part of the frames, to correspond to the energy consumption of different levels of storage targets, that is, according to the number of access requirements of the image data of all frames or part of the frames, to select the corresponding energy consumption of different levels. When the energy consumption is different, the number of times to read data from Sys$ or SysBuf and DRAM can be reasonably allocated. For example, the video decoder can also determine the behavior of accessing data by parsing the code stream in advance, and the frame rate up conversion device can know which areas will be used multiple times during processing through simple analysis, etc. Therefore, the behavior of the video decoder and the frame rate up conversion device is predictable.

[0144] The embodiments of the present application may also be applicable to fixed artificial intelligence (AI) network behaviors, where the portion of the AI ​​network behaviors that is repeatedly read is a feature map portion, and the AI ​​network behaviors are predictable.

[0145] See also Fig. 20 , Fig. 20 The schematic diagram of the structure of the apparatus for performing image processing in a video encoding apparatus provided in an embodiment of the present application. The apparatus 400 for performing image processing in a video encoding apparatus may include: a first determination module 401, a second determination module 402, a reading module 403, a third determination module 404, and an encoding module 405.

[0146] A first determining module 401 is used to determine a block to be encoded from a current frame image;

[0147] A second determination module 402 is used to determine a first area that needs to be repeatedly read from the reconstructed frame image of the current frame image, and store the image data of the first area in a preset memory, wherein the power consumption of the preset memory is less than a preset power consumption threshold;

[0148] A reading module 403, configured to read the image data of the first area from the preset memory;

[0149] A third determination module 404 is used to determine a matching block that matches the block to be encoded from the first area according to the image data of the first area;

[0150] The encoding module 405 is configured to encode the block to be encoded according to the relative relationship between the matching block and the block to be encoded.

[0151] In one implementation, the first determining module 401 may also be used to:

[0152] Determining a block row to be encoded from the current frame image;

[0153] The block to be encoded is determined from the row of blocks to be encoded.

[0154] In one implementation, the preset memory includes a first memory and a second memory, the power consumption of the second memory is greater than a first preset multiple of the power consumption of the first memory, and the second determining module 402 may be used to:

[0155] Determine a first area that needs to be repeatedly read from a reconstructed frame image of the current frame image;

[0156] The image data of the first area is stored in the first memory and the second memory respectively.

[0157] In one implementation, the first area includes a plurality of block rows, and the second determining module 402 may be configured to:

[0158] If the first area is shifted down by one block line, the image data of the shifted block line is stored in the first memory and the second memory respectively;

[0159] Block rows that are not used when encoding the next block row to be encoded are removed from the first memory.

[0160] In one implementation, the first area includes a plurality of block rows, each of the block rows includes a plurality of blocks, and the second determining module 402 may be configured to:

[0161] If a block is added to the bottom block row of the first area, image data of the added block is stored in the first memory and the second memory respectively;

[0162] Blocks that are not used when encoding the next block to be encoded are removed from the first memory.

[0163] In one implementation, the reading module 403 may be used to:

[0164] The image data of the first area is read from the first memory block by block for a first preset number of times, and the image data of unread block lines in the first area is read from the second memory block by block for a second preset number of times.

[0165] In one implementation, the third determining module 404 may be configured to:

[0166] Determine image data of a search window from the read image data of the first area, the search window is located in the first area, and the left side of the block to be encoded is adjacent to the right side of the bottom block row of the search window;

[0167] storing the image data of the search window in a third memory, wherein the reading and writing speed of the third memory is greater than a second preset multiple of the reading and writing speed of the first memory;

[0168] Reading the image data of the search window from the third memory, and determining a block having the smallest coding cost with the block to be coded from the search window according to the image data of the search window;

[0169] The block with the smallest coding cost with the block to be coded is used as the matching block.

[0170] In one implementation, the third determining module 404 may be configured to:

[0171] Reading the image data of the search window from the third memory;

[0172] Reducing the search window according to a preset number of levels to obtain a reduced search window;

[0173] According to the image data in the reduced search window, determining from the reduced search window a reduced block having the minimum coding cost with the block to be coded;

[0174] According to the position of the reduced block in the reduced search window, a block having the smallest coding cost with the block to be encoded is determined from the search window.

[0175] In one implementation, the relative relationship is a block vector and a residual, the block vector is a relative displacement between the matching block and the block to be encoded, and the encoding module 405 may be used to:

[0176] The block to be encoded is encoded according to the block vectors and the residuals between the matching block and the block to be encoded.

[0177] In one implementation, the encoding module 405 may be used to:

[0178] Performing forward transformation and quantization on the residual between the matching block and the block to be encoded;

[0179] Performing entropy coding on the block vectors of the matching block and the block to be coded and the first residual data after forward transformation and quantization to obtain video stream coded data; or

[0180] Inversely quantize and transform the first residual data after forward transformation and quantization to obtain second residual data;

[0181] The block to be encoded is reconstructed according to the second residual data.

[0182] In one embodiment, the first memory includes a system cache or a system buffer memory disposed outside the video encoding device, and the second memory includes a dynamic random access memory disposed outside the video encoding device.

[0183] In one implementation, the third memory includes a cache or buffer disposed inside the video encoding device.

[0184] An embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed on a computer, the computer executes the process in the method for performing image processing in a video encoding device as provided in the present embodiment.

[0185] An embodiment of the present application also provides an electronic device, including a memory, a processor and a video encoding device, wherein the processor is used to execute the process of the method for performing image processing in the video encoding device provided in this embodiment by calling a computer program stored in the memory.

[0186] For example, the electronic device may be a mobile terminal such as a tablet computer or a smart phone. Fig.21 , Fig.21 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0187] The electronic device 500 may include a video encoding device 501, a memory 502, a processor 503 and other components. Those skilled in the art will appreciate that Fig.21 The electronic device structure shown in the figure does not constitute a limitation of the electronic device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0188] The video encoding device 501 can be used to encode screen content to compress the screen content of the video image.

[0189] The memory 502 may be used to store applications and data. The applications stored in the memory 502 include executable codes. The applications may be composed of various functional modules. The processor 503 executes various functional applications and data processing by running the applications stored in the memory 502.

[0190] The processor 503 is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device. It executes various functions of the electronic device and processes data by running or executing applications stored in the memory 502 and calling data stored in the memory 502, thereby monitoring the electronic device as a whole.

[0191] In this embodiment, the processor 503 in the electronic device will load the executable code corresponding to the process of one or more application programs into the memory 502 according to the following instructions, and the processor 503 will run the application program stored in the memory 502 to execute:

[0192] Determine a block to be encoded from the current frame image;

[0193] Determine a first area that needs to be repeatedly read from a reconstructed frame image of the current frame image, and store image data of the first area in a preset memory, wherein power consumption of the preset memory is less than a preset power consumption threshold;

[0194] Reading the image data of the first area from the preset memory;

[0195] Determining a matching block that matches the block to be encoded from the first area according to the image data of the first area;

[0196] The block to be encoded is encoded according to the relative relationship between the matching block and the block to be encoded.

[0197] See also Fig. 22 The electronic device 500 may include a video encoding device 501, a memory 502, a processor 503, a battery 504, an input unit 505, an output unit 506 and other components.

[0198] The video encoding device 501 can be used to encode screen content to compress the screen content of the video image.

[0199] The memory 502 may be used to store applications and data. The applications stored in the memory 502 include executable codes. The applications may be composed of various functional modules. The processor 503 executes various functional applications and data processing by running the applications stored in the memory 502.

[0200] The processor 503 is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device. It executes various functions of the electronic device and processes data by running or executing applications stored in the memory 502 and calling data stored in the memory 502, thereby monitoring the electronic device as a whole.

[0201] The battery 504 may be used to provide power support for various components of the electronic device, thereby ensuring the normal operation of each component.

[0202] The input unit 505 may be used to receive an input video stream of a video image, for example, may be used to receive a video stream that needs to be compressed.

[0203] The output unit 506 may be configured to output the compressed video stream.

[0204] In this embodiment, the processor 503 in the electronic device will load the executable code corresponding to the process of one or more application programs into the memory 502 according to the following instructions, and the processor 503 will run the application program stored in the memory 502 to execute:

[0205] Determine a block to be encoded from the current frame image;

[0206] Determine a first area that needs to be repeatedly read from a reconstructed frame image of the current frame image, and store image data of the first area in a preset memory, wherein power consumption of the preset memory is less than a preset power consumption threshold;

[0207] Reading the image data of the first area from the preset memory;

[0208] Determining a matching block that matches the block to be encoded from the first area according to the image data of the first area;

[0209] The block to be encoded is encoded according to the relative relationship between the matching block and the block to be encoded.

[0210] In one implementation, when the processor 503 executes the step of determining the block to be encoded from the current frame image, it may also execute: determining a row of blocks to be encoded from the current frame image; and determining the block to be encoded from the row of blocks to be encoded.

[0211] In one embodiment, the preset memory includes a first memory and a second memory, the power consumption of the second memory is greater than a first preset multiple of the power consumption of the first memory, and when the processor 503 executes the steps of determining a first area that needs to be repeatedly read from the reconstructed frame image of the current frame image and storing the image data of the first area in the preset memory, it can also execute: determining a first area that needs to be repeatedly read from the reconstructed frame image of the current frame image; and storing the image data of the first area in the first memory and the second memory, respectively.

[0212] In one embodiment, the first area includes multiple block rows. When the processor 503 executes the step of storing the image data of the first area in the first memory and the second memory respectively, it can also execute: if the first area is moved down by one block row, the image data of the moved down block row is stored in the first memory and the second memory respectively; and the block rows that are not used when encoding the next block row to be encoded are removed from the first memory.

[0213] In one embodiment, the first area includes multiple block rows, each of which includes multiple blocks. When the processor 503 executes the step of storing the image data of the first area in the first memory and the second memory respectively, it can also execute: if a block is added to the bottom block row of the first area, the image data of the newly added block is stored in the first memory and the second memory respectively; and blocks that are not used when encoding the next block to be encoded are removed from the first memory.

[0214] In one embodiment, when the processor 503 executes the reading of the image data of the first area from the preset memory, it can also execute: reading the image data of the first area from the first memory block by block for a first preset number of times, and reading the image data of the unread block rows in the first area from the second memory block by block for a second preset number of times.

[0215] In one embodiment, when the processor 503 executes the step of determining a matching block that matches the block to be encoded from the first area based on the image data of the first area, it may also execute: determining image data of a search window from the read image data of the first area, the search window being located within the first area, and the left side of the block to be encoded being adjacent to the right side of the bottom block row of the search window; storing the image data of the search window in a third memory, the read and write speed of the third memory being greater than a second preset multiple of the read and write speed of the first memory; reading the image data of the search window from the third memory, and determining a block with the smallest encoding cost with the block to be encoded from the search window based on the image data of the search window; and using the block with the smallest encoding cost with the block to be encoded as the matching block.

[0216] In one embodiment, when the processor 503 executes the reading of the image data of the search window from the third memory and determines, based on the image data of the search window, a block with the smallest coding cost relative to the block to be encoded from the search window, it may also execute: reading the image data of the search window from the third memory; reducing the search window according to a preset number of levels to obtain a reduced search window; determining, based on the image data of the reduced search window, a reduced block with the smallest coding cost relative to the block to be encoded from the reduced search window; and determining, based on the position of the reduced block in the reduced search window, a block with the smallest coding cost relative to the block to be encoded from the search window.

[0217] In one implementation, the relative relationship is a block vector and a residual, the block vector is a relative displacement between the matching block and the block to be encoded, and when the processor 503 executes the encoding of the block to be encoded according to the relative relationship between the matching block and the block to be encoded, it may also execute: encoding the block to be encoded according to the block vector and the residual between the matching block and the block to be encoded.

[0218] In one embodiment, when the processor 503 executes the encoding of the block to be encoded based on the block vectors and residuals of the matching block and the block to be encoded, it can also execute: forward transforming and quantizing the residuals of the matching block and the block to be encoded; entropy encoding the block vectors of the matching block and the block to be encoded and the first residual data after forward transformation and quantization to obtain video stream encoded data; or inverse quantizing and transforming the first residual data after forward transformation and quantization to obtain second residual data; and reconstructing the block to be encoded based on the second residual data.

[0219] In one embodiment, the first memory includes a system cache or a system buffer memory disposed outside the video encoding device, and the second memory includes a dynamic random access memory disposed outside the video encoding device.

[0220] In one implementation, the third memory includes a cache or buffer disposed inside the video encoding device.

[0221] The present application also provides an image processing system. Fig.23 and Fig.24 , Fig.23 It is a structural diagram of the image processing system provided in an embodiment of the present application. Fig.246 is another structural diagram of an image processing system provided by an embodiment of the present application. The image processing system 600 includes a video encoding device 601, a first memory 602, and a second memory 603, wherein the power consumption of the second memory 603 is greater than the first preset multiple of the power consumption of the first memory 602, the video encoding device 601 may include a third memory, the reading speed of the third memory is greater than the second preset multiple of the reading speed of the first memory, the first memory 602 and the second memory 603 respectively store image data that need to be repeatedly read in the reconstructed frame image of the current frame image, and the video encoding device 601 reads the image data that need to be repeatedly read from the first memory 602 according to the first preset number of times and from the second memory 603 according to the second preset number of times when encoding, that is, when reading the image data that need to be repeatedly read, it can be divided into reading the first preset number of times from the first memory 602 and reading the second preset number of times from the second memory 603. Then, the image data in the search window can be determined from the image data that need to be repeatedly read, and the image data in the search window can be stored in the third memory.

[0222] For example, after determining the first region that needs to be repeatedly read multiple times from the reconstructed frame image of the current frame image, the image data of the first region that needs to be repeatedly read multiple times determined from the reconstructed frame image of the current frame image can be stored in the first memory 602 and the second memory 603, respectively. Therefore, the first memory 602 and the second memory 603 store the image data of the first region that needs to be repeatedly read multiple times determined from the reconstructed frame image of the current frame image, respectively.

[0223] During the encoding process, if the first area is moved down by one block row, the image data of the moved down block row will be stored in the first memory 602 and the second memory 603 respectively, and the block rows that are not used when encoding the next block row to be encoded will be removed from the first memory 602. When reading data, the image data of the first area can be read block by block from the first memory 602 for a first preset number of times, and the image data of the unread block rows in the first area can be read block by block from the second memory 603 for a second preset number of times.

[0224] It should be noted that when reading image data from the second memory 603, the video encoding device 601 can read the image data directly from the second memory 603, or the first memory 602 reads the image data from the second memory 603 and then stores it, and the video encoding device 601 reads the image data directly from the first memory 602.

[0225] The video encoding device 601 can read the image data in the search window from the third memory, determine the matching block that matches the module to be encoded from the search window based on the image data in the search window read from the third memory, and encode according to the block vector and residual of the matching block and the block to be encoded.

[0226] In the above embodiments, the description of each embodiment has its own emphasis. For the part that is not described in detail in a certain embodiment, please refer to the detailed description of the method for performing image processing in the video encoding device above, which will not be repeated here.

[0227] The device for performing image processing in a video encoding device provided in the embodiment of the present application belongs to the same concept as the method for performing image processing in a video encoding device in the above embodiment. Any method provided in the embodiment of the method for performing image processing in a video encoding device can be run on the device for performing image processing in a video encoding device. The specific implementation process is detailed in the embodiment of the method for performing image processing in a video encoding device, which will not be repeated here.

[0228] It should be noted that, for the method of performing image processing in a video encoding device described in the embodiment of the present application, a person of ordinary skill in the art can understand that the implementation of all or part of the process of the method of performing image processing in a video encoding device described in the embodiment of the present application can be completed by controlling the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, such as a memory, and executed by at least one processor, and the execution process may include the process of the embodiment of the method of performing image processing in a video encoding device. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), etc.

[0229] For the device for performing image processing in the video encoding device of the embodiment of the present application, each functional module can be integrated into a processing chip, or each module can exist physically separately, or two or more modules can be integrated into one module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium, such as a read-only memory, a disk or an optical disk.

[0230] The above is a detailed introduction to a method, device, storage medium, electronic device and system for performing image processing in a video encoding device provided by an embodiment of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A method for performing image processing in a video encoding device, characterized in that: The method comprises: Determine a block to be encoded from the current frame image; Determine a first area that needs to be repeatedly read multiple times from the reconstructed frame image of the current frame image, and store image data of the first area in a first memory and a second memory respectively, wherein power consumption of the second memory is greater than a first preset multiple of power consumption of the first memory, and the sum of power consumption of the first memory and power consumption of the second memory is less than a preset power consumption threshold; Reading image data of the first area from the first memory and the second memory; Determining a matching block that matches the block to be encoded from the first area according to the image data of the first area; The block to be encoded is encoded according to the relative relationship between the matching block and the block to be encoded.

2. The method for performing image processing in a video encoding device according to claim 1, characterized in that: The step of determining a block to be encoded from the current frame image includes: Determining a block row to be encoded from the current frame image; The block to be encoded is determined from the row of blocks to be encoded.

3. The method for performing image processing in a video encoding device according to claim 1, characterized in that: The first area includes a plurality of block rows, and the storing the image data of the first area in the first memory and the second memory respectively includes: If the first area is shifted down by one block line, the image data of the shifted block line is stored in the first memory and the second memory respectively; Block rows that are not used when encoding the next block row to be encoded are removed from the first memory.

4. The method for performing image processing in a video encoding device according to claim 1, characterized in that: The first area includes a plurality of block rows, each of the block rows includes a plurality of blocks, and the storing of the image data of the first area in the first memory and the second memory respectively includes: If a block is added to the bottom block row of the first area, image data of the added block is stored in the first memory and the second memory respectively; Blocks that are not used when encoding the next block to be encoded are removed from the first memory.

5. The method for performing image processing in a video encoding device according to claim 3 or 4, characterized in that: The step of reading the image data of the first area from the first memory and the second memory includes: The image data of the first area is read from the first memory block by block for a first preset number of times, and the image data of unread block lines in the first area is read from the second memory block by block for a second preset number of times.

6. The method for performing image processing in a video encoding device according to claim 5, characterized in that: The determining, from the first region according to the image data of the first region, a matching block that matches the block to be encoded comprises: Determine image data of a search window from the read image data of the first area, the search window is located in the first area, and the left side of the block to be encoded is adjacent to the right side of the bottom block row of the search window; storing the image data of the search window in a third memory, wherein the reading and writing speed of the third memory is greater than a second preset multiple of the reading and writing speed of the first memory; Reading the image data of the search window from the third memory, and determining a block having the smallest coding cost with the block to be coded from the search window according to the image data of the search window; The block with the smallest coding cost with the block to be coded is used as the matching block.

7. The method for performing image processing in a video encoding device according to claim 6, characterized in that: The step of reading the image data of the search window from the third memory and determining, based on the image data of the search window, a block having the smallest coding cost with the block to be coded from the search window comprises: Reading the image data of the search window from the third memory; Reducing the search window according to a preset number of levels to obtain a reduced search window; According to the image data in the reduced search window, determining from the reduced search window a reduced block having the minimum coding cost with the block to be coded; According to the position of the reduced block in the reduced search window, a block having the smallest coding cost with the block to be encoded is determined from the search window.

8. The method for performing image processing in a video encoding device according to claim 6, characterized in that: The relative relationship is a block vector and a residual, the block vector is a relative displacement between the matching block and the block to be encoded, and encoding the block to be encoded according to the relative relationship between the matching block and the block to be encoded includes: The block to be encoded is encoded according to the block vectors and the residuals between the matching block and the block to be encoded.

9. The method for performing image processing in a video encoding device according to claim 8, characterized in that: The step of encoding the block to be encoded according to the block vector and the residual between the matching block and the block to be encoded comprises: Performing forward transformation and quantization on the residual between the matching block and the block to be encoded; Performing entropy coding on the block vectors of the matching block and the block to be coded and the first residual data after forward transformation and quantization to obtain video stream coded data; or Inversely quantize and transform the first residual data after forward transformation and quantization to obtain second residual data; The block to be encoded is reconstructed according to the second residual data.

10. The method for performing image processing in a video encoding device according to claim 1, characterized in that: The first memory includes a system cache or a system buffer memory arranged outside the video encoding device, and the second memory includes a dynamic random access memory arranged outside the video encoding device.

11. The method for performing image processing in a video encoding device according to claim 6, characterized in that: The third memory includes a cache or buffer disposed inside the video encoding device.

12. A device for performing image processing in a video encoding device, characterized in that: The device comprises: A first determination module, used to determine a block to be encoded from a current frame image; a second determining module, configured to determine a first area that needs to be repeatedly read from the reconstructed frame image of the current frame image, and store image data of the first area in a first memory and a second memory respectively, wherein power consumption of the second memory is greater than a first preset multiple of power consumption of the first memory, and a sum of power consumption of the first memory and power consumption of the second memory is less than a preset power consumption threshold; A reading module, used for reading the image data of the first area from the first memory and the second memory; A third determination module, configured to determine a matching block matching the block to be encoded from the first area according to the image data of the first area; The encoding module is used to encode the block to be encoded according to the relative relationship between the matching block and the block to be encoded.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed on a computer, the computer is caused to execute the method according to any one of claims 1 to 11.

14. An electronic device comprising a memory, a processor and a video encoding device, characterized in that: The processor executes the method according to any one of claims 1 to 11 by calling the computer program stored in the memory.

15. An image processing system, characterized in that: The invention comprises a video encoding device, a first memory and a second memory, wherein the power consumption of the second memory is greater than a first preset multiple of the power consumption of the first memory, the video encoding device comprises a third memory, the reading speed of the third memory is greater than a second preset multiple of the reading speed of the first memory, the first memory and the second memory respectively store image data that need to be repeatedly read in a reconstructed frame image of a current frame image, when encoding, the video encoding device reads the image data that need to be repeatedly read from the first memory according to a first preset number of times and from the second memory according to a second preset number of times, and determines the image data within a search window from the image data that need to be repeatedly read, and stores the image data within the search window in the third memory, the video encoding device reads the image data within the search window from the third memory, determines a matching block that matches a block to be encoded, and encodes according to a block vector and a residual between the matching block and the block to be encoded.

16. The image processing system according to claim 15, characterized in that: The first memory and the second memory respectively store image data of a first area that needs to be repeatedly read multiple times, which is determined from the reconstructed frame image of the current frame image. If the first area is moved down by one block row, the image data of the moved down block row is stored in the first memory and the second memory respectively, and the block rows that are not used when encoding the next block row to be encoded are removed from the first memory. When reading, the video encoding device reads the image data of the first area from the first memory block by block for a first preset number of times, and reads the image data of the unread block rows in the first area from the second memory block by block for a second preset number of times.

Images