Method, device and system for image processing in video encoding device

Abstract

The present application discloses a method, device, storage medium, electronic device and system for image processing in a video encoding device. The method includes: determining a block to be encoded from a current frame image; determining a first area that needs to be read repeatedly from a reconstructed frame image of a historical frame image, and storing the image data of the first area in a preset memory, wherein the power consumption of the preset memory is less than a preset power consumption threshold; reading the image data of the first area from the preset memory; determining a matching block that matches the block to be encoded from the first area based on the read image data of the first area; encoding the block to be encoded based on the relative relationship between the matching block and the block to be encoded. The present application can reduce the power consumption of a video encoding device.

Description

Technical Field

[0001] The present application belongs to the technical field of electronic equipment, and in particular, relates to a method, device, storage medium, electronic equipment and system for performing image processing in a video encoding device. Background Art

[0002] With the continuous development of technology, the functions of video encoding devices are becoming more and more powerful. Video encoding devices can encode video images. When encoding a frame of video image, it is usually necessary to read the amount of data of multiple frames of encoded video images. However, in the related art, when reading the data of the encoded video image, the power consumption of the video encoding device is relatively large. Summary of the invention

[0003] Embodiments of the present application provide a method, device, storage medium, electronic device, and system for performing image processing in a video encoding device, which can reduce the power consumption of the video encoding device.

[0004] In a first aspect, an embodiment of the present application provides a method for performing image processing in a video encoding device, the method comprising:

[0005] Determine the block to be encoded from the current frame image;

[0006] Determine a first area that needs to be repeatedly read from a reconstructed frame image of a historical frame image, and store image data of the first area in a preset memory, wherein power consumption of the preset memory is less than a preset power consumption threshold;

[0007] Reading the image data of the first area from the preset memory;

[0008] Determine a matching block matching the block to be encoded from the first area according to the image data of the first area read;

[0009] The block to be encoded is encoded according to the relative relationship between the matching block and the block to be encoded.

[0010] In a second aspect, an embodiment of the present application provides a device for performing image processing in a video encoding device, the device comprising:

[0011] A first determination module, used to determine a block to be encoded from a current frame image;

[0012] A second determination module is used to determine a first area that needs to be repeatedly read from the reconstructed frame image of the historical frame image, and store the image data of the first area in a preset memory, wherein the power consumption of the preset memory is less than a preset power consumption threshold;

[0013] A reading module, used for reading the image data of the first area from the preset memory;

[0014] A third determination module, configured to determine a matching block matching the block to be encoded from the first area according to the image data of the first area read;

[0015] The encoding module is used to encode the block to be encoded according to the relative relationship between the matching block and the block to be encoded.

[0016] In a third aspect, an embodiment of the present application provides a storage medium on which a computer program is stored. When the computer program is executed on a computer, the computer executes the method for performing image processing in a video encoding device provided in an embodiment of the present application.

[0017] In a fourth aspect, an embodiment of the present application further provides an electronic device, comprising a memory, a processor and a video encoding device, wherein the processor is used to execute the method for performing image processing in the video encoding device provided in an embodiment of the present application by calling a computer program stored in the memory.

[0018] In the fifth aspect, an embodiment of the present application also provides an image processing system, including a video encoding device, a first memory and a second memory, the power consumption of the second memory is greater than a first preset multiple of the power consumption of the first memory, the video encoding device includes a third memory, the reading speed of the third memory is greater than a second preset multiple of the reading speed of the first memory, the first memory and the second memory respectively store image data repeatedly read from reconstructed frame images of historical frame images, when encoding, the video encoding device reads the repeatedly read image data from the first memory and the second memory respectively according to a preset number of times, and determines the image data within a search window (SearchWindow, SWin) therefrom, and stores the image data within the search window in the third memory, the video encoding device reads the image data within the search window from the third memory, and determines a matching block that matches the block to be encoded, and encodes according to the motion vector and residual of the matching block and the block to be encoded.

[0019] In an embodiment of the present application, the video encoding device can determine the block to be encoded from the current frame image, determine the first area that needs to be read repeatedly from the reconstructed frame image of the historical frame image, and store the image data of the first area in a preset memory, the power consumption of the preset memory is less than the preset power consumption threshold. Then, the image data of the first area is read from the preset memory, and a matching block that matches the block to be encoded is determined from the first area based on the read image data of the first area. Afterwards, the block to be encoded is encoded based on the relative relationship between the matching block and the block to be encoded. That is, in an embodiment of the present application, the image data of the first area is stored in a preset memory with lower power consumption, so as to achieve the purpose of reducing the power consumption of the video encoding device. Therefore, the embodiment of the present application can reduce the power consumption of the video encoding device. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The technical solution and beneficial effects of the present application will be made apparent by describing in detail the specific implementation modes of the present application in conjunction with the accompanying drawings.

[0021] Figure 1 This is a first flow chart of a method for performing image processing in a video encoding device provided in an embodiment of the present application.

[0022] Figure 2 It is a structural diagram of a video compression system in the related art.

[0023] Figure 3 It is a schematic diagram of data storage in a video encoding device in the related art.

[0024] Figure 4 It is a schematic diagram of increasing the number of channels of dynamic random access memory (DRAM) for data access in the related art.

[0025] Figure 5 It is a comparison diagram of the square blocks provided in the embodiment of the present application.

[0026] Figure 6 It is a schematic diagram of the hierarchical search provided in the embodiment of the present application.

[0027] Figure 7 It is a schematic diagram of a non-hierarchical search provided in an embodiment of the present application.

[0028] Figure 8 This is a second flow chart of the method for performing image processing in a video encoding device provided in an embodiment of the present application.

[0029] Fig. 9 It is a schematic diagram of a scene for searching in a reconstructed frame image of a historical frame image provided in an embodiment of the present application.

[0030] Fig.10 It is a schematic diagram comparing the energy consumed by a static random access memory (SRAM) and a dynamic random access memory when reading data provided by an embodiment of the present application.

[0031] Fig.11 It is a schematic diagram of the architecture of a video compression system using a system cache (Sys$) provided in an embodiment of the present application.

[0032] Fig.12 This is another architectural diagram of a video compression system using a system cache provided in an embodiment of the present application.

[0033] Fig.13 It is a schematic diagram of the architecture of a video compression system using a system buffer memory (System Buffer, SysBuf) provided in an embodiment of the present application.

[0034] Fig.14 It is a schematic diagram of a scene when the reconstructed frame image of the historical frame image provided by an embodiment of the present application moves down by one block row.

[0035] Fig.15 It is a schematic diagram of a power consumption curve when reading and writing data from a multi-channel DRAM provided in an embodiment of the present application.

[0036] Fig.16 It is a schematic diagram of power consumption curves when reading and writing data from Sys$ or SysBuf and DRAM respectively, provided in an embodiment of the present application.

[0037] Fig.17 A schematic diagram of a scene showing the search range of a search window in a reconstructed frame image of a historical frame image provided by an embodiment of the present application.

[0038] Fig.18 It is a schematic diagram of a scene encoded by a video encoding device provided in an embodiment of the present application.

[0039] Fig.19 This is a third flow chart of the method for performing image processing in a video encoding device provided in an embodiment of the present application.

[0040] Fig. 20 It is a structural schematic diagram of a device for performing image processing in a video encoding device provided in an embodiment of the present application.

[0041] Fig.21 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0042] Fig. 22It is another structural schematic diagram of the electronic device provided in an embodiment of the present application.

[0043] Fig.23 It is a structural diagram of the image processing system provided in an embodiment of the present application.

[0044] Fig.24 It is another structural schematic diagram of the image processing system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0045] Please refer to the drawings, where the same component symbols represent the same components. The principle of the present application is illustrated by implementing it in an appropriate computing environment. The following description is based on the illustrated specific embodiments of the present application, which should not be considered as limiting other specific embodiments of the present application that are not described in detail herein.

[0046] See also Figure 1 , Figure 1 1 is a schematic diagram of a first flow chart of a method for performing image processing in a video encoding device provided by an embodiment of the present application. The method for performing image processing in a video encoding device can be applied to a video encoding device. The flow chart of the method for performing image processing in a video encoding device may include:

[0047] 101. Determine a block to be encoded from the current frame image.

[0048] With the continuous development of technology, the functions of video encoding devices are becoming more and more powerful. Video encoding devices can encode video images. When encoding a frame of video image, it is usually necessary to read the amount of data of multiple frames of encoded video images. However, in the related art, when reading the data of the encoded video image, the power consumption of the video encoding device is relatively large.

[0049] See also Figure 2 , Figure 2The figure is a schematic diagram of the structure of the video compression system in the related art. In the video compression system, the central processing unit (CPU), video encoding device, image signal processor (ISP) and neural network processing unit (NPU) read and write data from DRAM through the bus and dynamic random access memory controller (DRAMC). The central processing unit, video encoding device, image processor and neural network processor share the bandwidth in a time-sharing manner. The priority of the central processing unit, image processor and neural network processor is higher than that of the video encoding device. The video encoding device needs to perform a search action when encoding, which will occupy a large bandwidth.

[0050] Video encoding devices attach great importance to cost. In order to achieve the lowest cost and highest production yield, DRAM is usually used as the main storage space for frame buffering. Figure 3 , Figure 3 The figure is a schematic diagram of data storage in a video encoding device in the related art. Among them, the current frame image, the reference frame image, the reconstructed frame image, the bitstreams and the temporary data are all stored in the DRAM in the video encoding device. However, the bandwidth provided by DRAM is relatively small.

[0051] It should be noted that after encoding the current frame image, it becomes a reconstructed frame image, and the reconstructed frame image of the current frame image can be used as a reference frame image of the next frame image. Similarly, after encoding the previous frame image, it becomes a reconstructed frame image of the previous frame image, and the reconstructed frame image of the previous frame image can be used as a reference frame image of the current frame image. Temporal data can be a temporal motion vector (TMV), scaled frames, or other data.

[0052] With the emergence of new video standards, such as High Efficiency Video Coding (H.265 / HEVC), Versatile Video Coding (H.266 / VVC), Alliance for Open Media Video 1 (AV1), Essential Video Coding (MPEG-5 / EVC), etc., which are targeting larger and larger screen sizes and higher and higher frame rates, it is common to increase DRAM bandwidth or increase DRAM frequency to accelerate data throughput.

[0053] Even though the hierarchical search motion estimation (ME) alleviates the problem of reading multiple reference frames, in the case of large size and high frame rate, a higher DRAM throughput is required. The increase in throughput is usually achieved by increasing the number of DRAM channels, which will cause excessive power consumption.

[0054] See also Figure 4 , Figure 4 This is a schematic diagram of increasing the number of DRAM channels for data access in the related art. By increasing the number of DRAM channels, the bandwidth can be increased and the frequency can be increased to increase the speed at which DRAM can handle data, but this will result in greater power consumption. For example, in order to meet the requirements of the video encoding device to achieve a read speed, the bandwidth of the system DRAM consumes a large amount of energy. However, regardless of whether the video encoding device is performing an immediate operation or a non-immediate operation, it is very important to maintain the highest efficiency. The method in the related art, when the video encoding device completes the encoding at the expected time, will result in a huge amount of power consumption of the DRAM.

[0055] Video encoding devices generally use blocks (which can be considered as pixel blocks) as basic units. The blocks can be rectangles, squares, trapezoids, or triangles. In this case, a comparison algorithm based on blocks appears. Figure 5 , Figure 5 It is a schematic diagram of the comparison of square blocks provided in an embodiment of the present application. The block to be compressed of the current frame image is compared with the block of the reference frame image in the form of a square block, and the reference frame image is a reconstructed image of the historical frame image, that is, an encoded image of the historical frame image. Among them, the block to be compressed and the block of the reference frame image are N×N blocks, N is an integer greater than or equal to 4. By comparing the blocks, the information redundancy in the time domain can be minimized to achieve the effect of compressing video data. Figure 5This is an example of alignment based on square blocks, but the same alignment method can also be used for blocks made up of rectangles, trapezoids or triangles.

[0056] In the embodiment of the present application, when performing motion estimation, the image is divided into a plurality of non-overlapping blocks, which form a rectangular array, each of which is a block of N×N pixels, for example, a 4×4 block, a 32×32 block, a 128×128 block, etc., where 4×4, 32×32, and 128×128 refer to the number of pixels. For each block to be encoded, the block that best matches it is searched around the same position in the reconstructed frame image of the historical frame image, i.e., the matching block. The movement of the matching block relative to the block to be encoded is called a motion vector (MV).

[0057] In the embodiment of the present application, a block to be encoded is determined from the current frame image, and the block to be encoded is a block to be compressed in the current frame image, that is, a block to be encoded in the current frame image. The block to be encoded can be a block of size N×N. When encoding the block to be encoded, it is usually necessary to compare it with a block in a reference frame image, so it is necessary to search for the block to be compared in the reference frame image. The reference frame image is a reconstructed frame image of the historical frame image, that is, an encoded image of the historical frame image.

[0058] 102. Determine a first area that needs to be repeatedly read from a reconstructed frame image of a historical frame image, and store image data of the first area in a preset memory, wherein power consumption of the preset memory is less than a preset power consumption threshold.

[0059] For example, only after searching for the block to be compared from the reconstructed frame image of the historical frame image, can the block be compared with the block to be encoded. The reconstructed frame image of the historical frame image can be the one with the lowest encoding cost among the reconstructed frame images of multiple historical frame images. Therefore, before searching for the block, it is necessary to know the search area in advance. Therefore, when searching for the block in the reconstructed frame image of the historical frame image, it is necessary to know the search range (search range, SRng) in the reconstructed frame image of the historical frame image, that is, it is necessary to determine the first area to be read repeatedly. When searching for blocks in the first area, it is necessary to read the data of the first area repeatedly. Therefore, in the embodiment of the present application, after determining the first area to be read repeatedly from the reconstructed frame image of the historical frame image, the image data of the first area is stored in a preset memory, so as to facilitate the subsequent search to read the image data of the first area from the preset memory. In addition, the power consumption of the preset memory is less than the preset power consumption threshold. By using a preset memory with low power consumption to read and write data, the power consumption of the video encoding device can be reduced.

[0060] 103. Read image data of the first area from a preset memory.

[0061] For example, after the image data of the first area is stored in the preset memory, when searching for blocks in the first area, the image data of the first area needs to be read from the preset memory to find a block that best matches the block to be encoded.

[0062] 104. Determine a matching block that matches the block to be encoded from the first area according to the read image data of the first area.

[0063] For example, by reading the image data of the first area stored in the preset memory, the first area can be searched. During the search, each block in the first area is compared with the block to be encoded in the current frame image, and the block that best matches the block to be encoded is found from the first area. The best matching block is the matching block.

[0064] The common block matching algorithm can adopt hierarchical search or non-hierarchical search, and obtains motion vectors and pixel value residues after searching for further compression coding. The pixel residue is obtained by subtracting the predicted value from the actual value of the pixel.

[0065] See also Figure 6 , Figure 6 Schematic diagram of hierarchical search provided by an embodiment of the present application. Hierarchical search is to reduce the block to be searched and the area to be searched by the same ratio, such as 1 / 2, 1 / 4 or 1 / 8, etc. On the reduced image (i.e. the area to be searched), first determine the approximate range of the block to be searched, and then return to the unreduced image to perform a more detailed block search. In the hierarchical search, the reduction ratio of each level can be the same or different, for example, the reduction ratio of each level can be 1 / 2, 1 / 4, 1 / 8 and 1 / 16.

[0066] Figure 6 Taking the three-level motion search as an example, the 1 / 4 reduced image is searched first, and then the motion vector obtained from the 1 / 4 reduced image range is searched more finely and in a smaller range in the 1 / 2 reduced image range. After that, the motion vector obtained from the 1 / 2 reduced image range is searched in the original size image range to obtain the final motion vector.

[0067] See also Figure 7 , Figure 7Schematic diagram of non-hierarchical search provided by an embodiment of the present application. Non-hierarchical search refers to performing block comparison tasks directly in an unreduced image, and common methods include full search, n-step search, and the like. Figure 7 It is to directly perform motion search on the original size image, that is, in the search window in the reconstructed frame image of the previous frame image, use the full search method to find which block in the current block and the reconstructed frame image of the previous frame image has the smallest coding cost. Among them, the smallest coding cost can be in many forms, such as the smallest coding cost can be the sum of the absolute values ​​of the residuals of each pixel of a certain block being searched and the current block is the smallest. Figure 7 The p in is the horizontal search range.

[0068] It should be noted that action estimation refers to block-based motion estimation. The basic idea is to divide each frame of the image sequence into many non-overlapping blocks, and assume that the displacement of all pixels in the block is the same. Then, for each block, the block that is most similar to the current block is found within a given specific search range of the reference frame according to a certain block matching criterion, that is, the matching block. The relative displacement between the matching block and the current block is the motion vector. Motion estimation searches for reconstructed pixels that have been encoded at different time points, that is, reconstructed pixels in the reconstructed frame image of the historical frame image.

[0069] In the inter-frame prediction mode, a preset number of historical frame images can be randomly selected from the historical frame images, and the reconstructed frame images of the selected historical frame images are searched. The search result is a motion vector, that is, the displacement between the matching block and the block to be encoded, and then the optimal motion vector is selected as the final search result. It can be understood that the position of the matching block in the reconstructed frame image of the historical frame image can be determined according to the motion vector.

[0070] 105. Encode the block to be encoded according to the relative relationship between the matching block and the block to be encoded.

[0071] For example, based on the relative displacement relationship and relative error relationship between the matching block and the block to be encoded, such as subtracting the two-dimensional pixels of the block to be encoded from the two-dimensional pixels of the corresponding position of the matching block to obtain the relative error relationship between the matching block and the block to be encoded, the block to be encoded can be encoded based on the relative displacement relationship and relative error relationship between the matching block and the block to be encoded.

[0072] It can be understood that in the embodiment of the present application, the video encoding device can determine the block to be encoded from the current frame image, determine the first area that needs to be repeatedly read from the reconstructed frame image of the historical frame image, and store the image data of the first area in a preset memory, the power consumption of the preset memory is less than the preset power consumption threshold. Then, the image data of the first area is read from the preset memory, and a matching block that matches the block to be encoded is determined from the first area based on the read image data of the first area. Afterwards, the block to be encoded is encoded based on the relative relationship between the matching block and the block to be encoded. That is, in the embodiment of the present application, the image data of the first area is stored in a preset memory with lower power consumption, so as to achieve the purpose of reducing the power consumption of the video encoding device. Therefore, the embodiment of the present application can reduce the power consumption of the video encoding device.

[0073] See also Figure 8 , Figure 8 A second flow chart of a method for performing image processing in a video encoding device provided in an embodiment of the present application. The method for performing image processing in a video encoding device can be applied to a video encoding device. The flow of the method for performing image processing in a video encoding device may include:

[0074] 201. Determine a block to be encoded from a current frame image.

[0075] For example, each frame image can be divided into multiple block lines, and each block line can be divided into multiple blocks. Before determining the block to be encoded of the current frame image, it is necessary to determine the block line to be encoded from the current frame image. The block line to be encoded refers to the block line where the block to be encoded is located. The block lines before the block line to be encoded in the current frame image are all encoded block lines.

[0076] After the block row to be encoded is determined, it is necessary to determine the block to be encoded from the block row to be encoded. In the block row to be encoded, the blocks to the left of the block to be encoded are all encoded blocks. Fig. 9 , Fig. 9 is a schematic diagram of a scene for searching in a reconstructed frame image of a historical frame image provided by an embodiment of the present application. Fig. 9 It can be seen from the figure that the block to be encoded is located within the range of the search window in the vertical projection direction.

[0077] 202. Determine a first area that needs to be repeatedly read from the reconstructed frame images of the historical frame images stored in the second memory.

[0078] For example, the first area may include multiple block rows. After the block to be encoded is determined, multiple block rows that need to be read repeatedly need to be determined from the reconstructed frame image of the historical frame image (which can be considered to be a historical frame image with the strongest correlation with the current frame image). The multiple block rows that need to be read repeatedly are the block rows located in the first area. Each block row includes multiple blocks, and the multiple blocks are arranged in a row.

[0079] The preset memory includes a first memory and a second memory. It should be noted that the reconstructed frame image of the historical frame image can be stored in the second memory in advance, and then the first area that needs to be read repeatedly is determined from the reconstructed frame image of the historical frame image stored in the second memory.

[0080] 203. Read the image data of the first area from the second memory and store it in the first memory.

[0081] For example, after determining the first area that needs to be read repeatedly from the reconstructed frame image of the historical frame image stored in the second memory, the image data of the first area can be read from the second memory, and the read image data of the first area can be stored in the first memory, waiting to be read when the video encoding device performs encoding.

[0082] It should be noted that, in the embodiment of the present application, the power consumption of the second memory is greater than a first preset multiple of the power consumption of the first memory, and the sum of the power consumption of the first memory and the power consumption of the second memory is less than a preset power consumption threshold, so that the power consumption when reading and writing data can be reduced. Among them, the preset power consumption threshold can be considered as the power consumption when all the image data of the first area is read and written by the second memory.

[0083] For example, both the first storage and the second storage are storages outside the video encoding device. For example, the first storage may include a system cache or a system buffer storage arranged outside the video encoding device, that is, the first storage may include a Sys$ or a SysBuf arranged outside the video encoding device, and the second storage may include a dynamic random access memory arranged outside the video encoding device, that is, the second storage may include a DRAM arranged outside the video encoding device. Of course, the first storage may also be other low-power storages, etc. The embodiment of the present application takes Sys$ or SysBuf as an example for explanation. Sys$ or SysBuf is composed of a plurality of SRAMs, and the second storage may be a DRAM. The power consumption of the DRAM is greater than a first preset multiple of the power consumption of the Sys$ or SysBuf outside the video encoding device, and the sum of the power consumption of the Sys$ or SysBuf outside the video encoding device and the power consumption of the DRAM is less than a preset power consumption threshold, so that the power consumption when reading and writing data can be reduced. The preset power consumption threshold can be considered as the power consumption when all the image data of the first area is read and written by the DRAM.

[0084] See also Fig.10 , Fig.10 The figure is a schematic diagram comparing the energy consumed by the static random access memory and the dynamic random access memory when reading data provided by the embodiment of the present application. The energy consumed by reading data in SRAM and reading data in DRAM differs by about 100 times, that is, the power consumption of reading data in SRAM is much less than the power consumption of reading data in DRAM. By storing the image data of multiple blocks in Sys$ or SysBuf and DRAM respectively, the power consumption when reading data can be reduced when reading image data in Sys$ or SysBuf.

[0085] The motion estimation step of the video encoding device requires a large bandwidth provided by DRAM, because during the search process, certain related areas (i.e., the first area) in the reconstructed frame image of the historical frame image will be read for block search and comparison. Due to cost considerations, the block rows covered by the search range (i.e., the first area) are usually not completely stored inside the video encoding device, and usually only the required size within the search range (such as within the search window range) is stored to meet the high-speed data access requirements during motion estimation.

[0086] If the image data of the first area is stored in the hardware of the video encoding device, that is, the cache or buffer, the cache or buffer includes multiple SRAMs. If the image data of the first area is stored in the hardware of the video encoding device, the SRAM inside the video encoding device needs to be divided into more units, each unit is a region (bank), which will cause the area of ​​a single bank to increase. As the area of ​​a single bank increases, the area of ​​the SRAM also increases, while the storage capacity of the SRAM remains unchanged, resulting in higher costs. For example, taking a width of 8192 pixels and a vertical search range of ±64 as an example, the 8-bit brightness (luma) part requires at least 1 megabyte (MB) of storage space. In addition, due to the use of a motion estimation algorithm, the SRAM needs to be divided into more units to meet the data in and out requirements, resulting in an increase in the area of ​​the SRAM.

[0087] It should be noted that the motion estimation design that stores data in the form of a search window requires block row data from multiple historical frame images to process the compression of a block row, which means that processing one frame of data requires reading multiple frames of data.

[0088] For example, when a video encoding device performs motion vector search, see Fig. 9, usually because the bandwidth required in the search window is very large, the image data of the search window is stored in a cache or buffer inside the video encoding device. The cache or buffer includes a finer SRAM group. The finer the division, the larger the area of ​​the same storage unit. For example, the average area of ​​1 bit in the bank is larger than the average area of ​​1 bit in the SRAM, so that sufficient data bandwidth can be provided to the motion estimation circuit. This not only makes the SRAM area larger, but also makes the layout routing more difficult because it is divided into more banks. Therefore, the entire first area block row will not be implemented using this method with a high storage unit (such as 1 bit) area.

[0089] That is to say, when the block rows covered by the first area move down one block row during each encoding process, the first area will be captured again. Usually the vertical search range will be multiple times the height of the block to be encoded, which means that the bandwidth for reading the image data in the first area will be multiple times the bandwidth for writing the image data in the first area. And this situation is even more serious when the picture to be encoded reaches 4K or 8K. The resolution of 4K picture is 3840×2160 pixels, and the resolution of 4K picture is 7680×4320 pixels. When encoding 4K and 8K pictures, the vertical search range must be larger than the 1080P resolution to a certain extent, otherwise the degree of picture compression will be greatly reduced.

[0090] In the embodiment of the present application, reading the image data of the first area from the second memory and storing it in the first memory in 203 may include:

[0091] If the first area is shifted down by one block row in the reconstructed frame image of the historical frame image, the image data of the shifted down block row is read from the second memory and stored in the first memory;

[0092] The block rows that are not used when encoding the next block row to be encoded in the first memory are removed.

[0093] For example, if the block row area (i.e., the first area) in the reconstructed frame image of the historical frame image that needs to be repeatedly read is stored in Sys$ or SysBuf outside the video encoding device in advance, please refer to Figures 11 to 13 , Fig.11 It is a schematic diagram of the architecture of a video compression system using a system cache provided in an embodiment of the present application. Fig.12 This is another architectural diagram of a video compression system using a system cache provided in an embodiment of the present application. Fig.13The schematic diagram of the architecture of the video compression system using the system buffer memory provided by the embodiment of the present application is shown in Figure 1. The image data of the first area is stored in Sys$ or SysBuf, and the cache or buffer inside the video encoding device stores the image data of the search window. Figures 11 to 13 Where n is a number, indicating the size of the storage capacity. For example, in one embodiment, the speed of DRAM reading and writing data is 0.5 GB / s to 2 GB / s, the speed of Sys$ or SysBuf reading and writing data is 3 GB / s to 8 GB / s, and the speed of cache or buffer reading and writing data is 10 GB / s to 50 GB / s.

[0094] It should be noted that, in other embodiments, the speed at which DRAM reads and writes data, the speed at which Sys$ or SysBuf reads and writes data, and the speed at which cache or buffer reads and writes data may also be other values, but the speed at which cache or buffer reads and writes data is greater than the speed at which Sys$ or SysBuf reads and writes data and the speed at which DRAM reads and writes data, and the speed at which Sys$ or SysBuf reads and writes data is greater than the speed at which DRAM reads and writes data.

[0095] by Fig.11 For example, Sys$ can read data from DRAM through DramC, and the data read by Sys$ from DRAM through DramC can be read by the central processing unit, the video encoding device, the image processor, and the neural network processor. When the first area moves down one block row in the reconstructed frame image of the historical frame image, Sys$ and DRAM both store the new block row, and Sys$ removes the unused block row at the same time. When the video encoding device needs to encode, it can directly read the data of the first area stored in Sys$. In addition, Sys$ also reads the image data of the first area from DRAM through DramC, and then is read by the video encoding device.

[0096] During storage, Sys$ or SysBuf outside the video encoding device will remove the block rows that will not be used when encoding the next row of block rows to be encoded, so that the number of times the cache or buffer inside the video encoding device reads the image data of the first area from DRAM can be changed from multiple times to 1 times. At the same time, since DRAM consumes 100 times more energy than SRAM for access, this can greatly reduce power consumption.

[0097] For example, since the position (i.e., the first area) and behavior (repeated reading) of the reconstructed frame image of the historical frame image are predictable, and the bandwidth required to read the reconstructed frame image of the historical frame image is multiple times the bandwidth required to read the current frame image. If the image data of the first area that is read multiple times is stored in a low-power storage space such as Sys$ or SysBuf, while effectively maintaining the operation of the video encoding device, the power consumption of the entire system can be greatly reduced, thereby improving the user experience. According to the structure of the reconstructed frame image of the historical frame image compressed by the video encoding device, it can be determined how many related block rows of the reconstructed frame image of the historical frame image to be stored in such a low-power storage space. Whenever the block row to be encoded (block row to be encoded) moves down by one row, the top block row stored in Sys$ or SysBuf is removed, and then the newly added block row when moving down in the first area is re-read, and the image data of the newly added block row is stored.

[0098] See also Fig.14 , Fig.14 It is a schematic diagram of the scene when the reconstructed frame image of the historical frame image provided by the embodiment of the present application moves down one block row. Each time the video encoding device encodes a block row downward, it expels the irrelevant block rows originally stored above Sys$ or SysBuf, and then sends the image data of the newly required block rows into Sys$ or SysBuf. That is, when the first area covered by the search range can be moved down with the block to be encoded, the unused area is expelled from Sys$ or SysBuf, and the block rows to be encoded that are about to be used are stored in Sys$ or SysBuf. That is, each time the video encoding device encodes a block row downward, it removes the irrelevant block rows above the first area stored in Sys$ or SysBuf, and then stores the newly required block rows during encoding in Sys$ or SysBuf outside the video encoding device.

[0099] 204. Read image data of the first area from the first memory block by block.

[0100] For example, when the video encoding device needs to encode, the image data of the first area can be read from the first memory block by block, such as reading the image data of the first area from Sys$ or SysBuf. When reading, the reading is performed block by block, that is, in a top-down order.

[0101] Take Advanced Video Coding (H.264 / AVC) as an example. Fig.14The size of the macro block in is 16×16 pixels, that is, 16 pixels horizontally multiplied by 16 pixels vertically. Of course, the size of the macro block can also be 32×32 pixels, 64×64 pixels, etc. This macro block is the block to be encoded in the current frame image. The vertical search range is ±64. In this case, the number of times the current frame image of the historical frame image is read is 9 (= (64+16+64) / 16) times that of the current frame image.

[0102] 205. If the number of times of reading from the first memory is greater than or equal to a preset number threshold, read the image data of the unread block rows in the first area from the second memory block row by block row.

[0103] For example, when reading the image data of the first area, the number of times originally read from the second memory can be divided into the first few times read from the first memory and the remaining few times read from the second memory. For example, under the condition of meeting the minimum demand of Sys$ or SysBuf, the 9 copies of data read from DRAM are split into 1 read from DRAM and 8 reads from Sys$ or SysBuf. The power consumption of reading data with the assistance of Sys$ or SysBuf is reduced to 11.81% of that without assistance, that is, (1×640+8×5)(9×640)=11.81%, and the power consumption in this case is relatively low. For example, according to specific needs, the 9 copies of data read from DRAM can be split into 2 reads from DRAM and 7 reads from Sys$ or SysBuf, and so on. When the power consumption is strict, the 9 copies of data read from DRAM can also be read from Sys$ or SysBuf. At this time, the power consumption is the lowest, but the cost will increase.

[0104] Since the cost of SRAM is high and the cost of DRAM is low, SRAM is generally not made too large in consideration of cost, while DRAM can be made relatively large. Therefore, in order to reduce the power consumption when reading data, the embodiment of the present application can split the original number of times of reading from DRAM into several times of reading from SRAM and several times of reading from DRAM, which can reduce the power consumption of reading data as a whole. Moreover, the number of times of reading from SRAM and the number of times of reading from DRAM can be adjusted to meet the requirements of different power consumption.

[0105] For example, when reading the image data of the first area, it can be read from Sys$ or SysBuf first. When the number of reads is greater than or equal to the preset number threshold, it is switched to reading the image data of the unread block rows in the first area from DRAM. When reading the same image data, the energy consumed by DRAM is 100 times greater than the energy consumed by SRAM. Therefore, by reading part of the image data in the first area from Sys$ or SysBuf and the other part of the data from DRAM, the power consumption of reading data can be reduced.

[0106] See also Fig.15 , Fig.15 It is a schematic diagram of a power consumption curve when reading and writing data from a multi-channel DRAM provided in an embodiment of the present application. Fig.15 In the figure, the horizontal axis is the position of the reconstructed frame image of the historical frame image, for example, the top position of the image, the middle position of the image, and the bottom position of the image, and the vertical axis is the power consumption of reading and writing data during video encoding. In the case where the video encoding device relies too much on DRAM or other cheap but power-consuming storage and high bandwidth, the power consumption limit provided by the video compression system is limited, which may make the video encoding device unable to meet the speed requirements or cause the video compression system to overheat. If the power consumption limit is considered, the speed of reading and writing data is limited and cannot reach the reading and writing speed when the power consumption limit is not considered.

[0107] See also Fig.16 , Fig.16 The embodiment of the present application provides a power consumption curve diagram when reading and writing data from Sys$ or SysBuf and DRAM. The video encoding device replaces a large amount of DRAM power consumption with the power consumption of Sys$ or SysBuf, greatly reducing power consumption.

[0108] 206. Determine image data of a search window from the read image data of the first area, where the search window is located in the first area.

[0109] For example, in order to further narrow the search range, a search window may be determined from the first region, so that the search range can be narrowed, from which matching blocks can be searched, thereby further reducing power consumption. For motion estimation, when a non-hierarchical search is used, a reconstructed frame image of a non-reduced historical frame image is used, and when a hierarchical search is used, a reconstructed frame image of a reduced or non-reduced historical frame image is used. Any search window that can predict a vertical position is applicable.

[0110] See also Fig.17 , Fig.17 A schematic diagram of a scene showing the search range of a search window in a reconstructed frame image of a historical frame image provided by an embodiment of the present application. Fig.17It can be seen from the figure that the search window is located in the first area, the area between adjacent dotted lines in the first area is the block line, and the searched motion vector can point to any place in the search window. Among them, L, R, T and B are the search ranges on the left, right, top and bottom of the block to be encoded in the search window, respectively. Among them, R and B are positive numbers, and L and T are negative numbers. And L is not necessarily equal to R, and T is not necessarily equal to B.

[0111] 207. Store the image data of the search window in a third memory, wherein the read and write speed of the third memory is greater than a second preset multiple of the read and write speed of the first memory.

[0112] For example, after the image data of the search window is determined, the image data of the search window is stored in a third memory, which may be a memory inside the video encoding device, and the third memory may include a cache or buffer set inside the video encoding device. Since the blocks within the search window are searched during motion estimation, the bandwidth requirement is high when performing block search, so the read and write speeds of the third memory are greater than the read and write speeds of the first memory and the read and write speeds of the second memory. To meet the requirements of search speed and high bandwidth. Among them, the read and write speeds of the third memory are greater than the second preset multiple of the read and write speeds of the first memory.

[0113] 208. Read the image data of the search window from the third memory, and determine, based on the image data of the search window, a block having the smallest coding cost with the block to be coded from the search window.

[0114] For example, when searching, the image data of the search window is read from the third memory, and the search can be performed in a hierarchical or non-hierarchical manner. According to the image data of the search window, the blocks in the search window read are compared with the blocks to be encoded, and the blocks with the smallest coding cost with the blocks to be encoded can be determined. For example, in one embodiment, the coding cost can include residuals, for example, in another embodiment, the coding cost can include block vectors and residuals, and so on. It can be seen that the block with the smallest coding cost can be the block with the smallest residual with the block to be encoded, and can also be the block with the smallest coding cost after comprehensively considering the block vectors and residuals with the block to be encoded.

[0115] For example, for motion estimation, the search window is scanned block by block, and the searched block is compared with the block to be encoded, so that the block with the smallest residual with the block to be encoded can be found from the search window. Among them, the motion vector can be the relative displacement between the searched block and the block to be encoded. The residual can be the difference between the two-dimensional pixel of the block to be encoded and the two-dimensional pixel of the corresponding position of the searched block.

[0116] For example, in one implementation, reading the image data of the search window from the third memory in 208, and determining the block with the smallest coding cost with the block to be coded from the search window according to the image data of the search window, may include:

[0117] Reading the image data of the search window from the third memory;

[0118] Reducing the search window according to a preset number of levels to obtain a reduced search window;

[0119] According to the image data in the reduced search window, determining from the reduced search window a reduced block having the minimum coding cost with the block to be coded;

[0120] According to the position of the reduced block in the reduced search window, a block having the smallest coding cost with the block to be encoded is determined from the search window.

[0121] For example, when searching, you can use a hierarchical search method. Depending on the number of levels, the search level will be different. For example, if a 2-level search is used, a 2-level search is performed. If a 3-level search is used, a 3-level search is performed. Of course, the more levels there are, the more accurate the search results will be, but it will also increase the consumption of system computing resources. In actual applications, the appropriate number of levels can be set according to specific needs. It should be noted that the reduction ratio of each level can be the same or different.

[0122] For example, after reading the image data of the search window from the third memory, the search window is reduced according to a preset number of levels, such as reducing the search window according to 2 levels, to obtain a reduced search window, which is 1 / 2 of the original search window size. Then, based on the image data of the reduced search window, a reduced block with the smallest coding cost with the block to be encoded is determined from the reduced search window, and the reduction ratio of the reduced block is the same as that of the reduced search window. On the image of the reduced search window, the approximate range of the reduced block to be searched is first determined, and then the image of the unreduced search window is returned to perform a more detailed block search, that is, based on the approximate range of the reduced block in the reduced search window, the original search window is searched more carefully, and the block with the smallest coding cost with the block to be encoded can be determined from the unreduced search window.

[0123] For another example, after reading the image data of the search window from the third memory, the search window is reduced according to a preset number of levels, such as reducing the search window according to 3 levels, to obtain a reduced search window, which is 1 / 4 of the original search window size. Then, based on the image data of the search window with a reduced range of 1 / 4, the reduced block with the smallest coding cost for the block to be encoded is determined from the search window with a reduced range of 1 / 4, and the corresponding block vector under the search window with a reduced range of 1 / 4 is obtained. After that, a more refined search with a smaller range is performed in the search window with a reduced range of 1 / 2, and finally, the range of the search window of the original size is searched again according to the block vector obtained from the search window with a reduced range of 1 / 2 to obtain the final block vector, so that the block with the smallest coding cost for the block to be encoded can be determined.

[0124] For another example, after reading the image data of the search window from the third memory, the search window is reduced according to a preset number of levels, such as reducing the search window according to 3 levels, to obtain a reduced search window, which is 1 / 6 of the original search window size. Then, based on the image data of the search window with a reduced range of 1 / 6, the reduced block with the smallest coding cost for the block to be encoded is determined from the search window with a reduced range of 1 / 6, and the corresponding block vector under the search window with a reduced range of 1 / 6 is obtained. After that, a more refined search with a smaller range is performed in the search window with a reduced range of 1 / 3, and finally, the range of the search window of the original size is searched again based on the block vector obtained from the search window with a reduced range of 1 / 3 to obtain the final block vector, so that the block with the smallest coding cost for the block to be encoded can be determined.

[0125] From this, it can be seen that on the image of the reduced search window, first determine the approximate range of the reduced block to be searched, and then return to the image of the unreduced search window to perform a more detailed block search, that is, based on the approximate range of the reduced block in the reduced search window, perform a more detailed search on the original search window, and determine the block with the smallest encoding cost with the block to be encoded from the unreduced search window.

[0126] 209. The block with the smallest coding cost with the block to be coded is taken as a matching block.

[0127] For example, after a block with the smallest coding cost (eg, the smallest residual) with the block to be coded is found in the search window, the block with the smallest coding cost with the block to be coded is used as a matching block.

[0128] 210. Encode the block to be encoded according to the motion vectors and residuals of the matching block and the block to be encoded.

[0129] For example, the relative relationship between the matching block and the block to be encoded may be a motion vector and a residual. After the matching block is found, the block to be encoded may be encoded according to the motion vector and the residual of the matching block and the block to be encoded.

[0130] In one implementation, encoding the block to be encoded according to the motion vector and the residual of the matching block and the block to be encoded in 210 may include:

[0131] Performing forward transform and quantization (FTQ) on the residual between the matching block and the block to be encoded;

[0132] Performing entropy coding (EC) on the motion vectors of the matching block and the block to be coded and the first residual data after forward transformation and quantization to obtain video stream coded data; or

[0133] Inversely quantize and transform the first residual data after forward transformation and quantization to obtain second residual data;

[0134] The block to be encoded is reconstructed according to the second residual data.

[0135] See also Fig.18 , Fig.18 Schematic diagram of a scene encoded by a video encoding device provided in an embodiment of the present application. Fig.18 It can be seen from the data flow relationship between motion estimation and other modules in the video encoding device. For example, motion estimation (which can use hierarchical search or non-hierarchical search) searches for reconstructed frame images of multiple historical frame images, and will find matching blocks. The relative displacement between the matching block and the current block (i.e., the block to be encoded) is the motion vector, and the residual is obtained based on the error between the current block and the matching block. The residual is forward transformed and quantized, wherein the forward transformation uses Fast Fourier Transformation (FFT) to obtain the spectrum. The horizontal axis on the spectrum curve is frequency, and the vertical axis is energy. After the forward transformation, the pixels in the space are converted into unrelated and energy-concentrated spectrum coefficients. After the forward transformation, the data is only converted to the frequency domain, and the amount of data does not change, which can reduce distortion. The matrix after the forward transformation is divided by the value of the corresponding position in the quantization matrix to achieve quantization. The spectrum coefficients are further compressed by quantization and entropy coding to obtain a compressed video stream. Among them, the quantization process removes some unimportant high-frequency information, which can compress the amount of image data, so quantization is the key to compression. After forward transformation and quantization, the first residual data is obtained.

[0136] The first residual data obtained after forward transformation and quantization is inversely quantized and transformed (De-Quantization & Inv. Transform, DQIT) to the spatial domain, that is, the second residual data of the matching block and the block to be encoded is obtained, and the block to be encoded of the current frame image is reconstructed through the picture block area (Block Reconstruction, BlkRec) as the neighbor of the next block to be encoded. The in-loop filter (In-loop Filter, InF) is used to deal with the continuity problem between blocks to make it smoother. The commonly used loop filter is a linear low-pass filter that can filter out high-frequency components and noise. Forward transformation and quantization can eliminate the spatial redundancy of video images, and entropy coding can eliminate coding redundancy.

[0137] It can be understood that the embodiment of the present application is based on the predictable data access behavior (i.e., repeated reading behavior) during video encoding, thereby realizing intelligent selection of data storage methods to reduce the power consumption of the video encoding device. Whether the data to be read should be first stored in the low-power Sys$ or SysBuf can be changed according to the frame reference reading strategy during encoding, so that the reconstructed frame image of part or all of the historical frame images stored in Sys$ or SysBuf is repeated the most times to minimize power consumption and ensure that the video encoding device can always maintain the lowest power consumption state when entering and exiting data. If the Sys$ or SysBuf also has high-speed bandwidth, since the Sys$ or SysBuf can meet the bandwidth required for repeated reading of data, the bandwidth of DRAM can be further reduced.

[0138] The embodiments of the present application can ensure that the power consumption of the video encoding device is controllable, and can enable the hardware or software of the video encoding device to complete the encoding work as quickly as possible, and make full use of the predictable behavior of the video encoding device to repeatedly read the image data of the first area to change the storage characteristics of the read data, because accessing data saves power, and the video encoding device can maintain its operating speed while reducing power consumption. The speed of reading data will not be limited by power consumption, so the video encoding device will not overheat. In addition, the delay of the SRAM in Sys$ or SysBuf when reading and writing is low, which can increase the processing frame rate and reduce the response delay. Since the power consumption can be greatly reduced, the battery life of the video encoding device can be increased, improving the user experience.

[0139] See also Fig.19 , Fig.19 1 is a third flow chart of a method for performing image processing in a video encoding device provided in an embodiment of the present application. The method for performing image processing in a video encoding device can be applied to a video encoding device, etc. The flow of the method for performing image processing in a video encoding device may include:

[0140] 301. Determine a block to be encoded from a current frame image.

[0141] The specific implementation of step 301 can refer to the embodiment of step 201, which will not be repeated here.

[0142] 302. Determine a plurality of first regions that need to be repeatedly read from a plurality of reconstructed frame images of a plurality of historical frame images stored in a second memory.

[0143] For example, after determining the block to be encoded, it is necessary to determine multiple first regions that need to be read repeatedly from the reconstructed frame images of multiple historical frame images, that is, the first region that needs to be read repeatedly is determined in the reconstructed frame image of each historical frame image. The first region may include multiple block rows, each block row includes multiple blocks, and the multiple blocks are arranged in a row.

[0144] The preset memory includes a first memory and a second memory. It should be noted that the reconstructed frame images of multiple historical frame images can be stored in the second memory in advance, and then multiple first areas that need to be repeatedly read can be determined from the reconstructed frame images of the multiple historical frame images stored in the second memory.

[0145] 303. Read image data of a plurality of first regions from a first memory and store the image data in the first memory.

[0146] For example, after multiple first areas that need to be read repeatedly are determined from the reconstructed frame images of multiple historical frame images stored in the second memory, the image data of the multiple first areas can be read from the second memory, and the read image data of the first areas can be stored in the first memory, waiting to be read when the video encoding device performs encoding.

[0147] It should be noted that, in the embodiment of the present application, for example, the power consumption of the second memory is greater than a first preset multiple of the power consumption of the first memory. After determining multiple first areas that need to be read repeatedly from the reconstructed frame images of multiple historical frame images stored in the second memory, the image data of the multiple first areas are read from the second memory, and the read image data of the multiple first areas are stored in the first memory, and the total power consumption of reading and writing data from the first memory and the second memory is less than the preset power consumption threshold, so that the power consumption when reading data can be reduced.

[0148] For example, the first memory may be Sys$ or SysBuf, and the second memory may be DRAM, and the energy consumption of the DRAM is greater than the energy consumption of Sys$ or SysBuf by a first preset multiple. Fig.10, the energy difference between reading SRAM and reading DRAM is about 100 times, that is, the energy of reading SRAM is much less than that of reading DRAM. By storing the image data of the first regions in Sys$ or SysBuf (Sys$ or SysBuf is composed of multiple SRAMs) and DRAM respectively, when the image data of the first regions is read from Sys$ or SysBuf and DRAM respectively, the power consumption when reading data can be reduced as a whole.

[0149] 304. Read image data of a plurality of first regions from the first memory block by block.

[0150] For example, image data of multiple first areas can be read from the first memory. For example, the number of times the image data is read from the first memory can be greater than the number of times the image data is read from the second memory, the number of times the image data is read from the first memory can be less than the number of times the image data is read from the second memory, or the number of times the image data is read from the first memory can be equal to the number of times the image data is read from the second memory. The specific number of times the image data is read from the first memory and the second memory respectively should be set accordingly according to the specific scenario, and the embodiments of the present application do not impose any specific restrictions on this.

[0151] For example, in one embodiment, when reading the image data of multiple first areas, it can be read from Sys$ or SysBuf (sys$ or SysBuf is composed of multiple SRAMs). When reading, it is read block by block, that is, the block rows in the first area are read from top to bottom. When the number of times read from Sys$ or SysBuf is greater than or equal to the preset number threshold, it switches to reading the remaining data from DRAM. When accessing the same data, the energy consumed by DRAM is 100 times greater than the energy consumed by SRAM. Therefore, by reading part of the image data of multiple first areas from Sys$ or SysBuf and the other part of the data from DRAM, the power consumption of reading data can be reduced. Fig.16 It can be seen that the video encoding device replaces a large amount of DRAM power consumption with the power consumption of Sys$ or SysBuf, greatly reducing power consumption.

[0152] 305. If the number of times of reading from the first memory is greater than or equal to a preset number threshold, read the image data of the unread block rows in the plurality of first regions from the second memory block row by block row.

[0153] The specific implementation of step 305 can refer to the embodiment of step 205, which will not be described in detail here.

[0154] 306. Determine image data of a plurality of search windows from the read image data of the plurality of first regions, each search window being located in a corresponding first region.

[0155] For example, after the video encoding device reads the image data of multiple first areas from the first memory and the second memory, the image data of the search window can be determined from the image data of each first area, that is, a search window can be determined in the first area of ​​the reconstructed frame image of each historical frame image, so that multiple search windows can be determined, each search window is located in the corresponding first area. When the search window is determined from the first area of ​​the reconstructed frame image of each historical frame image, its specific implementation can refer to the embodiment in step 206, which will not be repeated here.

[0156] 307. Store the image data of the plurality of search windows in a third memory, wherein a read / write speed of the third memory is greater than a second preset multiple of a read / write speed of the first memory.

[0157] For example, after the image data of the multiple search windows are determined, they can be stored in a third memory, and the read and write speed of the third memory is greater than the second preset multiple of the read and write speed of the first memory. For example, the third memory can be a cache or a buffer, and the read and write speed of the cache or the buffer is greater than the second preset multiple of the read and write speed of Sys$ or SysBuf.

[0158] 308. Read the image data of the multiple search windows from the third memory, and determine one or more blocks having the smallest coding cost with the block to be coded from the multiple search windows according to the image data of the multiple search windows.

[0159] For example, according to the image data of multiple search windows read, for the image data of each search window, the blocks in each block row in the search window are compared with the block to be encoded, so that the encoding cost of each block and the block to be encoded can be obtained, and one or more blocks can be determined in order of encoding cost from small to large, that is, one or more blocks with the smallest encoding cost with the block to be encoded are determined from each search window. For example, for motion estimation, the current search window is scanned block by block row, the blocks in the current search window are searched, the searched blocks are compared with the block to be encoded, and one or more blocks with the smallest encoding cost with the block to be encoded are found from the current search window.

[0160] For example, in one implementation, reading the image data of multiple search windows from the third memory in 308, and determining one or more blocks having the smallest coding cost with the block to be coded from the multiple search windows according to the image data of the multiple search windows, may include:

[0161] Reading the image data of the plurality of search windows from the third memory;

[0162] reducing the plurality of search windows according to a preset number of levels to obtain a plurality of reduced search windows;

[0163] According to the image data of the multiple reduced search windows, one or more reduced blocks having the smallest coding cost with the block to be encoded are respectively determined from the multiple reduced search windows;

[0164] According to the positions of the one or more reduced blocks in the reduced search window, one or more blocks having the smallest encoding cost with the block to be encoded are respectively determined from the multiple search windows.

[0165] For example, when searching, you can use a hierarchical search method. Depending on the number of levels, the search level will be different. For example, if a 2-level search is used, a 2-level search is performed. If a 3-level search is used, a 3-level search is performed. Of course, the more levels there are, the more accurate the search results will be, but it will also increase the consumption of system computing resources. In actual applications, the appropriate number of levels can be set according to specific needs. It should be noted that the reduction ratio of each level can be the same or different.

[0166] For example, after reading the image data of the multiple search windows from the third memory, the multiple search windows are reduced according to a preset number of levels, such as reducing the multiple search windows according to 2 levels, to obtain multiple reduced search windows, each of which is 1 / 2 of the size of the original search window. Then, based on the image data of the multiple reduced search windows, one or more reduced blocks with the smallest coding cost with the block to be encoded are determined from the multiple reduced search windows, and the reduction ratio of the one or more reduced blocks is the same as that of the reduced search window. On the image of each reduced search window, the approximate range of the one or more reduced blocks to be searched is first determined, and then the image of the unreduced search window is returned to perform a more detailed block search, that is, according to the approximate range of the reduced block in the reduced search window, the original search window is searched more finely, and one or more blocks with the smallest coding cost with the block to be encoded can be determined from the unreduced search window.

[0167] For another example, after reading the image data of multiple search windows from the third memory, the multiple search windows are reduced according to a preset number of levels, such as reducing the search windows according to 3 levels, to obtain multiple reduced search windows, each of which is 1 / 4 of the size of the original search window. Then, based on the image data of the multiple search windows with a 1 / 4 reduced range, one or more reduced blocks with the minimum coding cost for the block to be encoded are determined from the multiple search windows with a 1 / 4 reduced range, and one or more block vectors corresponding to the multiple search windows with a 1 / 4 reduced range are obtained. After that, a more refined search with a smaller range is performed in the multiple search windows with a 1 / 2 reduced range. Finally, the one or more block vectors obtained from the multiple search windows with a 1 / 2 reduced range are searched in the range of multiple search windows of the original size to obtain the final one or more block vectors, so that the one or more blocks with the minimum coding cost for the block to be encoded can be determined.

[0168] From this, it can be seen that on the images of multiple reduced search windows, first determine the approximate range of the reduced block to be searched, and then return to the images of multiple unreduced search windows to perform a more detailed block search, that is, based on the approximate range of the reduced block in the reduced search window, perform a more detailed search on the original search window, and one or more blocks with the smallest encoding cost with the block to be encoded can be determined from the multiple unreduced search windows.

[0169] 309. Determine a matching block from a plurality of blocks having the smallest coding cost with the block to be coded.

[0170] After one or more blocks with the smallest coding cost with the block to be coded are determined from each search window, the coding costs of these blocks and the block to be coded are compared again, and one or more blocks are determined in order of coding cost from small to large to further optimize the search results. It should be noted that the relative displacement between the one or more blocks with the smallest coding cost with the block to be coded and the block to be coded can be used as a motion vector, and the difference between the block to be coded and the one or more blocks with the smallest coding cost can be used as a residual.

[0171] For example, after finding one or more blocks with the smallest coding cost for the block to be coded from multiple search windows, since at least one block with the smallest coding cost for the block to be coded can be found in each search window, at least multiple blocks with the smallest coding cost for the block to be coded can be found in multiple search windows. From these blocks, one or more blocks can be selected in order of coding cost from small to large. Usually, one or two blocks with the smallest coding cost are selected as matching blocks. It can be seen that the number of matching blocks can be one, two, or more, depending on the number of reference blocks required. For example, when the number of reference blocks to be considered is two, two matching blocks need to be determined.

[0172] 310. Encode the block to be encoded according to the relative relationship between the matching block and the block to be encoded.

[0173] The specific implementation of step 310 can refer to the embodiment of step 210, which will not be repeated here.

[0174] It is understandable that the embodiments of the present application can select the target location or attribute of data reading according to the long-term shooting requirements of the photographic device, the low heat dissipation cost requirements, and the large power consumption caused by predictable behavior. For example, the data that needs to be read repeatedly is read from Sys$ or SysBuf and DRAM respectively, instead of all from DRAM. Since the power consumption of SRAM is much less than that of DRAM when reading the same data, the embodiments of the present application can greatly reduce the power consumption when reading data.

[0175] The embodiment of the present application uses motion estimation as an example to explain in detail how to reduce the power consumption of reading data. In other embodiments, it can also be applied to all modules and applications that require high bandwidth but have predictable data access behaviors, such as video decoders, frame rate up conversion devices, etc. The behaviors of these modules and applications are usually predictable, such as the number of repeated reads. Through these predictable behaviors, the corresponding storage characteristics can be pre-allocated, that is, the repeatedly read data is stored in a low-power memory, for example, according to the number of access requirements of the image data of all frames or part of the frames, the energy consumption of different levels of memory is matched, that is, according to the number of access requirements of the image data of all frames or part of the frames, the corresponding energy consumption of different levels is selected. When the energy consumption is different, the number of times to read data from Sys$ or SysBuf and DRAM can be reasonably allocated.

[0176] For example, a video decoder can determine the data access behavior by analyzing the bitstream in advance, and a frame rate booster can know which areas will be used multiple times during processing through simple analysis, etc. It can also be applied to fixed artificial intelligence (AI) network behavior. The part that the AI ​​network behavior repeatedly reads is the feature map part, and the AI ​​network behavior is predictable.

[0177] See also Fig. 20 , Fig. 20 The schematic diagram of the structure of the apparatus for performing image processing in a video encoding apparatus provided in an embodiment of the present application. The apparatus 400 for performing image processing in a video encoding apparatus may include: a first determination module 401, a second determination module 402, a reading module 403, a third determination module 404, and an encoding module 405.

[0178] A first determination module 401 is used to determine a block to be encoded from a current frame image;

[0179] A second determination module 402 is used to determine a first area that needs to be repeatedly read from the reconstructed frame image of the historical frame image, and store the image data of the first area in a preset memory, wherein the power consumption of the preset memory is less than a preset power consumption threshold;

[0180] A reading module 403, configured to read the image data of the first area from the preset memory;

[0181] A third determination module 404 is used to determine a matching block matching the block to be encoded from the first area according to the read image data of the first area;

[0182] The encoding module 405 is configured to encode the block to be encoded according to the relative relationship between the matching block and the block to be encoded.

[0183] In one implementation, the preset memory includes a first memory and a second memory, power consumption of the second memory is greater than a first preset multiple of power consumption of the first memory, the first area includes a plurality of block rows, and the second determining module 402 may be used to:

[0184] Determine a first area that needs to be repeatedly read from the reconstructed frame image of the historical frame image stored in the second memory;

[0185] Reading the image data of the first area from the second memory and storing it in the first memory;

[0186] The reading module 403 can be used for:

[0187] Reading the image data of the first area from the first memory block by block;

[0188] If the number of times of reading from the first memory is greater than or equal to a preset number threshold, the image data of the unread block rows in the first area are read from the second memory block by block row.

[0189] In one implementation, the second determining module 402 may be configured to:

[0190] If the first area is shifted down by one block row in the reconstructed frame image of the historical frame image, then reading the image data of the shifted block row from the second memory and storing it in the first memory;

[0191] The block rows that are not used when encoding the next block row to be encoded in the first memory are removed.

[0192] In one implementation, the third determining module 404 may be configured to:

[0193] Determine image data of a search window from the read image data of the first area, wherein the search window is located within the first area;

[0194] storing the image data of the search window in a third memory, wherein the reading and writing speed of the third memory is greater than a second preset multiple of the reading and writing speed of the first memory;

[0195] Reading the image data of the search window from the third memory, and determining a block having the smallest coding cost with the block to be coded from the search window according to the image data of the search window;

[0196] The block with the smallest coding cost with the block to be coded is used as the matching block.

[0197] In one implementation, the third determining module 404 may be configured to:

[0198] Reading the image data of the search window from the third memory;

[0199] Reducing the search window according to a preset number of levels to obtain a reduced search window;

[0200] According to the image data in the reduced search window, determining from the reduced search window a reduced block having the minimum coding cost with the block to be coded;

[0201] According to the position of the reduced block in the reduced search window, a block having the smallest coding cost with the block to be encoded is determined from the search window.

[0202] In one implementation, the relative relationship is a motion vector and a residual, and the encoding module 405 may be used to:

[0203] The block to be encoded is encoded according to the motion vectors and the residuals of the matching block and the block to be encoded.

[0204] In one implementation, the third determining module 404 may be configured to:

[0205] Performing forward transformation and quantization on the residual between the matching block and the block to be encoded;

[0206] Performing entropy coding on the motion vectors of the matching block and the block to be coded and the first residual data after forward transformation and quantization to obtain video stream coded data; or

[0207] Inversely quantize and transform the first residual data after forward transformation and quantization to obtain second residual data;

[0208] The block to be encoded is reconstructed according to the second residual data.

[0209] In one implementation, the preset memory includes a first memory and a second memory, power consumption of the second memory is greater than a first preset multiple of power consumption of the first memory, the first area includes a plurality of block rows, and the second determining module 402 may be used to:

[0210] Determine a plurality of first regions that need to be repeatedly read from the reconstructed frame images of the plurality of historical frame images stored in the second memory;

[0211] The image data of the plurality of first areas are read from the second memory and stored in the first memory.

[0212] The reading module 403 can be used for:

[0213] Reading the image data of the plurality of first regions from the first memory block by block;

[0214] If the number of times of reading from the first memory is greater than or equal to a preset number threshold, the image data of the unread block rows in the plurality of first areas are read from the second memory block row by block row.

[0215] In one implementation, the third determining module 404 may be configured to:

[0216] Determine image data of a plurality of search windows from the image data of the plurality of first regions read, each of the search windows being located in a corresponding first region;

[0217] storing the image data of the plurality of search windows in a third memory, wherein the reading and writing speed of the third memory is greater than a second preset multiple of the reading and writing speed of the first memory;

[0218] Reading the image data of the multiple search windows from the third memory, and determining one or more blocks having the smallest coding cost with the block to be coded from the multiple search windows according to the image data of the multiple search windows;

[0219] The matching block is determined from a plurality of blocks having the smallest encoding cost with the block to be encoded.

[0220] In one implementation, the third determining module 404 may be configured to:

[0221] Reading the image data of the plurality of search windows from the third memory;

[0222] reducing the plurality of search windows according to a preset number of levels to obtain a plurality of reduced search windows;

[0223] According to the image data of the multiple reduced search windows, one or more reduced blocks having the smallest coding cost with the block to be encoded are respectively determined from the multiple reduced search windows;

[0224] According to the positions of the one or more reduced blocks in the reduced search window, one or more blocks having the smallest encoding cost with the block to be encoded are respectively determined from the multiple search windows.

[0225] In one embodiment, the first memory includes a system cache or a system buffer memory disposed outside the video encoding device, and the second memory includes a dynamic random access memory disposed outside the video encoding device.

[0226] In one implementation, the third memory includes a cache or buffer disposed inside the video encoding device.

[0227] An embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed on a computer, the computer executes the process in the method for performing image processing in a video encoding device as provided in the present embodiment.

[0228] An embodiment of the present application also provides an electronic device, including a memory, a processor and a video encoding device, wherein the processor is used to execute the process of the method for performing image processing in the video encoding device provided in this embodiment by calling a computer program stored in the memory.

[0229] For example, the electronic device may be a mobile terminal such as a tablet computer or a smart phone. Fig.21 , Fig.21 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0230] The electronic device 500 may include a video encoding device 501, a memory 502, a processor 503 and other components. Those skilled in the art will appreciate that Fig.21 The electronic device structure shown in the figure does not constitute a limitation of the electronic device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0231] The video encoding device 501 may be used to encode a video image to compress the content of the video image.

[0232] The memory 502 may be used to store applications and data. The applications stored in the memory 502 include executable codes. The applications may be composed of various functional modules. The processor 503 executes various functional applications and data processing by running the applications stored in the memory 502.

[0233] The processor 503 is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device. It executes various functions of the electronic device and processes data by running or executing applications stored in the memory 502 and calling data stored in the memory 502, thereby monitoring the electronic device as a whole.

[0234] In this embodiment, the processor 503 in the electronic device will load the executable code corresponding to the process of one or more application programs into the memory 502 according to the following instructions, and the processor 503 will run the application program stored in the memory 502 to execute:

[0235] Determine a block to be encoded from the current frame image;

[0236] Determine a first area that needs to be repeatedly read from a reconstructed frame image of a historical frame image, and store image data of the first area in a preset memory, wherein power consumption of the preset memory is less than a preset power consumption threshold;

[0237] Reading the image data of the first area from the preset memory;

[0238] Determine a matching block that matches the block to be encoded from the first area according to the read image data of the first area;

[0239] The block to be encoded is encoded according to the relative relationship between the matching block and the block to be encoded.

[0240] See also Fig. 22 The electronic device 500 may include a video encoding module 501, a memory 502, a processor 503, a battery 504, an input unit 505, an output unit 506 and other components.

[0241] The video encoding module 501 can be used to encode video images to compress the content of the video images.

[0242] The memory 502 may be used to store applications and data. The applications stored in the memory 502 include executable codes. The applications may be composed of various functional modules. The processor 503 executes various functional applications and data processing by running the applications stored in the memory 502.

[0243] The processor 503 is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device. It executes various functions of the electronic device and processes data by running or executing applications stored in the memory 502 and calling data stored in the memory 502, thereby monitoring the electronic device as a whole.

[0244] The battery 504 may be used to provide power support to various components of the electronic device, thereby ensuring the normal operation of each component.

[0245] The input unit 505 may be used to receive an input video stream of a video image, for example, may be used to receive a video stream that needs to be compressed.

[0246] The output unit 506 may be configured to output a compressed video stream.

[0247] In this embodiment, the processor 503 in the electronic device will load the executable code corresponding to the process of one or more application programs into the memory 502 according to the following instructions, and the processor 503 will run the application program stored in the memory 502 to execute:

[0248] Determine a block to be encoded from the current frame image;

[0249] Determine a first area that needs to be repeatedly read from a reconstructed frame image of a historical frame image, and store image data of the first area in a preset memory, wherein power consumption of the preset memory is less than a preset power consumption threshold;

[0250] Reading the image data of the first area from the preset memory;

[0251] Determine a matching block that matches the block to be encoded from the first area according to the read image data of the first area;

[0252] The block to be encoded is encoded according to the relative relationship between the matching block and the block to be encoded.

[0253] In one embodiment, the preset memory includes a first memory and a second memory, the power consumption of the second memory is greater than a first preset multiple of the power consumption of the first memory, the first area includes multiple block rows, and when the processor 503 executes the determination of the first area that needs to be repeatedly read multiple times from the reconstructed frame image of the historical frame image and stores the image data of the first area in the preset memory, it can also execute: determining the first area that needs to be repeatedly read multiple times from the reconstructed frame image of the historical frame image stored in the second memory; reading the image data of the first area from the second memory and storing it in the first memory.

[0254] When the processor 503 executes the reading of the image data of the first area from the preset memory, it can also execute: reading the image data of the first area from the first memory block by block; if the number of readings from the first memory is greater than or equal to a preset number threshold, reading the image data of the unread block rows in the first area from the second memory block by block.

[0255] In one embodiment, when the processor 503 executes the reading of the image data of the first area from the second memory and storing it in the first memory, it can also execute: if the first area is moved down by one block row in the reconstructed frame image of the historical frame image, the image data of the moved down block row is read from the second memory and stored in the first memory; and the block rows that are not used when encoding the next block row to be encoded in the first memory are removed.

[0256] In one embodiment, when the processor 503 executes the determination of a matching block that matches the block to be encoded from the first area based on the image data of the first area read, it may also execute: determining image data of a search window from the image data of the first area read, the search window being located within the first area; storing the image data of the search window in a third memory, the read and write speed of the third memory being greater than a second preset multiple of the read and write speed of the first memory; reading the image data of the search window from the third memory, and determining a block with the smallest encoding cost with the block to be encoded from the search window based on the image data of the search window; and using the block with the smallest encoding cost with the block to be encoded as the matching block.

[0257] In one embodiment, when the processor 503 executes the reading of the image data of the search window from the third memory and determines, based on the image data of the search window, a block with the smallest coding cost relative to the block to be encoded from the search window, it may also execute: reading the image data of the search window from the third memory; reducing the search window according to a preset number of levels to obtain a reduced search window; determining, based on the image data of the reduced search window, a reduced block with the smallest coding cost relative to the block to be encoded from the reduced search window; and determining, based on the position of the reduced block in the reduced search window, a block with the smallest coding cost relative to the block to be encoded from the search window.

[0258] In one implementation, the relative relationship is a motion vector and a residual. When the processor 503 executes the encoding of the block to be encoded according to the relative relationship between the matching block and the block to be encoded, it may also execute: encoding the block to be encoded according to the motion vector and the residual between the matching block and the block to be encoded.

[0259] In one embodiment, when the processor 503 executes the encoding of the block to be encoded based on the motion vector and residual of the matching block and the block to be encoded, it can also execute: forward transforming and quantizing the residual of the matching block and the block to be encoded; entropy encoding the motion vector of the matching block and the block to be encoded and the first residual data after forward transformation and quantization to obtain video stream encoded data; or inverse quantizing and transforming the first residual data after forward transformation and quantization to obtain second residual data; and reconstructing the block to be encoded according to the second residual data.

[0260] In one embodiment, the preset memory includes a first memory and a second memory, the power consumption of the second memory is greater than a first preset multiple of the power consumption of the first memory, the first area includes multiple block rows, and when the processor 503 executes the step of determining the first area that needs to be repeatedly read multiple times from the reconstructed frame images of the historical frame images and storing the image data of the first area in the preset memory, it can also execute: determining multiple first areas that need to be repeatedly read multiple times from the reconstructed frame images of the multiple historical frame images stored in the second memory; reading the image data of the multiple first areas from the second memory and storing them in the first memory.

[0261] When the processor 503 executes the reading of the image data of the first area from the preset memory, it can also execute: reading the image data of the multiple first areas from the first memory block by block; if the number of times of reading from the first memory is greater than or equal to the preset number threshold, reading the image data of the unread block rows of the multiple first areas from the second memory block by block.

[0262] In one embodiment, when the processor 503 executes the determination of a matching block that matches the block to be encoded from the first area based on the image data of the first area read, it can also execute: determining image data of multiple search windows from the image data of the multiple first areas read, each of the search windows is located in the corresponding first area; storing the image data of the multiple search windows in a third memory, the read and write speed of the third memory being greater than a second preset multiple of the read and write speed of the first memory; reading the image data of the multiple search windows from the third memory, and determining one or more blocks with the smallest encoding cost with the block to be encoded from the multiple search windows based on the image data of the multiple search windows; and determining the matching block from the multiple blocks with the smallest encoding cost with the block to be encoded.

[0263] In one embodiment, when the processor 503 executes the reading of the image data of the multiple search windows from the third memory, and determines, based on the image data of the multiple search windows, one or more blocks with the smallest coding cost with the block to be encoded from the multiple search windows, the following steps may be performed: reading the image data of the multiple search windows from the third memory; reducing the multiple search windows according to a preset number of levels to obtain a plurality of reduced search windows; based on the image data of the multiple reduced search windows, determining, from the multiple reduced search windows, one or more reduced blocks with the smallest coding cost with the block to be encoded; and based on the positions of the one or more reduced blocks in the reduced search windows, determining, from the multiple search windows, one or more blocks with the smallest coding cost with the block to be encoded.

[0264] In one embodiment, the first memory includes a system cache or a system buffer memory disposed outside the video encoding device, and the second memory includes a dynamic random access memory disposed outside the video encoding device.

[0265] In one implementation, the third memory includes a cache or buffer disposed inside the video encoding device.

[0266] The present application also provides an image processing system. Fig.23 and Fig.24 , Fig.23 It is a structural diagram of the image processing system provided in an embodiment of the present application. Fig.246 is another structural schematic diagram of the image processing system provided by an embodiment of the present application. The image processing system 600 includes a video encoding device 601, a first memory 602, and a second memory 603, wherein the power consumption of the second memory 603 is greater than a first preset multiple of the power consumption of the first memory 602, the video encoding device 601 may include a third memory, the reading speed of the third memory is greater than a second preset multiple of the reading speed of the first memory, the first memory 602 and the second memory 603 respectively store image data that needs to be repeatedly read in the reconstructed frame image of the historical frame image, and the video encoding device 601 reads the image data that is repeatedly read from the first memory 602 and the second memory 603 according to the preset number of times when encoding, and determines the image data in the search window therefrom, and stores the image data in the search window in the third memory.

[0267] For example, after the reconstructed frame image of the historical frame image is stored in the second memory 603, the first area that needs to be repeatedly read can be determined from the reconstructed frame image of the historical frame image stored in the second memory 603, and then the image data of the first area is read from the second memory 603, and the read image data of the first area is stored in the first memory 602. During encoding, the video encoding device 601 can read the image data of the first area block by block from the first memory 602. If the number of times of reading from the first memory 602 is greater than or equal to the preset number threshold, the image data of the block lines that have not been read in the first area are read from the second memory 603 block by block.

[0268] It should be noted that when reading image data from the second memory 603, the video encoding device 601 can read the image data directly from the second memory 603, or the first memory 602 reads the image data from the second memory 603 and then stores it, and the video encoding device 601 reads the image data directly from the first memory 602.

[0269] The video encoding device 601 can read the image data in the search window from the third memory, determine the matching block that matches the module to be encoded from the search window based on the image data in the search window read from the third memory, and encode according to the motion vector and residual of the matching block and the block to be encoded.

[0270] In the above embodiments, the description of each embodiment has its own emphasis. For the part that is not described in detail in a certain embodiment, please refer to the detailed description of the method for performing image processing in the video encoding device above, which will not be repeated here.

[0271] The device for performing image processing in a video encoding device provided in the embodiment of the present application belongs to the same concept as the method for performing image processing in a video encoding device in the above embodiment. Any method provided in the embodiment of the method for performing image processing in a video encoding device can be run on the device for performing image processing in a video encoding device. The specific implementation process is detailed in the embodiment of the method for performing image processing in a video encoding device, which will not be repeated here.

[0272] It should be noted that, for the method of performing image processing in a video encoding device described in the embodiment of the present application, a person of ordinary skill in the art can understand that the implementation of all or part of the process of the method of performing image processing in a video encoding device described in the embodiment of the present application can be completed by controlling the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, such as a memory, and executed by at least one processor, and the execution process may include the process of the embodiment of the method of performing image processing in a video encoding device. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), etc.

[0273] For the device for performing image processing in the video encoding device of the embodiment of the present application, each functional module can be integrated into a processing chip, or each module can exist physically separately, or two or more modules can be integrated into one module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium, such as a read-only memory, a disk or an optical disk.

[0274] The above is a detailed introduction to a method, device, storage medium, electronic device and system for performing image processing in a video encoding device provided by an embodiment of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A method for performing image processing in a video encoding device, characterized in that: The method comprises: Determine a block to be encoded from the current frame image; Determine a first area that needs to be repeatedly read multiple times from a reconstructed frame image of a historical frame image stored in a second memory, wherein the first area includes a plurality of block rows; Reading the image data of the first area from the second memory and storing it in the first memory, the power consumption of the second memory being greater than a first preset multiple of the power consumption of the first memory; Reading the image data of the first area from the first memory block by block; If the number of times of reading from the first memory is greater than or equal to a preset number threshold, reading the image data of the unread block rows in the first area from the second memory block by block row; Determine a matching block that matches the block to be encoded from the first area according to the read image data of the first area; The block to be encoded is encoded according to the relative relationship between the matching block and the block to be encoded.

2. The method for performing image processing in a video encoding device according to claim 1, characterized in that: The step of reading the image data of the first area from the second memory and storing the image data in the first memory comprises: If the first area is shifted down by one block row in the reconstructed frame image of the historical frame image, then reading the image data of the shifted block row from the second memory and storing it in the first memory; The block rows that are not used when encoding the next block row to be encoded in the first memory are removed.

3. The method for performing image processing in a video encoding device according to claim 2, characterized in that: The determining, from the first area according to the read image data of the first area, a matching block that matches the block to be encoded, comprises: Determine image data of a search window from the read image data of the first area, wherein the search window is located within the first area; storing the image data of the search window in a third memory, wherein the reading and writing speed of the third memory is greater than a second preset multiple of the reading and writing speed of the first memory; Reading the image data of the search window from the third memory, and determining a block having the smallest coding cost with the block to be coded from the search window according to the image data of the search window; The block with the smallest coding cost with the block to be coded is used as the matching block.

4. The method for performing image processing in a video encoding device according to claim 3, characterized in that: The step of reading the image data of the search window from the third memory and determining, based on the image data of the search window, a block having the smallest coding cost with the block to be coded from the search window comprises: Reading the image data of the search window from the third memory; Reducing the search window according to a preset number of levels to obtain a reduced search window; According to the image data in the reduced search window, determining from the reduced search window a reduced block having the minimum coding cost with the block to be coded; According to the position of the reduced block in the reduced search window, a block having the smallest coding cost with the block to be encoded is determined from the search window.

5. The method for performing image processing in a video encoding device according to claim 1, characterized in that: The relative relationship is a motion vector and a residual, and encoding the block to be encoded according to the relative relationship between the matching block and the block to be encoded includes: The block to be encoded is encoded according to the motion vectors and the residuals of the matching block and the block to be encoded.

6. The method for performing image processing in a video encoding device according to claim 5, characterized in that: The step of encoding the block to be encoded according to the motion vector and the residual of the matching block and the block to be encoded comprises: Performing forward transformation and quantization on the residual between the matching block and the block to be encoded; Performing entropy coding on the motion vectors of the matching block and the block to be coded and the first residual data after forward transformation and quantization to obtain video stream coded data; or Inversely quantize and transform the first residual data after forward transformation and quantization to obtain second residual data; The block to be encoded is reconstructed according to the second residual data.

7. The method for performing image processing in a video encoding device according to claim 1, characterized in that: The first memory includes a system cache or a system buffer memory arranged outside the video encoding device, and the second memory includes a dynamic random access memory arranged outside the video encoding device.

8. The method for performing image processing in a video encoding device according to claim 3, characterized in that: The third memory includes a cache or buffer disposed inside the video encoding device.

9. A device for performing image processing in a video encoding device, characterized in that: The device comprises: A first determination module, used to determine a block to be encoded from a current frame image; A second determination module is used to: determine a first area that needs to be repeatedly read multiple times from the reconstructed frame image of the historical frame image stored in the second memory, the first area including a plurality of block rows; read the image data of the first area from the second memory and store it in the first memory, the power consumption of the second memory being greater than a first preset multiple of the power consumption of the first memory; A reading module, configured to: read the image data of the first area from the first memory block by block; if the number of times of reading from the first memory is greater than or equal to a preset number threshold, read the image data of the unread block in the first area from the second memory block by block; A third determination module, configured to determine a matching block matching the block to be encoded from the first area according to the image data of the first area read; The encoding module is used to encode the block to be encoded according to the relative relationship between the matching block and the block to be encoded.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed on a computer, the computer is caused to execute the method according to any one of claims 1 to 8.

11. An electronic device comprising a memory, a processor and a video encoding device, characterized in that: The processor executes the method according to any one of claims 1 to 8 by calling the computer program stored in the memory.

12. An image processing system, characterized in that: The invention comprises a video encoding device, a first memory and a second memory, wherein the power consumption of the second memory is greater than a first preset multiple of the power consumption of the first memory, the video encoding device comprises a third memory, the reading speed of the third memory is greater than a second preset multiple of the reading speed of the first memory, the first memory and the second memory respectively store image data repeatedly read from reconstructed frame images of historical frame images, when encoding, the video encoding device reads the repeatedly read image data from the first memory and the second memory respectively according to a preset number of times, and determines the image data within a search window therefrom, and stores the image data within the search window in the third memory, the video encoding device reads the image data within the search window from the third memory, and determines a matching block that matches the block to be encoded, and encodes according to the motion vector and residual of the matching block and the block to be encoded.

13. The image processing system according to claim 12, characterized in that: A first area that needs to be read repeatedly for multiple times is determined from the reconstructed frame image of the historical frame image stored in the second memory, and image data of the first area is read from the second memory and stored in the first memory. During encoding, the video encoding device reads the image data of the first area from the first memory block by block. If the number of times read from the first memory is greater than or equal to a preset number threshold, the image data of the unread block rows in the first area are read from the second memory block by block.

Images