Method, device and system for image processing in video decoding device

Abstract

The present application discloses a method, device, storage medium, electronic device and system for image processing in a video decoding device. The method includes: obtaining a video code stream; determining one or more reference positions from the video code stream; determining the reference times of one or more reference positions; determining the reference positions to be stored in a preset memory from one or more reference positions according to a preset power consumption threshold and the reference times of one or more reference positions, and storing them in the preset memory, the power consumption generated by storing the reference positions in the preset memory and reading the reference positions from the preset memory is less than or equal to the preset power consumption threshold; decoding the object to be decoded according to the reference positions. The present application can reduce the power consumption of a video decoding device.

Description

Technical Field

[0001] The present application belongs to the technical field of electronic equipment, and in particular, relates to a method, device, storage medium, electronic equipment and system for performing image processing in a video decoding device. Background Art

[0002] With the continuous development of technology, the functions of video decoding devices are becoming more and more powerful. Video decoding devices can decode video images. When decoding a frame of video image, data of multiple frames of decoded video images are usually referenced. However, in the related art, when reading the data of the decoded video image that needs to be referenced, the power consumption of the video decoding device is relatively large. Summary of the invention

[0003] Embodiments of the present application provide a method, device, storage medium, and electronic device for performing image processing in a video decoding device, which can reduce the power consumption of the video decoding device.

[0004] In a first aspect, an embodiment of the present application provides a method for performing image processing in a video decoding device, the method comprising:

[0005] Get the video stream;

[0006] Determining one or more reference positions from the video stream;

[0007] determining a reference number of the one or more reference positions;

[0008] According to a preset power consumption threshold and the reference number of the one or more reference positions, determining a reference position to be stored in a preset memory from the one or more reference positions, and storing the reference position in the preset memory, wherein the power consumption generated by storing the reference position in the preset memory and reading the reference position from the preset memory is less than or equal to the preset power consumption threshold; and

[0009] The object to be decoded is decoded according to the reference position.

[0010] In a second aspect, an embodiment of the present application provides a method for performing image processing in a video decoding device, the method comprising:

[0011] Get the video stream;

[0012] Acquire one or more reference motion vectors (MV) according to the video code stream;

[0013] Acquire one or more corresponding reference blocks from one or more image frames of the video code stream according to the one or more reference motion vectors;

[0014] determining a reference number of the one or more reference blocks;

[0015] According to a preset power consumption threshold and the number of references of the one or more reference blocks, determine one or more reference blocks to be stored in a preset memory from the one or more reference blocks, and store them in the preset memory, wherein power consumption generated by storing the reference blocks in the preset memory and reading the reference blocks from the preset memory is less than or equal to the preset power consumption threshold; and

[0016] The block to be decoded or a sub-block of the block to be decoded is decoded according to the reference block.

[0017] In a third aspect, an embodiment of the present application provides a device for performing image processing in a video decoding device, the device comprising:

[0018] The acquisition module is used to obtain the video code stream;

[0019] A first determination module, configured to determine one or more reference positions from the video bitstream;

[0020] A second determination module, configured to determine a reference number of the one or more reference positions;

[0021] A third determining module is used to determine a position to be stored in a preset memory from the one or more reference positions according to a preset power consumption threshold and a reference number of the one or more reference positions, and store the position in the preset memory, wherein the power consumption generated by storing the reference position in the preset memory and reading the reference position from the preset memory is less than or equal to the preset power consumption threshold; and

[0022] A decoding module is used to decode the object to be decoded according to the reference position.

[0023] In a fourth aspect, an embodiment of the present application provides an image processing device, the device comprising:

[0024] A first acquisition module, used to acquire a video code stream;

[0025] A second acquisition module, used to acquire one or more reference motion vectors according to the video code stream;

[0026] A third acquisition module, configured to acquire one or more corresponding reference blocks from one or more image frames of the video code stream according to the one or more reference motion vectors;

[0027] A first determining module, configured to determine a reference number of the one or more reference blocks;

[0028] A third determining module is used to determine one or more reference blocks that need to be stored in a preset memory from the one or more reference blocks according to a preset power consumption threshold and the number of references of the one or more reference blocks, and store them in the preset memory, wherein the power consumption generated by storing the reference blocks in the preset memory and reading the reference blocks from the preset memory is less than or equal to the preset power consumption threshold; and

[0029] A decoding module is used to decode the block to be decoded or a sub-block of the block to be decoded according to the reference block.

[0030] In a fifth aspect, an embodiment of the present application provides a storage medium on which a computer program is stored. When the computer program is executed on a computer, the computer executes the method for performing image processing in a video decoding device provided in an embodiment of the present application.

[0031] In a sixth aspect, an embodiment of the present application further provides an electronic device, comprising a memory, a processor and a video decoding device, wherein the processor is used to execute the method for performing image processing in a video decoding device provided in an embodiment of the present application by calling a computer program stored in the memory.

[0032] In the seventh aspect, an embodiment of the present application also provides an image processing system, including a video decoding device, a first memory, and a second memory, the power consumption of the first memory is greater than the power consumption of the second memory, the first memory stores reference positions with a reference number of once, or stores reference positions with a reference number of once and multiple times, and the second memory stores reference positions with a reference number of multiple times. When decoding, the video decoding device reads the reference position with a reference number of once from the first memory, and reads the reference position with a reference number of multiple times from the second memory, and decodes the object to be decoded according to the reference positions.

[0033] In an embodiment of the present application, a video decoding device can obtain a video code stream and determine one or more reference positions from the video code stream. Then, the reference times of one or more reference positions are determined, and based on a preset power consumption threshold and the reference times of one or more reference positions, a reference position that needs to be stored in a preset memory is determined from one or more reference positions, and the reference position is stored in the preset memory, and the power consumption generated by storing the reference position in the preset memory and reading the reference position from the preset memory is less than or equal to the preset power consumption threshold. Afterwards, the object to be decoded is decoded according to the reference position. That is, in an embodiment of the present application, the image data of the reference position that needs to be stored in the preset memory is stored in a preset memory with lower power consumption, so as to achieve the purpose of reducing the power consumption of the video decoding device. Therefore, an embodiment of the present application can reduce the power consumption of the video decoding device. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The technical solution and beneficial effects of the present application will be made apparent by describing in detail the specific implementation methods of the present application in conjunction with the accompanying drawings.

[0035] Figure 1 This is a first flow chart of a method for performing image processing in a video decoding device provided in an embodiment of the present application.

[0036] Figure 2 It is a structural diagram of a video decoding system in the related art.

[0037] Figure 3 It is a schematic diagram of data storage in a video decoding device in the related art.

[0038] Figure 4 It is a schematic diagram of increasing the number of channels of dynamic random access memory (DRAM) for data access in the related art.

[0039] Figure 5 It is a schematic diagram of a power consumption curve when reading and writing data from a multi-channel DRAM during video decoding in the related art.

[0040] Figure 6 It is a scene schematic diagram of the reference relationship between each image frame in an image group in the current video code stream provided by an embodiment of the present application.

[0041] Figure 7 This is a second flow chart of the method for performing image processing in a video decoding device provided in an embodiment of the present application.

[0042] Figure 8 It is a schematic diagram comparing the energy consumed by a static random access memory (SRAM) and a dynamic random access memory when reading data provided by an embodiment of the present application.

[0043] Fig. 9 It is a schematic diagram of a scenario in which image data that is referenced multiple times is stored in a system cache (system cache, Sys$) or a system buffer (System Buffer, SysBuf) provided by an embodiment of the present application.

[0044] Fig.10 It is a schematic diagram of a scenario for roughly analyzing the reference relationship between multiple image frames in a video stream provided by an embodiment of the present application.

[0045] Fig.11It is a schematic diagram of the architecture of a video decoding system using a system cache provided in an embodiment of the present application.

[0046] Fig.12 This is another architectural diagram of a video decoding system using a system cache provided in an embodiment of the present application.

[0047] Fig.13 1 is a schematic diagram of the architecture of a video decoding system using a system buffer memory provided in an embodiment of the present application.

[0048] Fig.14 It is a schematic diagram of the power consumption curve when reading and writing data from Sys$ or SysBuf provided in an embodiment of the present application.

[0049] Fig.15 This is a third flow chart of the method for performing image processing in a video decoding device provided in an embodiment of the present application.

[0050] Fig.16 It is a scene schematic diagram of the reference relationship between blocks in each image frame in an image group in a video code stream provided by an embodiment of the present application.

[0051] Fig.17 This is a fourth flow chart of the method for performing image processing in a video decoding device provided in an embodiment of the present application.

[0052] Fig.18 It is a schematic diagram of a scenario for finely analyzing the reference relationship of blocks in multiple image frames in a video code stream provided by an embodiment of the present application.

[0053] Fig.19 It is a schematic diagram of a decoding scenario of a video decoding device provided in an embodiment of the present application.

[0054] Fig. 20 It is a structural schematic diagram of the image processing device provided in an embodiment of the present application.

[0055] Fig.21 It is another structural schematic diagram of the image processing device provided in an embodiment of the present application.

[0056] Fig. 22 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0057] Fig.23 It is another structural schematic diagram of the electronic device provided in an embodiment of the present application.

[0058] Fig.24 It is a structural diagram of the image processing system provided in an embodiment of the present application.

[0059] Fig.25It is another structural schematic diagram of the image processing system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0060] Please refer to the drawings, where the same component symbols represent the same components. The principle of the present application is illustrated by implementing it in an appropriate computing environment. The following description is based on the illustrated specific embodiments of the present application, which should not be considered as limiting other specific embodiments of the present application that are not described in detail herein.

[0061] See also Figure 1 , Figure 1 1 is a schematic diagram of a first flow chart of a method for performing image processing in a video decoding device provided by an embodiment of the present application. The method for performing image processing in a video decoding device can be applied to a video decoding device. The flow chart of the method for performing image processing in a video decoding device may include:

[0062] 101. Obtain video stream.

[0063] With the continuous development of technology, the functions of video decoding devices are becoming more and more powerful. Video decoding devices can decode video images. When decoding a frame of video image, data of multiple frames of decoded video images are usually referenced. However, in the related art, when reading the data of the decoded video image that needs to be referenced, the power consumption of the video decoding device is relatively large.

[0064] See also Figure 2 , Figure 2 It is a structural diagram of a video decoding system in the related art. In the video decoding system, the central processing unit (CPU), video decoding device and display processor (DISP) read and write data from DRAM through the bus and dynamic random access memory controller (DRAMC). The central processing unit, video decoding device and display processor share bandwidth in a time-sharing manner. The priority of the central processing unit and the display processor is higher than that of the video encoder. It should be noted that, according to specific needs, a display processor may or may not be set in the video decoding system. The video decoding device needs to perform motion compensation (MC) when decoding, which will occupy a large bandwidth.

[0065] Video decoding devices attach great importance to cost. In order to achieve the lowest cost and highest production yield, DRAM is usually used as the main storage space for frame buffering. Figure 3 , Figure 3 Schematic diagram of data storage in a video decoding device in the related art. Bitstreams, image frames to be buffered, and temporary data are all stored in DRAM in the video decoding device. However, the bandwidth provided by DRAM is small. Temporary data may be a temporal motion vector (TMV) and other data.

[0066] Although the video decoding device can adopt an internal cache strategy of the video decoding device for various video bitstreams, for example, the video bitstream can be the first video and audio lossy compression standard organized by the Moving Picture Experts Group Phase 1 (MPEG-1), the second video and audio lossy compression standard organized by the Moving Picture Experts Group 2 (MPEG-2), the fourth video and audio lossy compression standard organized by the Moving Picture Experts Group 4 (MPEG-4), Essential Video Coding (MPEG-5 / EVC), the low bit rate video coding standard H.263 for video conferencing formulated by the International Telecommunication Union Telecommunication Standardization Sector (ITU Telecommunication Standardization Sector, ITU-T), Advanced Video Coding (H.264 / AVC), High Efficiency Video Coding (H.265 / HEVC), Versatile Video Coding ( Coding, H.266 / VVC), Video Predictor 8, VP8, Video Predictor 9, VP9, ​​and Alliance for Open Media Video 1, AV1.

[0067] However, with the emergence of new video standards, such as H.265 / HEVC, H.266 / VVC, AV1, MPEG-5, etc., which target larger and larger screen sizes and higher and higher frame rates, increasing DRAM bandwidth or increasing DRAM frequency is usually used to accelerate data throughput.

[0068] See also Figure 4 , Figure 4 This is a schematic diagram of data access by increasing the number of DRAM channels in the related art. By increasing the number of DRAM channels, the bandwidth can be increased and the frequency can be increased to increase the speed of DRAM data throughput, but it will cause greater power consumption.

[0069] For example, in order to meet the requirement of the video decoding device to achieve the expected decoding speed, the bandwidth of the system DRAM consumes a large amount of energy. However, whether the video decoding device performs real-time operation or non-real-time operation, it is very important to maintain the highest efficiency. The method in the related art will cause a great deal of DRAM power consumption when the video decoding device completes decoding within the expected time.

[0070] See also Figure 5 , Figure 5 It is a schematic diagram of a power consumption curve when reading and writing data from a multi-channel DRAM during video decoding in the related art. Figure 5 In the figure, the horizontal axis is the position of the reference image frame, such as the top position, middle position, bottom position, etc. of the reference image frame, and the vertical axis is the power consumption of reading and writing data during video decoding. When performing video decoding, data needs to be transferred in and out of DRAM. In the case where the video decoding device relies too much on DRAM or other cheap but power-consuming storage and high bandwidth, the power consumption limit provided by the video decoding system is limited, which will make the video decoding device unable to meet the decoding speed requirements, or will cause the video decoding system to overheat. If the power consumption limit is considered, the speed of reading and writing data from DRAM is limited, and cannot reach the reading and writing speed when the power consumption limit is not considered.

[0071] In an embodiment of the present application, a video stream is obtained, and the video stream may include one or more groups of pictures (Group of pictures, GOP), and one group of pictures includes multiple image frames. In an embodiment of the present application, a video stream including one group of pictures is used as an example for explanation. The obtained video stream is an encoded video stream, and the image frames in the video stream may not have been decoded, and some image frames may have been decoded, and other image frames may be waiting to be decoded. It should be noted that the decoded image frames can be used as reference image frames when decoding other subsequent image frames.

[0072] 102. Determine one or more reference positions from the video stream.

[0073] For example, in an embodiment of the present application, when decoding an image frame to be decoded, it is usually necessary to compare it with a reference position. In one embodiment, the reference position may include an image frame, a reference slice, or a reference area. For example, the image frame to be decoded is compared with a reference image frame, a reference slice, or a reference area, that is, the image frame to be decoded needs to refer to an image frame, a slice, or an area that has been decoded. Among them, the image frame to be decoded is an image frame that needs to be decoded. It should be noted that a strip contains part of the data or all of the data of an image frame. In other words, an image frame can be encoded into one or more strips. A strip contains at least one block and at most the data of the entire image frame.

[0074] In different coding implementations, the number of stripes in the same image frame may not be the same. For example, the purpose of designing stripes in H.264 is mainly to prevent the spread of bit errors. Because the decoding operations of different stripes are independent, the data referenced by the decoding process of a stripe cannot cross the boundary of the stripe.

[0075] In the embodiment of the present application, the video code stream can be analyzed by the software or hardware of the video decoding device, and one or more reference positions can be roughly determined from the video code stream, such as determining one or more reference image frames, reference strips or reference areas from the video code stream, that is, determining one or more reference image frames from the video code stream, or determining one or more reference strips from the video code stream, or determining one or more reference areas from the video code stream, and the determined one or more reference image frames, reference strips or reference areas form a reference queue, so as to refer to these reference image frames, reference strips or reference areas when decoding the image frames to be decoded.

[0076] For example, in one embodiment, the reference area may be an area in an image frame, which needs to be referenced by the image to be decoded. For another example, in another embodiment, the reference area may be an area in a stripe, which needs to be referenced by the image to be decoded, and so on.

[0077] 103. Determine a reference number of one or more reference positions.

[0078] For example, after one or more reference positions are determined, the reference times of the one or more reference positions may be determined.

[0079] For example, taking the reference image frame as an example, see Figure 6 , Figure 6 It is a scene schematic diagram of the reference relationship between image frames in an image group in a video code stream provided by an embodiment of the present application. Figure 6The example of an image group including 9 image frames is used for explanation. In other implementations, the number of image frames included in the image group can be adjusted according to specific needs. In an image group, the display order of the image frames may be the same as the decoding order, or may be different. Figure 6 The display order and decoding order of image frames in the image group displayed in are different.

[0080] from Figure 6 The direction of the arrow in the image group shows the number of times each image frame is referenced by other image frames in the image group. That is, the number of times each image frame is referenced by the image frame pointed by the arrow can be determined according to the direction of the arrow. When an image frame is referenced by other image frames in the image group once or more times, the image frame can be used as a reference image frame. For example, Figure 6 The I frame in the display order is referenced four times, the B frame in the display order is referenced twice, the B frame in the display order is referenced once, the P frame in the display order is referenced five times, the B frame in the display order is referenced twice, and so on. Figure 6 In the above, only a rough frame-level analysis is performed by the relevant hardware or software of the video decoding device. The reference relationship of each specific block in the reference frame image cannot be analyzed through this rough analysis method.

[0081] 104. According to a preset power consumption threshold and the reference number of one or more reference positions, a reference position to be stored in a preset memory is determined from one or more reference positions, and the reference position is stored in the preset memory, and the power consumption generated by storing the reference position in the preset memory and reading the reference position from the preset memory is less than or equal to the preset power consumption threshold.

[0082] For example, from Figure 6 It can be seen that the I frame with display order 0, the B frame with display order 2, the P frame with display order 4, the B frame with display order 6, and the B frame with display order 8 are referenced multiple times by other image frames in the image group.

[0083] However, when decoding a video, the part that needs to be referenced will go in and out of DRAM multiple times, that is, the part that needs to be referenced will be read from DRAM multiple times. If a piece of data is expected to be read multiple times, the energy consumed when it is read repeatedly will be as much as a hundred times. In order to improve the compression rate, today's video standards often use multiple reference image frames for encoding. This behavior means that some data is often repeatedly used because of the high correlation in the time domain. In addition, with the increasing number of high-frame rate video streams, the repeatability in the time domain increases, and the encoding effect of the same reference image frame can be reused, and most video encoders will generate such video streams. If the part that is repeatedly read during decoding is stored in a low-power storage medium, the energy consumption during video playback will be greatly reduced.

[0084] For example, in an embodiment of the present application, a reference position to be stored in a preset memory can be determined from one or more reference positions, such as storing a reference image frame, a reference strip, or a reference area with multiple reference times in a preset memory, so as to facilitate the reading of image data of the reference image frame, reference strip, or reference area during subsequent decoding of the object to be decoded (such as an image frame to be decoded, a strip to be decoded, or an area to be decoded). In addition, the power consumption generated by storing the reference position in the preset memory and reading the reference position from the preset memory is less than or equal to the preset power consumption threshold, and the power consumption of the video decoding device can be reduced by using a preset memory with low power consumption to read and write data.

[0085] 105. Decode the object to be decoded according to the reference position.

[0086] For example, the reference image frame, reference strip or reference area stored in the preset memory may be referenced by the object to be decoded multiple times, so when decoding, the reference image frame, reference strip or reference area stored in the preset memory will be read multiple times. After the image data of the reference image frame, reference strip or reference area is read from the preset memory, the object to be decoded can refer to the image data of the reference image frame, reference strip or reference area, that is, the object to be decoded can be decoded according to the read image data of the reference image frame, reference strip or reference area.

[0087] For example, in one embodiment, the object to be decoded may include a block to be decoded in an image frame to be decoded, a block to be decoded in a strip to be decoded, or a block to be decoded in an area to be decoded. The block to be decoded in the image frame to be decoded may be decoded according to the image data of the reference image frame read, the block to be decoded in the strip to be decoded may be decoded according to the image data of the reference strip read, or the block to be decoded in the area to be decoded may be decoded according to the image data of the reference area read.

[0088] It can be understood that in an embodiment of the present application, the video decoding device can obtain a video code stream and determine one or more reference positions from the video code stream. Then, the reference times of one or more reference positions are determined, and the reference positions that need to be stored in the preset memory are determined from the one or more reference positions according to the preset power consumption threshold and the reference times of one or more reference positions, and the reference positions are stored in the preset memory, and the power consumption generated by storing the reference positions in the preset memory and reading the reference positions from the preset memory is less than or equal to the preset power consumption threshold. Afterwards, the object to be decoded is decoded according to the reference position. That is, in an embodiment of the present application, the image data of the reference position that needs to be stored in the preset memory is stored in a preset memory with lower power consumption, so as to achieve the purpose of reducing the power consumption of the video decoding device. Therefore, an embodiment of the present application can reduce the power consumption of the video decoding device.

[0089] See also Figure 7 , Figure 7 A second flow chart of a method for performing image processing in a video decoding device provided in an embodiment of the present application. The method for performing image processing in a video decoding device can be applied to a video decoding device. The flow chart of the method for performing image processing in a video decoding device may include:

[0090] 201. Obtain video code stream.

[0091] The specific implementation of step 201 can refer to the embodiment of step 101, which will not be repeated here.

[0092] 202. Determine one or more reference image frames, reference strips or reference areas from the video code stream according to frame header information of an image frame in the video code stream or slice header information of one or more slices in the image frame.

[0093] For example, the reference relationship of each image frame can be determined based on the frame header information of each image frame in the video code stream or the strip header information of the strip in the image frame. Among them, the data of each image frame can be regarded as a network abstraction layer (NAL) unit, and the frame header information is used to distinguish the beginning of an image frame. The frame header information can also be regarded as the NAL unit header information. The frame header information can be used to determine which image frame it is, that is, the reference image frame can be determined. The strip header is used to save the overall information of the strip, such as the type of the current strip, etc. The strip header information can be used to determine which strip it is, so the reference strip can be determined. The reference area can be an area in the image frame or an area in the strip.

[0094] 203. Determine the reference times of one or more reference image frames, reference strips or reference areas by preset parameters, wherein the preset parameters include any one or more of the following: network abstraction layer parsing parameters, strip header parsing parameters, reference image list correction parameters and reference image frame marking parameters.

[0095] For example, in one embodiment, the reference position may include a reference image frame, a reference strip, or a reference area. After determining one or more reference image frames, reference strips, or reference areas from a video code stream, it is necessary to further determine the number of times each reference image frame, reference strip, or reference area is referenced by the object to be decoded, so as to obtain the number of times each reference frame, reference strip, or reference area is read.

[0096] It should be noted that, when determining the reference times of each reference image frame, reference strip or reference area, the reference times of one or more reference image frames, reference strips or reference areas may be determined by preset parameters, and the preset parameters may include any one or more of the following: network abstraction layer parsing parameters, slice header parsing parameters, reference image list modification parameters and reference image frame marking parameters, etc. For example, the network abstraction layer parsing parameters may be the nal_unit() function, the slice header parsing parameters may be the slice_header() function, the reference image list modification parameters may be the ref_pic_list_modification() function, and the reference image frame marking parameters may be the ref_pic_list_modification() function.

[0097] For example, taking H.264 as an example, when performing a rough frame-level analysis, after parsing the NAL unit header information or slice header information contained in multiple image frames, it is possible to distinguish which reference image frames will be referenced multiple times, such as by using the nal_ref_idc variable in the nal_unit() function, the num_ref_idx_active_override_flag variable in the slice_header() function, the ref_pic_list_modification() function, the dec_ref_pic_marking() function and other information to make a preliminary judgment.

[0098] For example, the nal_unit() function analyzes the NAL units starting with 00 00 00 01 and 00 00 01 from the H.264 image frame, and then directly fills in the length of the NAL unit. The nal_ref_idc variable represents the reference level, which represents the reference status of other image frames. The higher the reference level, the more important the reference image frame is.

[0099] The num_ref_idx_active_override_flag variable indicates whether the number of actually available reference frames for the current image frame needs to be overridden. The syntax elements num_ref_idx_l0_active_minus1 and num_ref_idx_l1_active_minus1 that already appear in the image parameter set specify the number of actually available reference frames in the current reference image queue. This pair of syntax elements can be overridden in the slice header to give a greater flexibility to a particular image frame. The position of the slice can be known through the num_ref_idx_active_override_flag variable.

[0100] The ref_pic_list_modification() function is a reference picture list modification function, which can be saved in the structure of the slice header. The definition of the ref_pic_list_modification() function is as follows: when ref_pic_list_modification_flag_l0 is 1, the reference picture list RefPicList0 is modified, and when ref_pic_list_modification_flag_l1 is 1, the reference picture list RefPicList1 is modified. The dec_ref_pic_marking() function is the decoded reference picture frame mark, and the marking operation is used to move the reference picture frame into or out of the reference picture frame queue and specify the symbol of the reference picture.

[0101] 204. If the reference times are multiple, store one or more reference image frames, reference strips or reference areas with multiple reference times in a system cache according to a preset power consumption threshold, and store them in a dynamic random access memory.

[0102] For example, the preset memory may include a first memory and a second memory, and the power consumption of the first memory is greater than that of the second memory. It should be noted that the first memory may include a dynamic random access memory arranged outside the video decoding device, and the second memory may include a system cache arranged outside the video decoding device. After determining the number of times one or more reference image frames, reference strips or reference areas are referenced by the object to be decoded, if the reference number is multiple times, the one or more reference image frames, reference strips or reference areas with the multiple reference times can be stored in the system cache and stored in the dynamic random access memory to wait for reading when the video decoding device decodes.

[0103] It should be noted that, according to the size of the preset power consumption threshold, the size of the data amount of one or more reference image frames, reference strips or reference areas with multiple reference times stored in the system cache can be adjusted.

[0104] It can be understood that when one or more reference image frames, reference strips or reference areas with multiple reference times are stored in the system cache and stored in the dynamic random access memory, the one or more reference image frames, reference strips or reference areas with multiple reference times can be first stored in the system cache and then stored in the dynamic random access memory, or the one or more reference image frames, reference strips or reference areas with multiple reference times can be stored in the dynamic random access memory and the system cache at the same time, or the one or more reference image frames, reference strips or reference areas with multiple reference times can be first stored in the dynamic random access memory and then stored in the system cache.

[0105] It should be noted that the number of references of the reference image frame, reference strip or reference area may be the number of times the reference is made by the same image frame to be decoded, strip or area to be decoded, or the number of times the reference is made by other image frames to be decoded, strips or areas to be decoded in an image group. When the reference image frame, reference strip or reference area with multiple reference times is stored in the system cache and in the first memory, the storage may be performed in units of frames, strips or areas.

[0106] It should be noted that, in the embodiment of the present application, the power consumption of the dynamic random access memory is greater than the power consumption of the system cache, and the power consumption generated when storing and reading the reference image frame, reference stripe or reference area in the dynamic random access memory and the system cache is less than the preset power consumption threshold, which can reduce the power consumption when reading and writing data. Among them, the preset power consumption threshold can be considered as the power consumption generated when the reference image frame, reference stripe or reference area is stored and read by the dynamic random access memory.

[0107] 205. If the reference number is multiple, store one or more reference image frames, reference strips or reference areas having the reference number multiple times in a system buffer memory according to a preset power consumption threshold.

[0108] For example, the first memory may include a dynamic random access memory arranged outside the video decoding device, and the second memory may include a system buffer memory arranged outside the video decoding device. After determining the number of times one or more reference image frames, reference strips or reference areas are referenced by the object to be decoded, if the number of references is multiple times, the one or more reference image frames, reference strips or reference areas with the multiple reference times may be stored in the system buffer memory to wait for reading when the video decoding device is decoding.

[0109] It should be noted that, according to the size of the preset power consumption threshold, the size of the data amount of one or more reference image frames, reference strips or reference areas with multiple reference times stored in the system buffer memory can be adjusted.

[0110] It should be noted that the number of references of a reference image frame, reference strip or reference region may be the number of times the reference is made by the same image frame to be decoded, strip or region to be decoded, or the number of times the reference is made by other image frames to be decoded, strips or regions to be decoded in an image group. When a reference image frame, reference strip or reference region with multiple references is stored in a system buffer memory, it may be stored in units of frames, strips or regions.

[0111] It should be noted that, in the embodiment of the present application, the power consumption of the dynamic random access memory is greater than the power consumption of the system buffer memory, and the power consumption generated when storing and reading the reference image frame, reference strip or reference area in the dynamic random access memory and the system buffer memory is less than the preset power consumption threshold, so that the power consumption when reading and writing data can be reduced. Among them, the preset power consumption threshold can be considered as the power consumption generated when the reference image frame, reference strip or reference area is stored and read by the dynamic random access memory.

[0112] 206. If the reference number is once, store one or more reference image frames, reference strips or reference areas whose reference number is once in a dynamic random access memory according to a preset power consumption threshold.

[0113] For example, the first memory may include a dynamic random access memory arranged outside the video decoding device. After determining the number of times one or more reference image frames, reference strips or reference areas are referenced by the object to be decoded, if the number of references is once, the one or more reference image frames, reference strips or reference areas with the number of references being once can be stored in the dynamic random access memory to wait for reading when the video decoding device decodes.

[0114] It should be noted that the number of references of a reference image frame, reference strip or reference area may be the number of times it is referenced by the same image frame to be decoded, strip or area to be decoded, or the number of times it is referenced by other image frames to be decoded, strips or areas to be decoded in an image group. When a reference image frame, reference strip or reference area with a reference number of one is stored in a dynamic random access memory, it may be stored in units of frames, strips or areas.

[0115] It should be noted that, in an embodiment of the present application, one or more reference image frames, reference strips or reference areas that are referenced once will be read once when they are referenced by the object to be decoded (for example, one or more reference image frames that are referenced once will be read once when they are referenced by the image frame to be decoded; for another example, one or more reference strips that are referenced once will be read once when they are referenced by the strip to be decoded; for another example, one or more reference areas that are referenced once will be read once when they are referenced by the area to be decoded).

[0116] In the embodiment of the present application, the power consumption generated when storing and reading the reference image frame, reference strip or reference area in the dynamic random access memory is less than the preset power consumption threshold, so that the power consumption when reading and writing data can be reduced. The preset power consumption threshold can be considered as the power consumption generated when the reference image frame, reference strip or reference area is stored and read by the dynamic random access memory.

[0117] For example, in one embodiment, the first memory may include a dynamic random access memory disposed outside the video encoder, that is, the first memory may include a DRAM disposed outside the video decoding device, and the second memory may include a system cache or a system buffer memory disposed outside the video decoding device, that is, the second memory may include a Sys$ or a SysBuf disposed outside the video decoding device. Of course, the second memory may also be other low-power memory, etc.

[0118] In the embodiment of the present application, Sys$ or SysBuf is composed of multiple SRAMs, the first memory can be DRAM, the power consumption of DRAM is greater than the power consumption of Sys$ or SysBuf outside the video decoding device, and the power consumption of storing and reading reference image frames, reference strips or reference areas to Sys$ and DRAM is less than a preset power consumption threshold, or the power consumption of storing and reading reference image frames, reference strips or reference areas from SysBuf and DRAM is less than a preset power consumption threshold. In this way, the power consumption when reading and writing data can be reduced, and the preset power consumption threshold can be considered as the power consumption generated when all reference image frames, reference strips or reference areas are stored and read by DRAM.

[0119] See also Figure 8 , Figure 8 It is a comparative schematic diagram of the energy consumed by the static random access memory and the dynamic random access memory when reading data provided by the embodiment of the present application. The energy consumed by reading data in SRAM and reading data in DRAM differs by about 100 times, that is, the power consumption of reading data in SRAM is much less than the power consumption of reading data in DRAM. By storing the reference image frame, reference strip or reference area with multiple reference times in Sys$ and DRAM, or only storing it in SysBuf, when reading the image data of the reference image frame, reference strip or reference area in Sys$ or SysBuf, the power consumption when reading data can be reduced. It can be seen from the above that by storing the reference image frame, reference strip or reference area that is referenced multiple times by the object to be decoded in Sys$ and DRAM, or only storing it in SysBuf, the energy consumption required for the overall video playback can be greatly reduced.

[0120] When decoding, the amount of data that is parsed into and out of high-energy storage (such as DRAM) for video bitstream is limited, so the parsing step does not become an energy bottleneck, whether it is NAL unit parsing and slice header parsing, or entropy decoding (Entropy decoding), the motion vector or other symbols are decoded from the video bitstream. During the decoding process, the part that requires the highest amount of data in and out is the data required by the motion compensation step. The motion compensation step of the video decoding device will require DRAM to provide a larger bandwidth. However, due to the prior analysis of the video bitstream, the reference image frames that are referenced multiple times or the parts of the reference image frames that are used multiple times can be pre-stored in low-energy storage (such as Sys$ or SysBuf composed of SRAM), thereby ensuring that the power consumption of reading data during video decoding is controlled near the expected value, and allowing the hardware or software of the video decoding device to complete the decoding work as soon as possible.

[0121] For example, see Fig. 9 , Fig. 9 It is a schematic diagram of a scenario in which image data that is referenced multiple times is stored in a system cache or a system buffer memory, provided by an embodiment of the present application. Fig. 9 In the present invention, the image data that is referenced by the object to be decoded many times is stored in Sys$ and DRAM, or the image data that is referenced by the object to be decoded many times is stored only in SysBuf, which can greatly reduce the energy consumption required for the overall video playback.

[0122] For example, Figure 6As shown, after the relevant hardware or software of the video decoding device performs a rough frame-level analysis or estimation, it can be determined that some reference image frames will be used multiple times, that is, it can be determined that some reference image frames are referenced multiple times by the image frames to be decoded. The reference image frames with multiple reference times can be stored in the power-saving Sys$ and stored in the DRAM, or the reference image frames with multiple reference times can be stored only in the power-saving SysBuf, so that the video decoding device can maintain the expected low power consumption state as much as possible, improve the use time of the video decoding system and prevent the system from overheating.

[0123] For example, in one embodiment, by storing the entire reference image frame in the video bitstream or the image data in the reference image frame that is referenced multiple times by the object to be decoded in a low-power storage space, such as Sys$ and DRAM, or only in SysBuf, etc., the power consumption of the entire video decoding system during the operation of the video decoding device can be effectively maintained, thereby improving the user experience. The prediction can be made by the hardware or software of the video decoding device to actually analyze the video bitstream, or it can be based on known factors such as application scenarios or group of pictures structure to infer which image data is suitable for writing into low-power storage such as Sys$ or SysBuf.

[0124] For example, assuming that only a rough judgment of the reference image frames is achieved, the reference times of these reference image frames are usually taken as priorities. The more reference times, the higher the priority of the frame stored in Sys$ or SysBuf. For example, the reference image frame with the most reference times is stored in Sys$ or SysBuf first, and so on. The reference image frames with multiple reference times are stored in Sys$ or SysBuf in descending order, so that the power consumption and energy consumption can be reduced to the expected target to the greatest extent during the decoding process. However, since this judgment method is quite rough, it is impossible to achieve 100% close to or below the expected power saving target.

[0125] See also Fig.10 , Fig.10 It is a schematic diagram of a scenario for roughly analyzing the reference relationship between multiple image frames in a video stream provided by an embodiment of the present application. Fig.10 In the example of H.264 video code stream, a rough analysis of H.264 video code stream is performed. By analyzing the header information of the NAL unit or the strip header information, it can be analyzed that some image frames are referenced more times than other image frames. Fig.10By roughly analyzing the code streams corresponding to the 7 image frames, we can know the reference image list and calculate how much DRAM data input and output volume can be saved. Then, select a few image frames that have the highest probability of being satisfied and store them in Sys$ or SysBuf.

[0126] For example, one or more reference image frames, reference strips or reference areas with multiple reference times are stored in Sys$ and stored in DRAM, so that the video decoding device can read from DRAM and Sys$ respectively during decoding.

[0127] It should be noted that if the reference number is one, one or more reference image frames, reference strips or reference areas with the reference number being one can be stored in the DRAM so as to be read during decoding by the video decoding device.

[0128] for example, Fig.10 In the example, the reference count of image frame 0 is once, the reference count of image frame 1 is three times, the reference count of image frame 2 is three times, and the reference count of image frame 3 is once. The reference list includes image frame 0, image frame 1, image frame 2, and image frame 3. Since the reference counts of image frame 1 and image frame 2 are three times, image frame 1 and image frame 2 are stored in Sys$ and also in DRAM, or only image frame 1 and image frame 2 are stored in SysBuf. Image frame 0 and image frame 3 are stored in DRAM.

[0129] For example, in one embodiment, the access power consumption model of DRAM, system bus, Sys$ and SysBuf can be obtained through some simple measurements or experiments, which will not be described in detail here. Assuming that there is already a power consumption model related to data flowing in and out of DRAM, system bus, Sys$ and SysBuf, it can be calculated how much the power consumption or energy consumption is reduced and how much the data access volume of DRAM is reduced. Based on the header information of the bitstream of the image frame or strip, it can be determined which image frames or strips need to be written to Sys$ or SysBuf, so that the expected power consumption reduction value during decoding can be achieved. This rough analysis method can achieve the expected power consumption reduction, but cannot achieve the optimal power consumption reduction.

[0130] For example, if the reference image frames, reference strips or reference regions with multiple reference times are stored in Sys$ and DRAM outside the video decoding device in advance, or stored in SysBuf outside the video decoding device in advance, please refer to Figures 11 to 13 , Fig.11 It is a schematic diagram of the architecture of a video decoding system using a system cache provided in an embodiment of the present application. Fig.12This is another architectural diagram of a video decoding system using a system cache provided in an embodiment of the present application. Fig.13 1 is a schematic diagram of the architecture of a video decoding system using a system buffer memory provided by an embodiment of the present application. Stored in Sys$ or SysBuf are reference image frames, reference strips or reference areas with multiple reference times.

[0131] by Fig.11 For example, Sys$ can read data from DRAM through DramC, and the data read by Sys$ from DRAM through DramC can be read by the central processing unit and the video decoding device. Fig.11 It is only one architecture of the video decoding system when the system cache is used. When the system cache is used, the video decoding system may also adopt other architectures, for example, the video decoding system also includes a display processor, etc. When the video decoding device needs to decode, it can directly read the image data of the reference image frame, reference strip or reference area stored in Sys$ with multiple reference times. In addition, Sys$ also reads the image data of the reference image frame, reference strip or reference area with one reference time from DRAM through DramC, and then is read by the video decoding device.

[0132] Since the access behavior of image data can be predicted during video decoding, the power consumption can be reduced accordingly according to the demand, and the storage method of the measured image data can be intelligently selected, thereby reducing the power consumption of the video decoding device. The location where the image data is stored can be changed according to the frame reference relationship during decoding, so that the number of repeated readings of the reference image frame stored in a low-power memory such as Sys$ is appropriately increased to appropriately reduce power consumption and ensure that the power consumption caused by the data in and out of the video decoding device can always maintain the expected state. If low-power memories such as Sys$ also have high-speed bandwidth, the bandwidth of DRAM can be further reduced.

[0133] It should be noted that Fig.12 and Fig.13 The video decoding system in the example is just one of the architectures. In specific applications, it can be modified accordingly according to actual needs, such as adding a display processor and so on.

[0134] 207. Read the required image data of the reference image frame, reference strip or reference area from the preset memory. If the image data of the reference image frame is read, the block to be decoded in the image frame to be decoded is decoded according to the image data of the reference image frame read. If the image data of the reference strip is read, the block to be decoded in the strip to be decoded is decoded according to the image data of the reference strip read. If the image data of the reference area is read, the block to be decoded in the area to be decoded is decoded according to the image data of the reference area read.

[0135] For example, an image frame, strip or region may be divided into multiple non-overlapping blocks, which form a rectangular array, each of which is a block of N×N pixels, such as a block of 4×4 pixels, a block of 32×32 pixels, a block of 128×128 pixels, and so on.

[0136] When decoding a block to be decoded in an image frame to be decoded, a strip to be decoded, or an area to be decoded, it is necessary to read the image data of the reference image frame, reference strip, or reference area to be referenced from a preset memory. For example, if the block to be decoded needs to refer to the image data of multiple reference image frames, reference strips, or reference areas when decoding, when reading the image data of the multiple reference image frames, reference strips, or reference areas, it can be read once from a first memory (such as DRAM) and read multiple times from a second memory (such as Sys$).

[0137] Since the cost of SRAM is high and the cost of DRAM is low, SRAM is generally not made too large in consideration of cost, while DRAM can be made relatively large. Therefore, in order to reduce the power consumption when reading data, the embodiment of the present application can split the original number of reads from DRAM into several reads from SRAM and several reads from DRAM, which can reduce the power consumption of reading data as a whole. It should be noted that the number of reads from SRAM and the number of reads from DRAM can be adjusted to meet the requirements of different power consumption.

[0138] For example, in one embodiment, when reading the image data of a reference image frame, a reference strip, or a reference area that is required to be referenced multiple times, it can be read from Sys$ first, and when the number of reads is greater than or equal to a preset number threshold, it is switched to reading the unread image data from DRAM. When reading the same image data, the energy consumed by DRAM is greater than 100 times the energy consumed by SRAM. Therefore, by reading part of the image data of the reference image frame, reference strip, or reference area that is required to be referenced multiple times from Sys$ and the other part of the data from DRAM, the power consumption of reading data can be reduced.

[0139] For example, in one embodiment, when reading the image data of a reference image frame, a reference strip, or a reference area that is required to be referenced multiple times, the data is directly read from SysBuf. When reading the same image data, the energy consumed by DRAM is 100 times greater than the energy consumed by SRAM. Therefore, by reading the image data of a reference image frame, a reference strip, or a reference area that is required to be referenced multiple times from SysBuf, the power consumption of reading data can be reduced.

[0140] It should be noted that if the image data read is a reference image frame, the block to be decoded in the image frame to be decoded is decoded according to the image data of the reference image frame read; if the image data read is a reference strip, the block to be decoded in the strip to be decoded is decoded according to the image data of the reference strip read; if the image data read is a reference area, the block to be decoded in the area to be decoded is decoded according to the image data of the reference area read.

[0141] See also Fig.14 , Fig.14 The power consumption curve diagram of reading and writing data from Sys$ or SysBuf provided by the embodiment of the present application is shown in FIG. The video decoding device replaces a large amount of DRAM power consumption with the power consumption of Sys$ or SysBuf, which can greatly reduce power consumption.

[0142] 208. If the reference number of the image frame to be decoded, the strip to be decoded or the area to be decoded is multiple times, the decoded blocks to be decoded are stored in the system cache and in the dynamic random access memory.

[0143] For example, after decoding a block to be decoded in a to-be-decoded image frame, a to-be-decoded strip, or a to-be-decoded area, if the decoded block of the block to be decoded will be subsequently referenced multiple times by other to-be-decoded blocks that need to be decoded (for example, other to-be-decoded blocks in image frames to be decoded, strips to be decoded, or areas to be decoded), the decoded block of the block to be decoded is stored in the system cache and in the dynamic random access memory.

[0144] 209. If the number of references of the image frame to be decoded, the strip to be decoded, or the area to be decoded is multiple times, the decoded blocks to be decoded are stored in the system buffer memory.

[0145] For example, after decoding a block to be decoded in an image frame to be decoded, a strip to be decoded, or a region to be decoded, if the decoded block of the block to be decoded will be subsequently referenced multiple times by other blocks to be decoded that need to be decoded (for example, blocks to be decoded in other image frames to be decoded, strips to be decoded, or regions to be decoded), the decoded block of the block to be decoded is stored in a system buffer memory.

[0146] 210. If the reference number of the image frame to be decoded, the strip to be decoded, or the area to be decoded is one, then the decoded block to be decoded is stored in a dynamic random access memory.

[0147] For example, after decoding a block to be decoded in an image frame to be decoded, a strip to be decoded, or a region to be decoded, if the decoded block of the block to be decoded will be referenced once by other blocks to be decoded that need to be decoded (for example, blocks to be decoded in other image frames to be decoded, strips to be decoded, or regions to be decoded), the decoded block of the block to be decoded is stored in a dynamic random access memory.

[0148] If there are other blocks to be decoded in the image frame to be decoded, the strip to be decoded or the area to be decoded, the other blocks to be decoded are decoded. If all the blocks to be decoded in the image frame to be decoded, the strip to be decoded or the area to be decoded have been decoded, the other image frames, strips or areas are decoded until the decoding of all the image frames, strips or areas to be decoded is completed.

[0149] It is understandable that the embodiments of the present application are based on predictable data access behavior (i.e., repeated reading behavior) during video decoding, thereby realizing intelligent selection of data storage methods to reduce power consumption of the video decoding device. The location where the image data is stored can be changed according to the frame reference relationship during decoding, so that the number of repeated readings of the reference image frame stored in a low-power memory such as Sys$ is appropriately increased to appropriately reduce power consumption and ensure that the power consumption caused by the data in and out of the video decoding device can always maintain the expected state. If a low-power memory such as Sys$ also has a high-speed bandwidth, the bandwidth of the DRAM can be further reduced.

[0150] The embodiments of the present application can ensure that the power consumption of the video decoding device is controllable, and can enable the hardware or software of the video decoding device to complete the decoding work as quickly as possible, and make full use of the predictable behavior of the video decoding device to repeatedly read the reference image frame or reference strip to change the storage characteristics of the read data, because accessing data saves power, and the data in and out will not bring about a power consumption bottleneck, so that the video decoding device can maintain its operating speed while reducing power consumption. The speed of reading data will not be limited by power consumption, so the video decoding device will not overheat. In addition, the SRAM in Sys$ or SysBuf has a low delay when reading and writing, which can increase the processing frame rate and reduce the response delay. Since the power consumption can be greatly reduced, the battery life in the video decoding device can be increased, improving the user experience.

[0151] See also Fig.15 , Fig.15 1 is a third flow chart of a method for performing image processing in a video decoding device provided in an embodiment of the present application. The method for performing image processing in a video decoding device can be applied to a video decoding device. The flow of the method for performing image processing in a video decoding device may include:

[0152] 301. Obtain video code stream.

[0153] The specific implementation of step 301 can refer to the embodiment of step 101, which will not be repeated here.

[0154] 302. Obtain one or more reference motion vectors according to the video code stream.

[0155] For example, by parsing the video stream, one or more reference motion vectors can be obtained from the video stream. Each reference motion vector corresponds to a reference block. When decoding the block to be decoded, the reference block needs to be referenced. The relative displacement between the reference block and the block to be encoded can be used as a reference motion vector. By obtaining the corresponding reference block through the reference motion vector, a refined analysis can be achieved. Each reference block may be referenced once or multiple times by other blocks to be decoded in a group. For example, when a reference block is referenced by only one block to be decoded, the reference block is referenced once. When a reference block is referenced by multiple blocks to be decoded, the reference block is referenced multiple times.

[0156] 303. Obtain one or more corresponding reference blocks from one or more image frames of the current video code stream according to the one or more reference motion vectors.

[0157] For example, after obtaining one or more reference motion vectors according to a video code stream, since each reference motion vector corresponds to a reference block, one or more corresponding reference blocks can be obtained from one or more image frames of the video code stream according to the one or more reference motion vectors.

[0158] 304. Determine the reference count of one or more reference blocks.

[0159] For example, after obtaining one or more corresponding reference blocks from one or more image frames of a video code stream according to one or more reference motion vectors, the number of references of the one or more reference blocks can be determined, that is, the number of times the one or more reference blocks are referenced by the block to be decoded or the sub-block of the block to be decoded can be determined.

[0160] For example, see Fig.16 , Fig.16 It is a scene schematic diagram of the reference relationship between blocks in each image frame in an image group in a video code stream provided by an embodiment of the present application. Fig.16 The example of an image group including 9 image frames is used for explanation. In other implementations, the number of image frames included in the image group can be adjusted according to specific needs. In an image group, the display order of the image frames may be the same as the decoding order, or may be different. Fig.16 The display order and decoding order of image frames in the image group displayed in are different.

[0161] from Fig.16 The direction of the arrow in the figure shows the number of times each block is referenced by other blocks in the group. That is, the number of times each block is referenced by other blocks pointed to by the arrow can be determined according to the direction of the arrow. When a block is referenced by other blocks once or more times, the block can be used as a reference block. For example, Fig.16The reference block in the I frame with a display order of 4 is referenced four times, the reference block in the B frame with a display order of 2 is referenced twice, the reference block in the B frame with a display order of 3 is referenced once, the reference block in the P frame with a display order of 4 is referenced five times, the reference block in the B frame with a display order of 6 is referenced twice, and so on.

[0162] 305. According to a preset power consumption threshold and the number of references of one or more reference blocks, determine one or more reference blocks that need to be stored in a preset memory from one or more reference blocks, and store them in the preset memory, and the power consumption generated by storing the reference blocks in the preset memory and reading the reference blocks from the preset memory is less than or equal to the preset power consumption threshold.

[0163] For example, in an embodiment of the present application, one or more reference blocks that need to be stored in a preset memory can be determined from one or more reference blocks. For example, the reference blocks that need to be stored in the preset memory can be reference blocks that have been referenced multiple times, that is, reference blocks that have been referenced multiple times by the block to be decoded or the sub-block of the block to be decoded are determined to be stored in the preset memory, so as to facilitate reading the image data of the reference block when the block to be decoded or the sub-block of the block to be decoded is subsequently decoded. It should be noted that the power consumption generated by storing the reference blocks in the preset memory and reading the reference blocks from the preset memory is less than or equal to the preset power consumption threshold. By using a preset memory with low power consumption to read and write data, the power consumption of the video decoding device can be reduced.

[0164] 306. Decode the block to be decoded or a sub-block of the block to be decoded according to the reference block.

[0165] For example, the reference block stored in the preset memory may be referenced by the block to be decoded or the sub-block of the block to be decoded multiple times, so when decoding, the reference block stored in the preset memory will be read multiple times. After the image data of the reference block is read from the preset memory, the block to be decoded or the sub-block of the block to be decoded can refer to the image data of the reference block, that is, the block to be decoded or the sub-block of the block to be decoded can be decoded according to the read image data of the reference block.

[0166] It can be understood that in the embodiment of the present application, the video decoding device can obtain a video code stream and obtain one or more reference motion vectors according to the video code stream. Then, according to the one or more reference motion vectors, obtain the corresponding one or more reference blocks from one or more image frames of the video code stream; determine the reference times of one or more reference blocks; according to the preset power consumption threshold and the reference times of one or more reference blocks, determine one or more reference blocks that need to be stored in the preset memory from the one or more reference blocks, and store them in the preset memory, and the power consumption of storing the reference blocks in the preset memory and reading the reference blocks from the preset memory is less than or equal to the preset power consumption threshold. Afterwards, decode the block to be decoded or the sub-block of the block to be decoded according to the reference block. That is, in the embodiment of the present application, the image data of the reference image frame or reference strip that needs to be stored in the preset memory is stored in the preset memory with lower power consumption, so as to achieve the purpose of reducing the power consumption of the video decoding device. Therefore, the embodiment of the present application can reduce the power consumption of the video decoding device.

[0167] See also Fig.17 , Fig.17 4 is a schematic diagram of a fourth flow chart of a method for performing image processing in a video decoding device provided in an embodiment of the present application. The method for performing image processing in a video decoding device can be applied to a video decoding device. The flow chart of the method for performing image processing in a video decoding device may include:

[0168] 401. Get the video stream.

[0169] The specific implementation of step 401 can refer to the embodiment of step 101, which will not be described in detail here.

[0170] 402. Perform entropy decoding on the video bitstream to obtain one or more motion vector differences (Motion Vector Difference, MVD).

[0171] For example, after obtaining the video bitstream, the video bitstream can be entropy decoded, such as decoding the frame header information of the image frame, the header information of the NAL unit or the slice header. After entropy decoding, one or more motion vector differences can be obtained, and the quantized residual can also be obtained. The residual refers to the difference between the block to be encoded and one or more blocks with the minimum encoding cost.

[0172] For example, in one implementation, entropy decoding the video bitstream in 402 to obtain one or more motion vector differences may include:

[0173] The video code stream is entropy decoded to obtain one or more motion vector differences and a quantized first residual.

[0174] For example, by entropy decoding the video bitstream, one or more motion vector differences can be obtained, and a quantized first residual can also be obtained. The quantized first residual refers to the first residual obtained after forward transforming and quantizing the residual during encoding, wherein the residual can be the difference obtained by subtracting the two-dimensional pixel of the searched block from the two-dimensional pixel of the block to be encoded.

[0175] 403. Obtain one or more reference motion vectors according to one or more motion vector differences and corresponding motion vector prediction values.

[0176] For example, after entropy decoding the video bitstream to obtain one or more motion vector differences, one or more reference motion vectors can be obtained based on the one or more motion vector differences and the corresponding motion vector prediction values, such as taking the sum of the motion vector difference and the motion vector prediction value as the reference motion vector.

[0177] 404. Obtain one or more corresponding reference blocks from one or more image frames of the video code stream according to the one or more reference motion vectors.

[0178] For example, each reference motion vector corresponds to a reference block. Therefore, based on one or more reference motion vectors, one or more corresponding reference blocks can be determined from one or more image frames of the video code stream, thereby obtaining the corresponding reference block, wherein the reference block is a decoded block.

[0179] 405. Determine the reference times of one or more reference blocks.

[0180] For example, after obtaining the reference blocks in the image frame, the reference times of one or more reference blocks may be determined, where the reference times refers to the number of times the reference blocks are referenced by the block to be decoded or a sub-block in the block to be decoded.

[0181] for example, Fig.16 After fine block-level analysis or estimation by relevant hardware or software of the video decoding device, it can be determined that certain areas of certain image frames will be referenced multiple times by certain areas of other image frames during decoding, or certain blocks of certain image frames will be referenced multiple times by certain blocks of other image frames during decoding, which are suitable for storage in power-saving low-power memories such as Sys$ and DRAM, or directly stored in low-power memories such as SysBuf, so that the data throughput during video decoding is maintained at the expected low power consumption value as much as possible, thereby improving the use time of the video decoding system and preventing the video decoding system from overheating.

[0182] For example, if the video code stream is analyzed more deeply, the motion vector difference obtained by entropy decoding in the decoding process is first taken out to restore the reference motion vector, and a detailed reference motion vector analysis can be performed, which reference image frames and areas in the adjacent image frames are used more deeply, and the number of times certain image frame areas are used as reference areas can be more accurately distinguished, which can be used as the basis for priority sorting in the low-energy memory. For example, please refer to Fig.18 , Fig.18 It is a schematic diagram of a scenario for finely analyzing the reference relationship of blocks in multiple image frames in a video code stream provided by an embodiment of the present application. Fig.18 In this paper, H.264 video code stream is used as an example to perform a detailed analysis of the H.264 video code stream. By analyzing the main body of the strip, it can be found that some blocks are referenced more times than other blocks. Fig.18 The bitstreams corresponding to the five image frames in the video are analyzed in detail. Each part of each image frame (such as each macroblock, also called block) can be determined by referring to the motion vector analysis to determine how many times it will be referenced by other blocks.

[0183] like Fig.18 , block 0, block 2, and block C in image frame 0 are all referenced once, block 8 in image frame 0 is referenced twice, block 6 in image frame 1 is referenced twice, block 4 in image frame 1 is referenced three times, block 9 in image frame 1 is referenced four times, block 5, block A, and block D in image frame 1 are all referenced once, and block 7, block 3, and block B in image frame 2 are all referenced once.

[0184] 406. If the reference times are multiple, store one or more reference blocks with multiple reference times in the system cache and in the dynamic random access memory according to a preset power consumption threshold.

[0185] For example, the preset memory may include a first memory and a second memory, and the power consumption of the first memory is greater than the power consumption of the second memory. The first memory may include a dynamic random access memory arranged outside the video decoding device, and the second memory may include a system cache arranged outside the video decoding device. After determining the number of references of one or more reference blocks, that is, determining the number of times one or more reference blocks are referenced by the block to be decoded or the sub-blocks in the block to be decoded, if the number of references is multiple times, the one or more reference blocks with multiple reference times may be stored in the system cache and in the dynamic random access memory.

[0186] For example, when it is expected that how much power consumption reduction is needed to make the video decoding system work, it is possible to infer how much the DRAM bandwidth should be reduced. The DRAM bandwidth that needs to be reduced can be calculated in advance for each image frame. By pre-calculating which areas (reference blocks) in the image frame need to be stored in Sys$, the data in and out limit of the DRAM can be met or lowered as much as possible during subsequent motion compensation, which is derived from how much power consumption or energy consumption needs to be reduced.

[0187] It should be noted that, in the embodiment of the present application, the power consumption of the dynamic random access memory is greater than the power consumption of the system cache. After obtaining one or more reference blocks with multiple reference times, that is, obtaining one or more reference blocks that are referenced multiple times by the block to be decoded or the sub-blocks in the block to be decoded, they can be stored in the system cache and in the dynamic random access memory.

[0188] like Fig.18 As shown, since the reference times of block 8 in image frame 0 is two times, the reference times of block 6 in image frame 1 is two times, the reference times of block 4 in image frame 1 is three times, and the reference times of block 9 in image frame 1 is four times, block 8 in image frame 0, block 6 in image frame 1, block 4 in image frame 1, and block 9 in image frame 1 are stored in the system cache and in the dynamic random access memory. It should be noted that the power consumption generated by storing and reading one or more reference blocks with multiple reference times in the dynamic random access memory and the system cache is less than or equal to the preset power consumption threshold, which can reduce the power consumption when reading data.

[0189] 407 . If the reference times are multiple, store one or more reference blocks having multiple reference times in a system buffer memory according to a preset power consumption threshold.

[0190] For example, the preset memory may include a second memory, and the second memory includes a system buffer memory disposed outside the video decoding device. After determining the number of reference times of one or more reference blocks, that is, determining the number of times that one or more reference blocks are referenced by the block to be decoded or the sub-block in the block to be decoded, if the number of reference times is multiple times, the one or more reference blocks with multiple reference times are stored in the system buffer memory according to the preset power consumption threshold.

[0191] For example, when it is expected that how much power consumption can be reduced to make the video decoding system work, it can be inferred how much the bandwidth of the dynamic random access memory should be reduced. The bandwidth of the dynamic random access memory that needs to be reduced can be calculated in advance for each image frame. By pre-calculating which areas (reference blocks) in the image frame need to be stored in SysBuf, the data in and out limit of the first memory can be met or lowered as much as possible during subsequent motion compensation, which is derived from how much power consumption or energy consumption needs to be reduced.

[0192] 408. If the reference number is one, store one or more reference blocks with the reference number being one in a dynamic random access memory according to a preset power consumption threshold.

[0193] For example, the preset memory may include a first memory, and the first memory may include a dynamic random access memory arranged outside the video decoding device. After determining the number of references of one or more reference blocks, that is, determining the number of times one or more reference blocks are referenced by the block to be decoded or the sub-block in the block to be decoded, if the number of references is once, the one or more reference blocks with the reference number of once are stored in the dynamic random access memory according to the preset power consumption threshold, so as to facilitate reading during decoding by the video decoding device. By storing reference blocks with multiple reference times in a low power consumption memory and storing reference blocks with the reference number of once in a dynamic random access memory, the power consumption generated when reading data can be reduced as a whole.

[0194] For example, the first memory may include a DRAM disposed outside the video decoding device, and the second memory may include a system cache or a system buffer memory disposed outside the video decoding device, that is, the second memory may include a Sys$ or a SysBuf disposed outside the video decoding device. Of course, the second memory may also be other low-power memory, etc. The energy consumption of DRAM is greater than that of Sys$ or SysBuf. Assuming that there is a power consumption model related to data flowing in and out of DRAM, system bus, Sys$ and SysBuf, it can be inferred how much the power consumption or energy consumption is reduced and how much the data access volume of DRAM is reduced. Based on the header information of the code stream of the image frame or strip, it can be determined which blocks need to be written to Sys$ or SysBuf, so that the expected power consumption reduction value during decoding can be achieved.

[0195] The reference motion vector information decoded from the bitstream details is used to determine which blocks are suitable for being written into Sys$ or SysBuf. Generally, the more times a reference block is referenced by the block to be decoded or the sub-block in the block to be decoded, the more suitable it is for being written into Sys$ or SysBuf, and a smaller Sys$ or SysBuf occupancy can be used to achieve a greater reduction in decoding power consumption. This detailed analysis method only seeks to achieve the expected power consumption / energy consumption reduction, not the optimal power consumption / energy consumption reduction. As long as the amount of data in and out reduction that will occur when the reconstruction area / region is stored in Sys$ or SysBuf is calculated, the amount of power consumption / energy consumption saved can be inferred.

[0196] See also Figure 8 , the energy difference between reading SRAM and reading DRAM is about 100 times, that is, the energy of reading SRAM is much less than that of reading DRAM. By storing the reference block with multiple reference times in Sys$ and DRAM, or storing the reference block with multiple reference times in SysBuf (Sys$ or SysBuf is composed of multiple SRAMs), when reading the image data of the reference block from Sys$ and DRAM, or from SysBuf, the power consumption when reading data can be reduced as a whole.

[0197] 409 . Read the required image data of the reference block from the preset memory, and decode the block to be decoded or the sub-block in the block to be decoded according to the read image data of the reference block.

[0198] For example, when decoding a block to be decoded or a sub-block in a block to be decoded, it is necessary to refer to the image data of the reference block, and the image data of the reference block must be read. Among them, a block may include multiple sub-blocks, and the multiple sub-blocks are arranged in a rectangular array. For example, when reading the image data of the reference block, if the block to be decoded or the sub-block in the block to be decoded needs to refer to the reference block once, the image data of the reference block is directly read from the first memory, if the block to be decoded or the sub-block in the block to be decoded needs to refer to the reference block multiple times, it can be read from the first memory once, and the remaining times are read from Sys$, or, if the block to be decoded or the sub-block in the block to be decoded needs to refer to the reference block multiple times, it can be read from SysBuf.

[0199] It can be understood that, for example, when the second memory is Sys$, the number of times read from Sys$ can be greater than the number of times read from DRAM, the number of times read from Sys$ can be less than the number of times read from DRAM, or the number of times read from Sys$ can be equal to the number of times read from DRAM. The specific number of times to read from DRAM and Sys$ should be set accordingly according to the specific scenario, and the embodiments of the present application do not impose specific restrictions on this. Therefore, by reading part of the image data of the reference block from Sys$ and the other part of the data from DRAM, the power consumption of reading data can be reduced. Fig.14 It can be seen that the video decoding device replaces a large amount of DRAM power consumption with the power consumption of Sys$ or SysBuf, greatly reducing power consumption.

[0200] For example, in one implementation, decoding the block to be decoded or a sub-block in the block to be decoded according to the read image data of the reference block in 409 may include:

[0201] Dequantizing and inverse transforming the first residual to obtain a second residual;

[0202] Obtaining a prediction value of the block to be decoded or a sub-block in the block to be decoded according to the reference motion vector and the reference block;

[0203] A decoded block of the block to be decoded or a decoded subblock of the block to be decoded is obtained according to the second residual and the prediction value of the block to be decoded or the subblock in the block to be decoded.

[0204] For example, in one implementation, decoding the block to be decoded or a sub-block in the block to be decoded according to the read image data of the reference block in 409 may also include:

[0205] The video stream decoding data is obtained according to the decoded block of the to-be-decoded block or the decoded sub-block of the sub-block in the to-be-decoded block.

[0206] For example, see Fig.19 , Fig.19: This is a schematic diagram of a decoding scene of a video decoding device provided by an embodiment of the present application. The video code stream is illustrated by taking the video code stream of H.264 as an example. After entropy decoding of the video code stream, one or more motion vector differences and a quantized first residual are obtained. According to the motion vector difference and the corresponding motion vector prediction value, a reference motion vector can be obtained, so that the reference block used for motion compensation can be more precisely known. Among them, entropy decoding can be implemented by an independent hardware design or by software. The parsing of the current video code stream and image buffering can be implemented by software through a driver or an open multimedia acceleration layer framework (Open MediaAcceleration, OpenMAX). After the first residual is inversely quantized and inversely transformed, a second residual can be obtained. According to the reference motion vector (the relative displacement between the reference block and the block to be decoded or the sub-block in the block to be decoded) and the reference block, the predicted value of the block to be decoded or the sub-block in the block to be decoded can be obtained. It should be noted that the predicted value of the block to be decoded or the sub-block in the block to be decoded can be obtained by intra-frame prediction or motion compensation. Inverse quantization and inverse transformation, intra / inter mode selection, intra prediction, motion compensation and deblocking filtering in the decoding process can be implemented through an application specific integrated circuit (ASIC).

[0207] After obtaining the predicted value of the block to be decoded or the sub-block in the block to be decoded, the second residual is added to the predicted value of the block to be decoded or the sub-block in the block to be decoded to obtain a decoded block of the block to be decoded or a decoded sub-block of the sub-block in the block to be decoded (actual value), and after filtering by a blocking effect filter according to the decoded block of the block to be decoded or the decoded sub-block of the sub-block in the block to be decoded, smooth video stream decoded data can be obtained.

[0208] 410. If the reference number of the block to be decoded or the sub-block in the block to be decoded is multiple times, store the decoded block of the block to be decoded or the decoded sub-block in the block to be decoded in the system cache and in the dynamic random access memory.

[0209] For example, after a block to be decoded or a sub-block in a block to be decoded is decoded, if the decoded block of the block to be decoded will be subsequently referenced multiple times by other blocks to be decoded, or the decoded sub-block of a sub-block in the block to be decoded will be subsequently referenced multiple times by other sub-blocks in the blocks to be decoded, then the decoded block of the block to be decoded or the decoded sub-block of the sub-block in the block to be decoded is stored in the system cache and in the dynamic random access memory as a reference block when decoding other blocks to be decoded or sub-blocks in the block to be decoded.

[0210] 411. If the reference times of the block to be decoded or the sub-block in the block to be decoded are multiple times, store the decoded block of the block to be decoded or the decoded sub-block of the sub-block in the block to be decoded in the system buffer memory.

[0211] For example, after the block to be decoded or a sub-block in the block to be decoded is decoded, if the decoded block of the block to be decoded will be referenced multiple times by other blocks to be decoded, or the decoded sub-block of the sub-block in the block to be decoded will be referenced multiple times by other sub-blocks in the block to be decoded, then the decoded block of the block to be decoded or the decoded sub-block of the sub-block in the block to be decoded is stored in the system buffer memory as a reference block when decoding other blocks to be decoded or sub-blocks in the block to be decoded.

[0212] 412. If the reference number of the block to be decoded or the sub-block in the block to be decoded is one, store the decoded block of the block to be decoded or the decoded sub-block of the sub-block in the block to be decoded in a dynamic random access memory.

[0213] For example, after the block to be decoded or a sub-block in the block to be decoded is decoded, if the decoded block of the block to be decoded will be referenced once by other blocks to be decoded later, or the decoded sub-block of the sub-block in the block to be decoded will be referenced once by other sub-blocks in the block to be decoded later, then the decoded block of the block to be decoded or the decoded sub-block of the sub-block in the block to be decoded is stored in the dynamic random access memory as a reference block when decoding other blocks to be decoded or sub-blocks in the block to be decoded.

[0214] If there are other blocks to be decoded or sub-blocks of other blocks to be decoded in the image frame to be decoded, the strip to be decoded or the area to be decoded that need to be decoded, then decode the other blocks to be decoded or sub-blocks of other blocks to be decoded. If all blocks to be decoded or sub-blocks of blocks to be decoded in the image frame to be decoded, the strip to be decoded or the area to be decoded have been decoded, then decode the other image frames, strips or areas until the decoding of all image frames, strips or areas that need to be decoded is completed.

[0215] It should be noted that Figure 7 The process and method of performing image processing in a video decoding device Fig.17 The processes of the method for performing image processing in a video decoding device can be combined in one system without interfering with each other, that is, when decoding a video code stream, it is possible to switch to another process in the middle. A more reasonable switching point is when a new image frame or strip starts to be decoded.

[0216] It is understandable that the embodiments of the present application are based on predictable data access behavior (i.e., repeated reading behavior) during video decoding, thereby realizing intelligent selection of data storage methods to reduce power consumption of the video decoding device. The location where the image data is stored can be changed according to the frame reference relationship during decoding, so that the number of repeated readings of the reference image frame stored in a low-power memory such as Sys$ or SysBuf is appropriately increased to appropriately reduce power consumption and ensure that the power consumption caused by the data in and out of the video decoding device can always maintain the expected state. If a low-power memory such as Sys$ also has a high-speed bandwidth, the bandwidth of the DRAM can be further reduced.

[0217] The embodiments of the present application can ensure that the power consumption of the video decoding device is controllable, and can enable the hardware or software of the video decoding device to complete the decoding work as quickly as possible, and make full use of the predictable behavior of the video decoding device to repeatedly read the reference block to change the storage characteristics of the read data, because accessing data saves power, and the data in and out will not bring about a power consumption bottleneck, so that the video decoding device can maintain its operating speed while reducing power consumption. The speed of reading data will not be limited by power consumption, so the video decoding device will not overheat. In addition, the delay of the SRAM in Sys$ or SysBuf itself is low when reading and writing, which can increase the processing frame rate and reduce the response delay. Since the power consumption can be greatly reduced, the battery life of the video decoding device can be increased, improving the user experience.

[0218] It is understandable that the embodiments of the present application can select the target location or attribute of data reading according to the long-term playback requirements and the large power consumption caused by the predictable behavior of the playback device. For example, the data that needs to be read repeatedly is read from Sys$ and DRAM, or from SysBuf, instead of all from DRAM. Since the power consumption of SRAM is much less than that of DRAM when reading the same data, the embodiments of the present application can greatly reduce the power consumption when reading data.

[0219] The embodiment of the present application uses video decoding as an example to explain in detail how to reduce the power consumption of reading data. In other embodiments, it can also be applied to all modules and applications that require high bandwidth but have predictable data access behaviors, such as video encoding devices, frame rate up conversion devices, etc. The behaviors of these modules and applications are usually predictable, such as the number of repeated reads. Through these predictable behaviors, the corresponding storage characteristics can be pre-allocated, that is, the repeatedly read data is stored in a low-power memory, for example, according to the number of access requirements of the image data of all frames or part of the frames, to correspond to the energy consumption of different levels of memory, that is, according to the number of access requirements of the image data of all image frames or part of the image frames, to select the corresponding energy consumption of different levels. When the energy consumption is different, the number of times to read data from Sys$ and DRAM can be reasonably allocated, or the number of times to read data from SysBuf can be reasonably allocated.

[0220] For example, a video encoding device can determine the access behavior of data by analyzing the video code stream in advance, and a frame rate enhancement device can know which areas will be used multiple times during processing through simple analysis, etc. It can also be applied to fixed artificial intelligence (AI) network behavior. The part that AI network behavior repeatedly reads is the feature map part, and the AI ​​network behavior is predictable.

[0221] See also Fig. 20 , Fig. 20 The image processing device 500 may include: an acquisition module 501 , a first determination module 502 , a second determination module 503 , a third determination module 504 , and a decoding module 505 .

[0222] The acquisition module 501 is used to acquire the video code stream;

[0223] A first determination module 502, configured to determine one or more reference positions from the video bitstream;

[0224] A second determining module 503, configured to determine a reference number of the one or more reference positions;

[0225] A third determining module 504 is configured to determine a reference position to be stored in a preset memory from the one or more reference positions according to a preset power consumption threshold and the reference number of the one or more reference positions, and store the reference position in the preset memory, wherein the power consumption generated by storing the reference position in the preset memory and reading the reference position from the preset memory is less than or equal to the preset power consumption threshold; and

[0226] The decoding module 505 is configured to decode the object to be decoded according to the reference position.

[0227] In one implementation, the reference position includes a reference image frame, a reference strip, or a reference area, and the first determination module 502 may be used to:

[0228] One or more reference image frames, reference strips or reference areas are determined from the video code stream according to frame header information of an image frame in the video code stream or strip header information of one or more strips in the image frame.

[0229] In one implementation, the reference position includes a reference image frame, a reference strip, or a reference area, and the second determination module 503 may be used to:

[0230] The reference times of the one or more reference image frames, reference strips or reference areas are determined by preset parameters, wherein the preset parameters include any one or more of the following: network abstraction layer parsing parameters, strip header parsing parameters, reference image list correction parameters and reference image frame marking parameters.

[0231] In one implementation, the preset memory includes a first memory and a second memory, and power consumption of the first memory is greater than power consumption of the second memory.

[0232] In one implementation, the first memory includes a dynamic random access memory disposed outside the video decoding device, the second memory includes a system cache disposed outside the video decoding device, the reference location includes a reference image frame, a reference stripe, or a reference area, and the third determination module 504 may be used to:

[0233] If the reference times are multiple, one or more reference image frames, reference strips or reference areas having multiple reference times are stored in the system cache and in the dynamic random access memory according to the preset power consumption threshold.

[0234] In one implementation, the second memory includes a system buffer memory disposed outside the video decoding device, the reference position includes a reference image frame, a reference strip, or a reference area, and the third determination module 504 may be configured to:

[0235] If the reference number is multiple, one or more reference image frames, reference strips or reference areas having the reference number is multiple are stored in the system buffer memory according to the preset power consumption threshold.

[0236] In one implementation, the first memory includes a dynamic random access memory disposed outside the video decoding device, the reference position includes a reference image frame, a reference strip, or a reference area, and the third determination module 504 may be used to:

[0237] If the reference number is once, one or more reference image frames, reference strips or reference areas with the reference number being once are stored in the dynamic random access memory according to the preset power consumption threshold.

[0238] In one implementation, the object to be decoded includes a block to be decoded in a to-be-decoded image frame, a block to be decoded in a to-be-decoded strip, or a block to be decoded in a to-be-decoded area, and the decoding module 505 may be used to:

[0239] Reading the image data of the required reference image frame, reference strip or reference area from the preset memory, if the image data of the reference image frame is read, decoding the block to be decoded in the image frame to be decoded according to the image data of the reference image frame read, if the image data of the reference strip is read, decoding the block to be decoded in the strip to be decoded according to the image data of the reference strip read, if the image data of the reference area is read, decoding the block to be decoded in the area to be decoded according to the image data of the reference area read;

[0240] If the reference times of the image frame to be decoded, the strip to be decoded or the area to be decoded are multiple times, the decoded blocks of the blocks to be decoded are stored in the system cache and in the dynamic random access memory.

[0241] In one implementation, the object to be decoded includes a block to be decoded in a to-be-decoded image frame, a block to be decoded in a to-be-decoded strip, or a block to be decoded in a to-be-decoded area, and the decoding module 505 may be used to:

[0242] Reading the image data of the required reference image frame, reference strip or reference area from the preset memory, if the image data of the reference image frame is read, decoding the block to be decoded in the image frame to be decoded according to the image data of the reference image frame read, if the image data of the reference strip is read, decoding the block to be decoded in the strip to be decoded according to the image data of the reference strip read, if the image data of the reference area is read, decoding the block to be decoded in the area to be decoded according to the image data of the reference area read;

[0243] If the reference times of the image frame to be decoded, the strip to be decoded or the area to be decoded are multiple times, the decoded blocks of the blocks to be decoded are stored in the system buffer memory.

[0244] In one implementation, the object to be decoded includes a block to be decoded in a to-be-decoded image frame, a block to be decoded in a to-be-decoded strip, or a block to be decoded in a to-be-decoded area, and the decoding module 505 may be used to:

[0245] Reading the image data of the required reference image frame, reference strip or reference area from the preset memory, if the image data of the reference image frame is read, decoding the block to be decoded in the image frame to be decoded according to the image data of the reference image frame read, if the image data of the reference strip is read, decoding the block to be decoded in the strip to be decoded according to the image data of the reference strip read, if the image data of the reference area is read, decoding the block to be decoded in the area to be decoded according to the image data of the reference area read;

[0246] If the reference number of the image frame to be decoded, the strip to be decoded or the area to be decoded is one, the decoded block of the block to be decoded is stored in the dynamic random access memory.

[0247] See also Fig.21 , Fig.21 Another structural diagram of an image processing device provided in an embodiment of the present application: The image processing device 600 may include: a first acquisition module 601 , a second acquisition module 602 , a third acquisition module 603 , a first determination module 604 , a second determination module 605 , and a decoding module 606 .

[0248] The first acquisition module 601 is used to acquire a video code stream;

[0249] A second acquisition module 602, configured to acquire one or more reference motion vectors according to the video bit stream;

[0250] A third acquisition module 603 is configured to acquire one or more corresponding reference blocks from one or more image frames of the video code stream according to the one or more reference motion vectors;

[0251] A first determining module 604 is configured to determine a reference number of the one or more reference blocks;

[0252] A second determining module 605 is configured to determine one or more reference blocks to be stored in a preset memory from the one or more reference blocks according to a preset power consumption threshold and the number of references of the one or more reference blocks, and store the one or more reference blocks in the preset memory, wherein the power consumption generated by storing the reference blocks in the preset memory and reading the reference blocks from the preset memory is less than or equal to the preset power consumption threshold; and

[0253] The decoding module 606 is configured to decode the block to be decoded or a sub-block of the block to be decoded according to the reference block.

[0254] In one implementation, the second acquisition module 602 may be used to:

[0255] Performing entropy decoding on the video code stream to obtain one or more motion vector differences;

[0256] The one or more reference motion vectors are obtained according to the one or more motion vector differences and corresponding motion vector prediction values.

[0257] In one implementation, the preset memory includes a first memory and a second memory, and power consumption of the first memory is greater than power consumption of the second memory.

[0258] In one implementation, the first memory includes a dynamic random access memory disposed outside the video decoding device, the second memory includes a system cache disposed outside the video decoding device, and the second determining module 605 may be configured to:

[0259] If the reference times are multiple times, one or more reference blocks with the reference times being multiple times are stored in the system cache and in the dynamic random access memory according to the preset power consumption threshold.

[0260] In one implementation, the second memory includes a system buffer memory disposed outside the video decoding device, and the second determining module 605 may be configured to:

[0261] If the reference times are multiple times, one or more reference blocks having the reference times are stored in the system buffer memory according to the preset power consumption threshold.

[0262] In one implementation, the first memory includes a dynamic random access memory disposed outside the video decoding device, and the second determining module 605 may be configured to:

[0263] If the reference number is one, one or more reference blocks with the reference number being one are stored in the dynamic random access memory according to the preset power consumption threshold.

[0264] In one implementation, the decoding module 606 may be configured to:

[0265] Reading required image data of a reference block from the preset memory, and decoding the block to be decoded or a sub-block in the block to be decoded according to the read image data of the reference block;

[0266] If the reference times of the block to be decoded or the sub-block in the block to be decoded are multiple times, the decoded block of the block to be decoded or the decoded sub-block in the block to be decoded is stored in the system cache and in the dynamic random access memory.

[0267] In one implementation, the decoding module 606 may be configured to:

[0268] Reading required image data of a reference block from the preset memory, and decoding the block to be decoded or a sub-block in the block to be decoded according to the read image data of the reference block;

[0269] If the reference times of the block to be decoded or the sub-block in the block to be decoded are multiple times, the decoded block of the block to be decoded or the decoded sub-block in the block to be decoded is stored in the system buffer memory.

[0270] In one implementation, the decoding module 606 may be configured to:

[0271] Reading required image data of a reference block from the preset memory, and decoding the block to be decoded or a sub-block in the block to be decoded according to the read image data of the reference block;

[0272] If the reference number of the block to be decoded or the sub-block in the block to be decoded is one, the decoded block of the block to be decoded or the decoded sub-block in the block to be decoded is stored in the dynamic random access memory.

[0273] In one implementation, the second acquisition module 602 may be used to:

[0274] Performing entropy decoding on the current video bitstream to obtain one or more motion vector differences and a quantized first residual;

[0275] The decoding module 606 may be used to:

[0276] Dequantizing and inverse transforming the first residual to obtain a second residual;

[0277] Obtaining a prediction value of the block to be decoded or a sub-block in the block to be decoded according to the reference motion vector and the reference block;

[0278] A decoded block of the block to be decoded or a decoded subblock of the block to be decoded is obtained according to the second residual and the prediction value of the block to be decoded or the subblock in the block to be decoded.

[0279] In one implementation, the decoding module 606 may be configured to:

[0280] The video stream decoding data is obtained according to the decoded block of the to-be-decoded block or the decoded sub-block of the sub-block in the to-be-decoded block.

[0281] In one implementation, the prediction value of the block to be decoded or the sub-block in the block to be decoded is obtained by intra-frame prediction or motion compensation.

[0282] An embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed on a computer, the computer executes the process in the method for performing image processing in a video decoding device as provided in the present embodiment.

[0283] An embodiment of the present application also provides an electronic device, including a memory, a processor and a video decoding device, wherein the processor is used to execute the process of the method for performing image processing in the video decoding device provided in this embodiment by calling a computer program stored in the memory.

[0284] For example, the electronic device may be a mobile terminal such as a tablet computer or a smart phone. Fig. 22 , Fig. 22 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0285] The electronic device 700 may include components such as a video decoding device 701, a memory 702, and a processor 703. Those skilled in the art will appreciate that Fig. 22 The electronic device structure shown in the figure does not constitute a limitation of the electronic device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0286] The video decoding device 701 may be used to decode the encoded video image to restore the original video image.

[0287] The memory 702 may be used to store applications and data. The applications stored in the memory 702 include executable codes. The applications may be composed of various functional modules. The processor 703 executes various functional applications and data processing by running the applications stored in the memory 702.

[0288] The processor 703 is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device. It executes various functions of the electronic device and processes data by running or executing applications stored in the memory 702 and calling data stored in the memory 702, thereby monitoring the electronic device as a whole.

[0289] In this embodiment, the processor 703 in the electronic device will load the executable code corresponding to the process of one or more application programs into the memory 702 according to the following instructions, and the processor 703 will run the application program stored in the memory 702 to execute:

[0290] Get the video stream;

[0291] Determining one or more reference positions from the video stream;

[0292] determining a reference number of the one or more reference positions;

[0293] According to a preset power consumption threshold and the reference number of the one or more reference positions, determining a reference position to be stored in a preset memory from the one or more reference positions, and storing the reference position in the preset memory, wherein the power consumption generated by storing the reference position in the preset memory and reading the reference position from the preset memory is less than or equal to the preset power consumption threshold; and

[0294] Decoding the object to be decoded according to the reference position; or executing:

[0295] Get the video stream;

[0296] Acquire one or more reference motion vectors according to the video code stream;

[0297] Acquire one or more corresponding reference blocks from one or more image frames of the video code stream according to the one or more reference motion vectors;

[0298] determining a reference number of the one or more reference blocks;

[0299] According to a preset power consumption threshold and the number of references of the one or more reference blocks, determine one or more reference blocks to be stored in a preset memory from the one or more reference blocks, and store them in the preset memory, wherein power consumption generated by storing the reference blocks in the preset memory and reading the reference blocks from the preset memory is less than or equal to the preset power consumption threshold; and

[0300] The block to be decoded or a sub-block of the block to be decoded is decoded according to the reference block.

[0301] See also Fig.23 The electronic device 700 may include a video decoding device 701, a memory 702, a processor 703, a battery 704, an input unit 705, an output unit 706 and other components.

[0302] The video decoding device 701 may be used to decode the encoded video image to restore the original video image.

[0303] The memory 702 may be used to store applications and data. The applications stored in the memory 702 include executable codes. The applications may be composed of various functional modules. The processor 703 executes various functional applications and data processing by running the applications stored in the memory 702.

[0304] The processor 703 is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device. It executes various functions of the electronic device and processes data by running or executing applications stored in the memory 702 and calling data stored in the memory 702, thereby monitoring the electronic device as a whole.

[0305] The battery 704 can be used to provide power support for various components of the electronic device, thereby ensuring the normal operation of each component.

[0306] The input unit 705 may be used to receive an encoded input video stream of a video image, for example, may be used to receive a video stream that needs to be decoded.

[0307] The output unit 706 may be configured to output the decoded video stream.

[0308] In this embodiment, the processor 703 in the electronic device will load the executable code corresponding to the process of one or more application programs into the memory 702 according to the following instructions, and the processor 703 will run the application program stored in the memory 702 to execute:

[0309] Get the video stream;

[0310] Determining one or more reference positions from the video stream;

[0311] determining a reference number of the one or more reference positions;

[0312] According to a preset power consumption threshold and the reference number of the one or more reference positions, determining a reference position to be stored in a preset memory from the one or more reference positions, and storing the reference position in the preset memory, wherein the power consumption generated by storing the reference position in the preset memory and reading the reference position from the preset memory is less than or equal to the preset power consumption threshold; and

[0313] Decoding the object to be decoded according to the reference position; or executing:

[0314] Get the video stream;

[0315] Acquire one or more reference motion vectors according to the video code stream;

[0316] Acquire one or more corresponding reference blocks from one or more image frames of the video code stream according to the one or more reference motion vectors;

[0317] determining a reference number of the one or more reference blocks;

[0318] According to a preset power consumption threshold and the number of references of the one or more reference blocks, determine one or more reference blocks to be stored in a preset memory from the one or more reference blocks, and store them in the preset memory, wherein power consumption generated by storing the reference blocks in the preset memory and reading the reference blocks from the preset memory is less than or equal to the preset power consumption threshold; and

[0319] The block to be decoded or a sub-block of the block to be decoded is decoded according to the reference block.

[0320] In one embodiment, the reference position includes a reference image frame, a reference strip, or a reference area. When the processor 703 executes the step of determining one or more reference positions from the video code stream, it may also execute: determining the one or more reference image frames, reference strips, or reference areas from the video code stream based on frame header information of an image frame in the video code stream or strip header information of one or more strips in the image frame.

[0321] In one embodiment, the reference position includes a reference image frame, a reference strip or a reference area. When the processor 703 executes the determination of the reference number of the one or more reference positions, it can also execute: determining the reference number of the one or more reference image frames, reference strips or reference areas through preset parameters, and the preset parameters include any one or more of the following: network abstraction layer parsing parameters, strip header parsing parameters, reference image list correction parameters and reference image frame marking parameters.

[0322] In one implementation, the preset memory includes a first memory and a second memory, and power consumption of the first memory is greater than power consumption of the second memory.

[0323] In one embodiment, the first memory includes a dynamic random access memory arranged outside the video decoding device, the second memory includes a system cache arranged outside the video decoding device, the reference position includes a reference image frame, a reference strip or a reference area, and the processor 703 executes the reference number according to the preset power consumption threshold and the one or more reference positions, determines the reference position to be stored in the preset memory from the one or more reference positions, and stores it in the preset memory, and may also execute: if the reference number is multiple, then according to the preset power consumption threshold, the one or more reference image frames, reference strips or reference areas with the reference number being multiple times are stored in the system cache and in the dynamic random access memory.

[0324] In one embodiment, the second memory includes a system buffer memory arranged outside the video decoding device, and the reference position includes a reference image frame, a reference strip or a reference area; when the processor 703 executes the reference number based on the preset power consumption threshold and the one or more reference positions, determines the reference position to be stored in the preset memory from the one or more reference positions, and stores it in the preset memory, it can also execute: if the reference number is multiple, then store the one or more reference image frames, reference strips or reference areas with the reference number being multiple times in the system buffer memory according to the preset power consumption threshold.

[0325] In one embodiment, the first memory includes a dynamic random access memory arranged outside the video decoding device, and the reference position includes a reference image frame, a reference strip or a reference area; when the processor 703 executes the reference number of times according to the preset power consumption threshold and the one or more reference positions, determines the reference position to be stored in the preset memory from the one or more reference positions, and stores it in the preset memory, it can also execute: if the reference number is once, then according to the preset power consumption threshold, the one or more reference image frames, reference strips or reference areas with the reference number of once being stored in the dynamic random access memory.

[0326] In one embodiment, the object to be decoded includes a block to be decoded in an image frame to be decoded, a block to be decoded in a strip to be decoded, or a block to be decoded in an area to be decoded. When the processor 703 performs the decoding of the object to be decoded according to the reference position, it may also perform: reading the required image data of the reference image frame, the reference strip, or the reference area from the preset memory; if the image data of the reference image frame is read, decoding the block to be decoded in the image frame to be decoded according to the read image data of the reference image frame; if the image data of the reference strip is read, decoding the block to be decoded in the strip to be decoded according to the read image data of the reference strip; if the image data of the reference area is read, decoding the block to be decoded in the area to be decoded according to the read image data of the reference area; if the number of references of the image frame to be decoded, the strip to be decoded, or the area to be decoded is multiple times, storing the decoded block of the block to be decoded in the system cache and in the dynamic random access memory.

[0327] In one embodiment, the object to be decoded includes a block to be decoded in an image frame to be decoded, a block to be decoded in a strip to be decoded, or a block to be decoded in an area to be decoded. When the processor 703 executes the decoding of the object to be decoded according to the reference position, it can also execute: reading the required image data of the reference image frame, reference strip or reference area from the preset memory; if the image data of the reference image frame is read, the block to be decoded in the image frame to be decoded is decoded according to the image data of the reference image frame; if the image data of the reference strip is read, the block to be decoded in the strip to be decoded is decoded according to the image data of the reference strip; if the image data of the reference area is read, the block to be decoded in the area to be decoded is decoded according to the image data of the reference area; if the number of references of the image frame to be decoded, the strip to be decoded or the area to be decoded is multiple times, the decoded block of the block to be decoded is stored in the system buffer memory.

[0328] In one embodiment, the object to be decoded includes a block to be decoded in an image frame to be decoded, a block to be decoded in a strip to be decoded, or a block to be decoded in an area to be decoded. When the processor 703 executes the decoding of the object to be decoded according to the reference position, it can also execute: reading the required image data of the reference image frame, reference strip or reference area from the preset memory; if the image data of the reference image frame is read, the block to be decoded in the image frame to be decoded is decoded according to the read image data of the reference image frame; if the image data of the reference strip is read, the block to be decoded in the strip to be decoded is decoded according to the read image data of the reference strip; if the image data of the reference area is read, the block to be decoded in the area to be decoded is decoded according to the read image data of the reference area; if the reference number of the image frame to be decoded, the strip to be decoded or the area to be decoded is once, the decoded block of the block to be decoded is stored in the dynamic random access memory.

[0329] In one embodiment, when the processor 703 executes the step of obtaining one or more reference motion vectors based on the video bitstream, it may also execute: entropy decoding the video bitstream to obtain one or more motion vector differences; and obtaining the one or more reference motion vectors based on the one or more motion vector differences and the corresponding motion vector prediction values.

[0330] In one implementation, the preset memory includes a first memory and a second memory, and power consumption of the first memory is greater than power consumption of the second memory.

[0331] In one embodiment, the first memory includes a dynamic random access memory arranged outside the video decoding device, and the second memory includes a system cache arranged outside the video decoding device. When the processor 703 executes the step of determining one or more reference blocks that need to be stored in the preset memory from the one or more reference blocks according to the preset power consumption threshold and the number of references of the one or more reference blocks, it may also execute: if the reference number is multiple times, then storing the one or more reference blocks with multiple reference times in the system cache according to the preset power consumption threshold, and storing them in the dynamic random access memory.

[0332] In one embodiment, the second memory includes a system buffer memory arranged outside the video decoding device. When the processor 703 executes the step of determining one or more reference blocks that need to be stored in the preset memory from the one or more reference blocks according to the preset power consumption threshold and the number of references of the one or more reference blocks, it may also execute: if the reference number is multiple times, then storing the one or more reference blocks with multiple reference times in the system buffer memory according to the preset power consumption threshold.

[0333] In one embodiment, the first memory includes a dynamic random access memory arranged outside the video decoding device. When the processor 703 executes the step of determining one or more reference blocks that need to be stored in the preset memory from the one or more reference blocks according to the preset power consumption threshold and the reference number of the one or more reference blocks, it may also execute: if the reference number is one, then storing the one or more reference blocks with the reference number of one time in the dynamic random access memory according to the preset power consumption threshold.

[0334] In one embodiment, when the processor 703 executes the decoding of the block to be decoded or the sub-block of the block to be decoded according to the reference block, it can also execute: reading the image data of the required reference block from the preset memory, and decoding the block to be decoded or the sub-block in the block to be decoded according to the read image data of the reference block; if the number of references of the block to be decoded or the sub-block in the block to be decoded is multiple times, storing the decoded block of the block to be decoded or the decoded sub-block of the sub-block in the block to be decoded in the system cache, and storing it in the dynamic random access memory.

[0335] In one embodiment, when the processor 703 executes the decoding of the block to be decoded or the sub-block in the block to be decoded according to the reference block, it can also execute: reading the image data of the required reference block from the preset memory, and decoding the block to be decoded or the sub-block in the block to be decoded according to the read image data of the reference block; if the number of references of the block to be decoded or the sub-block in the block to be decoded is multiple times, storing the decoded block of the block to be decoded or the decoded sub-block of the sub-block in the block to be decoded in the system buffer memory.

[0336] In one embodiment, when the processor 703 executes the decoding of the block to be decoded or the sub-block in the block to be decoded according to the reference block, it can also execute: reading the image data of the required reference block from the preset memory, and decoding the block to be decoded or the sub-block in the block to be decoded according to the read image data of the reference block; if the reference number of the block to be decoded or the sub-block in the block to be decoded is once, storing the decoded block of the block to be decoded or the decoded sub-block of the sub-block in the block to be decoded in the dynamic random access memory.

[0337] In one implementation, when the processor 703 performs the entropy decoding of the video stream to obtain one or more motion vector differences, it may also perform: entropy decoding of the video stream to obtain one or more motion vector differences and a quantized first residual.

[0338] When the processor 703 executes the decoding of the block to be decoded or the sub-block in the block to be decoded according to the image data of the read reference block, it can also execute: inverse quantization and inverse transformation of the first residual to obtain a second residual; obtain a prediction value of the block to be decoded or the sub-block in the block to be decoded according to the reference motion vector and the reference block; obtain a decoded block of the block to be decoded or a decoded sub-block of the sub-block in the block to be decoded according to the second residual and the prediction value of the block to be decoded or the sub-block in the block to be decoded.

[0339] In one embodiment, when the processor 703 executes the decoding of the block to be decoded or the sub-block in the block to be decoded according to the image data of the read reference block, it can also execute: obtaining video stream decoding data according to the block decoded from the block to be decoded or the sub-block decoded from the sub-block in the block to be decoded.

[0340] In one implementation, the prediction value of the block to be decoded or the sub-block in the block to be decoded is obtained by intra-frame prediction or motion compensation.

[0341] The present application also provides an image processing system. Fig.24 and Fig.25 , Fig.24 It is a structural diagram of the image processing system provided in an embodiment of the present application. Fig.25800 is another structural diagram of an image processing system provided by an embodiment of the present application. The image processing system 800 includes a video decoding device 801, a first memory 802, and a second memory 803, wherein the power consumption of the first memory 802 is greater than the power consumption of the second memory 803, such as the first memory may be a DRAM, the second memory 803 may be a Sys$ or a SysBuf, the first memory 802 stores a reference position with a reference number of once, or stores a reference position with a reference number of once and multiple times, and the second memory 803 stores a reference position with a reference number of multiple times. It should be noted that the reference number is the number of times the reference position is referenced by the object to be decoded.

[0342] In one embodiment, the reference position may include a reference image frame, a reference strip, a reference area, or a reference block. For example, the first memory 802 may store a reference image frame, a reference strip, or a reference block with a reference count of once, or may store a reference image frame, a reference strip, a reference area, or a reference block with a reference count of once and multiple times, that is, the first memory 802 may store a reference image frame, a reference strip, or a reference block with a reference count of once or multiple times, and the second memory 803 may store a reference image frame, a reference strip, or a reference block with a reference count of multiple times.

[0343] When the video decoding device 801 is decoding, the video decoding device 801 reads a reference position with a reference number of one time from the first memory 802, and reads a reference position with a reference number of multiple times from the second memory 803, and decodes the object to be decoded according to the reference position. The object to be decoded may include a block to be decoded in a to-be-decoded image frame, a block to be decoded in a to-be-decoded strip, a block to be decoded in a to-be-decoded area, or a sub-block of the block to be decoded.

[0344] For example, when the video decoding device 801 is decoding, a reference image frame, a reference strip or a reference block with a reference number of one time can be read from the first memory 802, and a reference image frame, a reference strip or a reference block with a reference number of multiple times can be read from the second memory 803, and the block to be decoded in the image frame to be decoded can be decoded according to the reference image frame, the block to be decoded in the strip to be decoded can be decoded according to the reference strip, the block to be decoded in the area to be decoded can be decoded according to the reference area, or the block to be decoded or the sub-block of the block to be decoded can be decoded according to the reference block.

[0345] For example, the second memory 803 can be Sys$. During decoding, if the reference image frame, reference strip, reference area or reference block needs to be referenced once, the image data of the reference image frame, reference strip, reference area or reference block is read directly from the first memory 802. If the reference image frame, reference strip, reference area or reference block needs to be referenced multiple times, when reading the reference image frame, reference strip, reference area or reference block, it can be read once from the first memory 802, and the remaining times are read from Sys$.

[0346] It can be understood that the number of times read from Sys$ can be greater than the number of times read from the first memory 802, the number of times read from Sys$ can be less than the number of times read from the first memory 802, or the number of times read from Sys$ can be equal to the number of times read from the first memory 802. The specific number of times to read from the first memory 802 and Sys$ respectively should be set accordingly according to the specific scenario, and the embodiments of the present application do not impose specific restrictions on this.

[0347] For another example, the second memory 803 may be a SysBuf. During decoding, if the reference image frame, reference strip, reference area or reference block needs to be referenced once, the image data of the reference image frame, reference strip, reference area or reference block is directly read from the first memory 802. If the reference image frame, reference strip, reference area or reference block needs to be referenced multiple times, the reference image frame, reference strip, reference area or reference block can be read from SysBuf.

[0348] In the above embodiments, the description of each embodiment has its own emphasis. For the part that is not described in detail in a certain embodiment, please refer to the detailed description of the method for performing image processing in the video decoding device above, which will not be repeated here.

[0349] The image processing device provided in the embodiment of the present application belongs to the same concept as the method for performing image processing in a video decoding device in the above embodiment. Any method provided in the embodiment of the method for performing image processing in a video decoding device can be run on the image processing device. The specific implementation process is detailed in the embodiment of the method for performing image processing in a video decoding device, which will not be repeated here.

[0350] It should be noted that, for the method of performing image processing in a video decoding device described in the embodiment of the present application, a person of ordinary skill in the art can understand that the implementation of all or part of the process of the method of performing image processing in a video decoding device described in the embodiment of the present application can be completed by controlling the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, such as a memory, and executed by at least one processor, and the execution process may include the process of the embodiment of the method of performing image processing in a video decoding device. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), etc.

[0351] For the image processing device of the embodiment of the present application, each functional module can be integrated into a processing chip, or each module can exist physically separately, or two or more modules can be integrated into one module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium, such as a read-only memory, a disk or an optical disk.

[0352] The above is a detailed introduction to a method, device, storage medium, electronic device and system for performing image processing in a video decoding device provided by an embodiment of the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A method for performing image processing in a video decoding device, characterized in that: The method comprises: Get the video stream; Determining one or more reference positions from the video code stream, wherein the reference positions include reference image frames, reference strips, or reference areas; determining a reference number of the one or more reference positions; According to a preset power consumption threshold and the reference number of the one or more reference positions, a reference position to be stored in a preset memory is determined from the one or more reference positions, and the reference position is stored in the preset memory, including: if the reference number is multiple times, one or more reference image frames, reference strips or reference areas with multiple reference times are stored in a system cache according to the preset power consumption threshold, and stored in a dynamic random access memory; or, if the reference number is multiple times, one or more reference image frames, reference strips or reference areas with multiple reference times are stored in a system buffer memory according to the preset power consumption threshold; or, if the reference number is once, one or more reference image frames, reference strips or reference areas with one reference time are stored in a dynamic random access memory according to the preset power consumption threshold; wherein the preset memory includes a first memory and / or a second memory, the first memory includes a dynamic random access memory arranged outside the video decoding device, and the second memory includes a system cache or a system buffer memory arranged outside the video decoding device; The power consumption generated by storing the reference position in the preset memory and reading the reference position from the preset memory is less than or equal to the preset power consumption threshold; and The object to be decoded is decoded according to the reference position.

2. The method for performing image processing in a video decoding device according to claim 1, characterized in that: The reference position includes a reference image frame, a reference strip, or a reference area, and determining one or more reference positions from the video code stream includes: One or more reference image frames, reference strips or reference areas are determined from the video code stream according to the frame header information of the image frame in the video code stream or the strip header information of one or more strips in the image frame.

3. The method for performing image processing in a video decoding device according to claim 1, characterized in that: The reference position includes a reference image frame, a reference strip, or a reference area, and determining the reference number of the one or more reference positions includes: The reference times of the one or more reference image frames, reference strips or reference areas are determined by preset parameters, wherein the preset parameters include any one or more of the following: network abstraction layer parsing parameters, strip header parsing parameters, reference image list correction parameters and reference image frame marking parameters.

4. The method for performing image processing in a video decoding device according to claim 1, characterized in that: The power consumption of the first memory is greater than the power consumption of the second memory.

5. The method for performing image processing in a video decoding device according to claim 1, characterized in that: The object to be decoded includes a block to be decoded in an image frame to be decoded, a block to be decoded in a strip to be decoded, or a block to be decoded in an area to be decoded, and decoding the object to be decoded according to the reference position includes: Reading the image data of the required reference image frame, reference strip or reference area from the preset memory, if the image data of the reference image frame is read, decoding the block to be decoded in the image frame to be decoded according to the image data of the reference image frame read, if the image data of the reference strip is read, decoding the block to be decoded in the strip to be decoded according to the image data of the reference strip read, if the image data of the reference area is read, decoding the block to be decoded in the area to be decoded according to the image data of the reference area read; If the reference times of the image frame to be decoded, the strip to be decoded or the area to be decoded are multiple times, the decoded blocks of the blocks to be decoded are stored in the system cache and in the dynamic random access memory.

6. The method for performing image processing in a video decoding device according to claim 1, characterized in that: The object to be decoded includes a block to be decoded in an image frame to be decoded, a block to be decoded in a strip to be decoded, or a block to be decoded in an area to be decoded, and decoding the object to be decoded according to the reference position includes: Reading the image data of the required reference image frame, reference strip or reference area from the preset memory, if the image data of the reference image frame is read, decoding the block to be decoded in the image frame to be decoded according to the image data of the reference image frame read, if the image data of the reference strip is read, decoding the block to be decoded in the strip to be decoded according to the image data of the reference strip read, if the image data of the reference area is read, decoding the block to be decoded in the area to be decoded according to the image data of the reference area read; If the reference times of the image frame to be decoded, the strip to be decoded or the area to be decoded are multiple times, the decoded blocks of the blocks to be decoded are stored in the system buffer memory.

7. The method for performing image processing in a video decoding device according to claim 1, characterized in that: The object to be decoded includes a block to be decoded in an image frame to be decoded, a block to be decoded in a strip to be decoded, or a block to be decoded in an area to be decoded, and decoding the object to be decoded according to the reference position includes: Reading the image data of the required reference image frame, reference strip or reference area from the preset memory, if the image data of the reference image frame is read, decoding the block to be decoded in the image frame to be decoded according to the image data of the reference image frame read, if the image data of the reference strip is read, decoding the block to be decoded in the strip to be decoded according to the image data of the reference strip read, if the image data of the reference area is read, decoding the block to be decoded in the area to be decoded according to the image data of the reference area read; If the reference number of the image frame to be decoded, the strip to be decoded or the area to be decoded is one, the decoded block of the block to be decoded is stored in the dynamic random access memory.

8. A method for performing image processing in a video decoding device, characterized in that: The method comprises: Get the video stream; Acquire one or more reference motion vectors according to the video code stream; Acquire one or more corresponding reference blocks from one or more image frames of the video code stream according to the one or more reference motion vectors; determining a reference number of the one or more reference blocks; According to a preset power consumption threshold and the reference times of the one or more reference blocks, one or more reference blocks to be stored in a preset memory are determined from the one or more reference blocks, and stored in the preset memory, including: if the reference times are multiple times, one or more reference blocks with multiple reference times are stored in a system cache and in a dynamic random access memory according to the preset power consumption threshold; or, if the reference times are multiple times, one or more reference blocks with multiple reference times are stored in a system buffer memory according to the preset power consumption threshold; or, if the reference times are once, one or more reference blocks with one reference time are stored in the dynamic random access memory according to the preset power consumption threshold; wherein the preset memory includes a first memory and / or a second memory, the first memory includes a dynamic random access memory arranged outside the video decoding device, and the second memory includes a system cache or a system buffer memory arranged outside the video decoding device; The power consumption generated by storing the reference block in the preset memory and reading the reference block from the preset memory is less than or equal to the preset power consumption threshold; and The block to be decoded or a sub-block in the block to be decoded is decoded according to the reference block.

9. The method for performing image processing in a video decoding device according to claim 8, characterized in that: The obtaining one or more reference motion vectors according to the video code stream includes: Performing entropy decoding on the video code stream to obtain one or more motion vector differences; The one or more reference motion vectors are obtained according to the one or more motion vector differences and corresponding motion vector prediction values.

10. The method for performing image processing in a video decoding device according to claim 8, characterized in that: The power consumption of the first memory is greater than the power consumption of the second memory.

11. The method for performing image processing in a video decoding device according to claim 8, characterized in that: The decoding of the block to be decoded or a sub-block in the block to be decoded according to the reference block comprises: Reading required image data of a reference block from the preset memory, and decoding the block to be decoded or a sub-block in the block to be decoded according to the read image data of the reference block; If the reference times of the block to be decoded or the sub-block in the block to be decoded are multiple times, the decoded block of the block to be decoded or the decoded sub-block in the block to be decoded is stored in the system cache and in the dynamic random access memory.

12. The method for performing image processing in a video decoding device according to claim 8, characterized in that: The decoding of the block to be decoded or a sub-block in the block to be decoded according to the reference block comprises: Reading required image data of a reference block from the preset memory, and decoding the block to be decoded or a sub-block in the block to be decoded according to the read image data of the reference block; If the reference times of the block to be decoded or the sub-block in the block to be decoded are multiple times, the decoded block of the block to be decoded or the decoded sub-block in the block to be decoded is stored in the system buffer memory.

13. The method for performing image processing in a video decoding device according to claim 8, characterized in that: The decoding of the block to be decoded or a sub-block in the block to be decoded according to the reference block comprises: Reading required image data of a reference block from the preset memory, and decoding the block to be decoded or a sub-block in the block to be decoded according to the read image data of the reference block; If the reference number of the block to be decoded or the sub-block in the block to be decoded is one, the decoded block of the block to be decoded or the decoded sub-block in the block to be decoded is stored in the dynamic random access memory.

14. The method for performing image processing in a video decoding device according to any one of claims 11 to 13, characterized in that: The entropy decoding of the video code stream to obtain one or more motion vector differences includes: Performing entropy decoding on the video bitstream to obtain one or more motion vector differences and a quantized first residual; The decoding of the block to be decoded or a sub-block in the block to be decoded according to the read image data of the reference block comprises: Dequantizing and inverse transforming the first residual to obtain a second residual; Obtaining a prediction value of the block to be decoded or a sub-block in the block to be decoded according to the reference motion vector and the reference block; A decoded block of the block to be decoded or a decoded subblock of the block to be decoded is obtained according to the second residual and the prediction value of the block to be decoded or the subblock in the block to be decoded.

15. The method for performing image processing in a video decoding device according to claim 14, characterized in that: The decoding of the block to be decoded or a sub-block in the block to be decoded according to the read image data of the reference block further includes: The video stream decoding data is obtained according to the decoded block of the to-be-decoded block or the decoded sub-block of the sub-block in the to-be-decoded block.

16. The method for performing image processing in a video decoding device according to claim 14, characterized in that: The prediction value of the block to be decoded or a sub-block in the block to be decoded is obtained by intra-frame prediction or motion compensation.

17. A device for performing image processing in a video decoding device, characterized in that: The device comprises: The acquisition module is used to obtain the video code stream; A first determination module, configured to determine one or more reference positions from the video code stream, wherein the reference positions include reference image frames, reference strips or reference areas; A second determination module, configured to determine a reference number of the one or more reference positions; A third determining module is configured to determine a reference position to be stored in a preset memory from the one or more reference positions according to a preset power consumption threshold and a reference number of the one or more reference positions, and store the reference position in the preset memory, wherein the power consumption generated by storing the reference position in the preset memory and reading the reference position from the preset memory is less than or equal to the preset power consumption threshold, including: if the reference number is multiple times, storing the one or more reference image frames, reference strips or reference areas with multiple reference times in the system cache according to the preset power consumption threshold, and storing them in a dynamic random access memory; or, if the reference number is multiple times, storing the one or more reference image frames, reference strips or reference areas with multiple reference times in the system buffer memory according to the preset power consumption threshold; or, if the reference number is once, storing the one or more reference image frames, reference strips or reference areas with once reference times in the dynamic random access memory according to the preset power consumption threshold; wherein the preset memory includes a first memory and / or a second memory, the first memory includes a dynamic random access memory arranged outside the video decoding device, and the second memory includes a system cache or a system buffer memory arranged outside the video decoding device; and A decoding module is used to decode the object to be decoded according to the reference position.

18. A device for performing image processing in a video decoding device, characterized in that: The device comprises: A first acquisition module, used to acquire a video code stream; A second acquisition module, used to acquire one or more reference motion vectors according to the video code stream; A third acquisition module, configured to acquire one or more corresponding reference blocks from one or more image frames of the video code stream according to the one or more reference motion vectors; A first determining module, configured to determine a reference number of the one or more reference blocks; A second determining module is configured to determine one or more reference blocks to be stored in a preset memory from the one or more reference blocks according to a preset power consumption threshold and a reference number of the one or more reference blocks, and store the reference blocks in the preset memory, wherein the power consumption generated by storing the reference blocks in the preset memory and reading the reference blocks from the preset memory is less than or equal to the preset power consumption threshold, including: if the reference number is multiple times, storing the one or more reference blocks with multiple reference times in a system cache and in a dynamic random access memory according to the preset power consumption threshold; or, if the reference number is multiple times, storing the one or more reference blocks with multiple reference times in a system buffer memory according to the preset power consumption threshold; or, if the reference number is once, storing the one or more reference blocks with one reference time in the dynamic random access memory according to the preset power consumption threshold; wherein the preset memory includes a first memory and / or a second memory, the first memory includes a dynamic random access memory arranged outside the video decoding device, and the second memory includes a system cache or a system buffer memory arranged outside the video decoding device; and A decoding module is used to decode the block to be decoded or a sub-block of the block to be decoded according to the reference block.

19. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed on a computer, the computer is caused to execute the method according to any one of claims 1 to 7 or 8 to 16.

20. An electronic device comprising a memory, a processor and a video decoding device, characterized in that: The processor executes the method according to any one of claims 1 to 7 or 8 to 16 by calling the computer program stored in the memory.

21. An image processing system, characterized in that: The invention comprises a video decoding device, a first memory and a second memory, wherein the first memory comprises a dynamic random access memory arranged outside the video decoding device, and the second memory comprises a system cache or a system buffer memory arranged outside the video decoding device; the power consumption of the first memory is greater than the power consumption of the second memory, the first memory stores reference positions with a reference count of once, or stores reference positions with a parameter count of once and multiple times, and the second memory stores reference positions with a reference count of multiple times; when decoding, the video decoding device reads reference positions with a reference count of once from the first memory, and reads reference positions with a reference count of multiple times from the second memory, and decodes the object to be decoded according to the reference positions; wherein the reference positions comprise reference image frames, reference strips, reference areas or reference blocks.

22. The image processing system according to claim 21, characterized in that: The object to be decoded includes a block to be decoded in an image frame to be decoded, a block to be decoded in a strip to be decoded, a block to be decoded in a region to be decoded, or a sub-block of the block to be decoded.

Images