Image processing method, device, storage medium and electronic device

Abstract

The present application discloses an image processing method, apparatus, storage medium, and electronic device. The method comprises: obtaining a video stream, determining an image reference frame based on a group of pictures structure in the video stream, generating a power consumption limit instruction, and determining a first decoding accuracy and a second decoding accuracy based on the power consumption limit instruction; decoding the to-be-decoded blocks of the image reference frame in the image group according to the first decoding accuracy, and decoding the to-be-decoded blocks of the non-image reference frame in the image group according to the second decoding accuracy. In embodiments of the present application, different decoding accuracies can be set for the image reference frames and non-image reference frames in the image group to be decoded when power consumption limit is required, thereby reducing the power consumption of the electronic device by sacrificing a small amount of video quality.

Description

Technical Field

[0001] The present application belongs to the technical field of electronic equipment, and in particular relates to an image processing method, device, storage medium and electronic equipment. Background Art

[0002] With the rapid development of science and technology, electronic devices are becoming increasingly powerful. Electronic devices such as video decoders (VDECs) can decode video images. When decoding a single frame of video, data from multiple previously decoded frames is often referenced. However, in related technologies, reading the referenced decoded video data consumes significant power. Summary of the Invention

[0003] The present application provides an image processing method, device, storage medium, and electronic device, which can reduce the power consumption of the electronic device.

[0004] In a first aspect, an embodiment of the present application provides an image processing method, comprising:

[0005] Get the video stream;

[0006] Determining an image reference frame according to a group of pictures structure in the video code stream;

[0007] generating a power consumption limit instruction, and determining a first decoding accuracy and a second decoding accuracy according to the power consumption limit instruction;

[0008] The blocks to be decoded of the picture reference frames in the picture group are decoded according to the first decoding accuracy, and the blocks to be decoded of the non-picture reference frames in the picture group are decoded according to the second decoding accuracy.

[0009] In a second aspect, an embodiment of the present application provides an image processing device, comprising:

[0010] Acquisition module, used to obtain video stream;

[0011] A determination module, configured to determine an image reference frame according to a group of pictures structure in the video stream;

[0012] a receiving module, configured to generate a power consumption limit instruction, and determine a first decoding accuracy and a second decoding accuracy according to the power consumption limit instruction;

[0013] A decoding module is configured to decode the to-be-decoded blocks of the image reference frame in the image group according to the first decoding accuracy, and to decode the to-be-decoded blocks of the non-image reference frame in the image group according to the second decoding accuracy.

[0014] In a third aspect, an embodiment of the present application provides a storage medium on which a computer program is stored. When the computer program is run on a computer, the computer executes the above-mentioned image processing method.

[0015] In a fourth aspect, an embodiment of the present application provides an electronic device, including a processor and a memory, wherein the memory stores a plurality of instructions, and the processor loads the instructions in the memory to perform the following steps:

[0016] Get the video stream;

[0017] Determining an image reference frame according to a group of pictures structure in the video code stream;

[0018] generating a power consumption limit instruction, and determining a first decoding accuracy and a second decoding accuracy according to the power consumption limit instruction;

[0019] The blocks to be decoded of the picture reference frames in the picture group are decoded according to the first decoding accuracy, and the blocks to be decoded of the non-picture reference frames in the picture group are decoded according to the second decoding accuracy.

[0020] The image processing method provided in an embodiment of the present application can obtain a video stream, determine an image reference frame based on the image group structure in the video stream, generate a power consumption limit instruction, and determine a first decoding accuracy and a second decoding accuracy based on the power consumption limit instruction. The to-be-decoded blocks of the image reference frame in the image group are decoded according to the first decoding accuracy, and the to-be-decoded blocks of the non-image reference frame in the image group are decoded according to the second decoding accuracy. When power consumption limit is required, the embodiment of the present application can set different decoding accuracies for the image reference frame and the non-image reference frame in the image group to be decoded, thereby reducing the power consumption of the electronic device by sacrificing a small amount of video quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 A flowchart of an image processing method provided in an embodiment of the present application.

[0023] Figure 2 A schematic diagram of the structure of a video decoding system provided in an embodiment of the present application.

[0024] Figure 3 A schematic diagram of data storage in a video decoder provided in an embodiment of the present application.

[0025] Figure 4 A schematic diagram of a scene illustrating the reference relationship between image frames in an image group provided in an embodiment of the present application.

[0026] Figure 5 A schematic diagram of an image group provided in an embodiment of the present application using different decoding accuracies with or without power consumption constraints.

[0027] Figure 6 Another flowchart of the image processing method provided in an embodiment of the present application.

[0028] Figure 7 A schematic diagram of a decoding scenario of a video decoder provided in an embodiment of the present application.

[0029] Figure 8 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application.

[0030] Figure 9 Another structural schematic diagram of the image processing device provided in an embodiment of the present application.

[0031] Figure 10 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0032] Figure 11 Another structural schematic diagram of the electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0033] Please refer to the drawings, in which the same component symbols represent the same components. The principles of this application are illustrated by implementing them in an appropriate computing environment. The following description is based on the illustrated specific embodiments of this application and should not be considered as limiting other specific embodiments of this application that are not described in detail herein.

[0034] In the following description, the specific embodiments of the present application will be described with reference to steps and symbols performed by one or more computers, unless otherwise stated. Therefore, these steps and operations will be mentioned several times as being performed by a computer, and the computer execution referred to herein includes the operation of a computer processing unit by an electronic signal representing data in a structured form. This operation converts the data or maintains it at a location in the computer's memory system, which can be reconfigured or otherwise change the operation of the computer in a manner familiar to testers in the field. The data structure in which the data is maintained is a physical location in the memory, which has specific characteristics defined by the data format. However, the principles of the present application are described in the above text, which does not represent a limitation, and testers in the field will understand that the various steps and operations described below can also be implemented in hardware.

[0035] The terms "first," "second," and "third," etc., used in this application are used to distinguish between different objects, not to describe a specific order. Furthermore, the terms "including," "having," and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or modules is not limited to the listed steps or modules, but rather some embodiments may include steps or modules not listed, or some embodiments may include other steps or modules that are inherent to the process, method, product, or apparatus.

[0036] See also Figure 1 , Figure 1 A schematic diagram of a flow chart of an image processing method provided in an embodiment of the present application. The image processing method provided in an embodiment of the present application is applied to electronic devices, and the specific flow can be as follows:

[0037] Step 101: Obtain video stream.

[0038] There are two international organizations that develop video coding and decoding technologies: the International Telecommunication Union (ITU-T), which develops standards such as H.261, H.263, and H.263+; and the International Organization for Standardization (ISO), which develops standards such as MPEG-1, MPEG-2, and MPEG-4. H.264 is a new digital video coding standard jointly developed by the Joint Video Team (JVT), a collaboration between the two organizations. In the embodiments of this application, when performing video decoding, the video bitstream is first obtained.

[0039] See also Figure 2 , Figure 2The figure is a schematic diagram of the structure of a video decoding system in the related art. In this video decoding system, the central processing unit (CPU), embedded neural network processing unit (NPU), image signal processing unit (ISP), video decoder (VDEC) and display processing unit (DISP) read and write data from DRAM through the bus and dynamic random access memory controller (DRAMC). The central processing unit, video decoder and display processor share the bandwidth in a time-sharing manner. The priority of the central processing unit and display processor is higher than that of the video encoder. It should be noted that, depending on specific needs, a display processor can be set in the video decoding system or not. The video decoder needs to perform motion compensation (MC) during decoding, which will occupy a large amount of bandwidth.

[0040] Electronic devices such as video decoders place great emphasis on cost. In order to achieve the lowest cost and highest production yield, DRAM is usually used as the primary storage space for frame buffers. Figure 3 , Figure 3 This diagram illustrates data storage in a video decoder in the related art. Bitstreams, image frames that need to be buffered, and temporary data are all stored in the DRAM within the video decoder. However, DRAM offers limited bandwidth. Temporary data can include temporal motion vectors (TMVs) and other data.

[0041] Although a video decoder can adopt an internal cache strategy for various video bitstreams, for example, the video bitstream can be the first video and audio lossy compression standard organized by the Moving Picture Experts Group Phase 1 (MPEG-1), the second video and audio lossy compression standard organized by the Moving Picture Experts Group 2 (MPEG-2), the fourth video and audio lossy compression standard organized by the Moving Picture Experts Group 4 (MPEG-4), Essential Video Coding (MPEG-5 / EVC), the low bit rate video coding standard H.263 for video conferencing developed by the ITU Telecommunication Standardization Sector (ITU-T), Advanced Video Coding (H.264 / AVC), High Efficiency Video Coding (H.265 / HEVC), and Versatile Video Coding (VVC). Coding, H.266 / VVC), Video Predictor 8 (VP8), Video Predictor 9 (VP9), Alliance for Open Media Video 1 (AV1), and other standards.

[0042] However, with the emergence of new video standards such as H.265 / HEVC, H.266 / VVC, AV1, and MPEG-5, which target ever-larger image sizes and higher frame rates, increasing DRAM bandwidth or frequency is often used to accelerate data throughput.

[0043] In an embodiment of the present application, a current video stream is obtained. The video stream may include one or more groups of pictures (GOPs), and a group of pictures includes multiple image frames. In the embodiment of the present application, a current video stream including one group of pictures is used as an example for explanation. The obtained current video stream is an encoded video stream. The image frames in the current video stream may not have been decoded, or some image frames may have been decoded, while other image frames are waiting to be decoded. It should be noted that the decoded image frames can be used as image reference frames when decoding other subsequent image frames.

[0044] Step 102: Determine an image reference frame according to the GOP structure in the video stream.

[0045] In this embodiment of the present application, since image reference frames and non-image reference frames in a picture group will subsequently need to be processed differently based on power consumption requirements, this step requires determining the reference frames within the picture group based on the picture group structure. Specifically, this can be determined through bitstream analysis, thereby selecting the bitstreams that the VDEC actually requires for high-precision (high-quality) decoding. Based on the selected bitstream segments, it is inferred that sufficient image accuracy is sufficient to achieve the required display timeliness requirements in certain situations. This method minimizes the VDEC's energy consumption and power consumption, allowing for system cooling or maintaining energy allocation to the most critical modules.

[0046] Specifically, the aforementioned image reference frame and the number of times the reference frame is referenced can be determined by preset parameters, which may include network abstraction layer parsing parameters, slice header parsing parameters, reference picture list modification parameters, and image reference frame marking parameters. For example, the network abstraction layer parsing parameters may be the nal_unit() function, the slice header parsing parameters may be the slice_header() function, the reference picture list modification parameters may be the ref_pic_list_modification() function, and the image reference frame marking parameters may be the ref_pic_list_modification() function.

[0047] For example, taking H.264 as an example, when performing a rough frame-level analysis, after parsing the NAL unit header information or slice header information contained in multiple image frames, it is possible to determine which image reference frames will be referenced multiple times. For example, this can be determined in advance using the nal_ref_idc variable in the nal_unit() function, the num_ref_idx_active_override_flag variable in the slice_header() function, the ref_pic_list_modification() function, the dec_ref_pic_marking() function and other information.

[0048] For example, the nal_unit() function extracts NAL units starting with 00 00 00 01 and 00 00 01 from an H.264 image frame and then directly fills in the length of the NAL unit. The nal_ref_idc variable represents the reference level, which indicates whether other image frames are used as references. The higher the reference level, the more important the reference frame is.

[0049] The num_ref_idx_active_override_flag variable indicates whether the number of available reference frames for the current image frame should be overridden. The num_ref_idx_l0_active_minus1 and num_ref_idx_l1_active_minus1 syntax elements, which already appear in the picture parameter set, specify the number of available reference frames in the current image reference frame queue. This pair of syntax elements can be overridden in the slice header to allow greater flexibility for a particular image frame. The num_ref_idx_active_override_flag variable indicates the slice position.

[0050] The ref_pic_list_modification() function is a reference picture list modification function that can be stored in the slice header structure. The definition of the ref_pic_list_modification() function is as follows: when ref_pic_list_modification_flag_l0 is 1, the reference picture list RefPicList0 is modified; when ref_pic_list_modification_flag_l1 is 1, the reference picture list RefPicList1 is modified. The dec_ref_pic_marking() function is used to identify the decoded picture reference frame. The marking operation is used to move the picture reference frame into or out of the picture reference frame queue and to specify the symbol of the reference picture.

[0051] For example, see Figure 4 , Figure 4 A schematic diagram of a scene illustrating a reference relationship between image frames in an image group provided in an embodiment of the present application. Figure 4 The following description is based on an example in which an image group includes 13 image frames. In other embodiments, the number of image frames included in an image group can be adjusted according to specific needs. In an image group, the display order of the image frames may be the same as the decoding order, or may be different. Figure 4 The display order of the image frames in the image group displayed in the video is different from the decoding order. In this embodiment, taking the H.264 video stream as an example, a rough analysis of the H.264 video stream is performed. By analyzing the header information of the NAL unit or the slice header information, it is possible to analyze whether certain image frames are referenced and the number of times they are referenced. By roughly analyzing the code streams corresponding to these 13 image frames, the reference image list can be obtained. Figure 4 The direction of the arrows in the image shows the reference relationship and reference times between each image frame and other image frames. When an image frame is referenced by other image frames once or multiple times, the image frame can be used as an image reference frame. For example, Figure 4 The reference number of the I0 frame in the display order is 4 times, the reference number of the B2 frame with a display order of 2 is 4 times, the reference number of the B3 frame with a display order of 3 is 2 times, the reference number of the P4 frame with a display order of 4 is 3 times, the reference number of the B6 frame with a display order of 6 is 1 time, the reference number of the B8 frame with a display order of 8 is 5 times, the reference number of the P12 frame with a display order of 12 is 4 times, and so on. That is, in this embodiment, after bitstream analysis, it is found that the frames that really need to be accurately interpreted are the reference frames I0, B2, B3, P4, B6, B8, and P12 for the frames that need to be played. The remaining image frames B1, B5, B7, P9, P10, and P11 are non-image reference frames. It should be noted that Figure 4 The above analysis is only a rough frame-level analysis by the hardware or software of the video decoder. The reference relationship of each specific block in the reference frame image cannot be analyzed by this rough analysis method.

[0052] Step 103: Generate a power consumption limit instruction, and determine a first decoding accuracy and a second decoding accuracy according to the power consumption limit instruction.

[0053] In an embodiment of the present application, after completing the bit rate analysis, the decoder can decide whether to reduce the decoded image quality according to the power consumption requirements. If the processor of the electronic device generates a power consumption instruction limit instruction, a first decoding accuracy and a second decoding accuracy can be further obtained, wherein the above-mentioned first decoding accuracy is the accuracy when decoding the image reference frame, and the second decoding accuracy is the accuracy when decoding the non-image reference frame, wherein the above-mentioned first decoding accuracy is greater than the second decoding accuracy.

[0054] For example, continue to refer to Figure 4 Frames I0, B2, B3, P4, B6, B8, and P12 in the figure are used as reference frames by other frames, so during decoding, a higher first decoding accuracy is achieved, such as 100% accuracy. However, this is not limited to 100% accuracy in actual implementation. The remaining frames B1, B5, B7, P9, P10, and P11 are not used as reference frames. Therefore, to meet power consumption targets, image quality can be sacrificed and a lower second decoding accuracy can be achieved, such as 80% accuracy.

[0055] The first decoding accuracy and the second decoding accuracy may be pre-stored in a memory of the electronic device and directly retrieved upon receiving a power consumption limit instruction. In other embodiments, the first decoding accuracy and the second decoding accuracy may also be included in the power consumption limit instruction. For example, upon receiving the power consumption limit instruction, the first decoding accuracy and the second decoding accuracy in the instruction are further retrieved.

[0056] In one embodiment, the power consumption limit instruction may be automatically generated by the electronic device. For example, when the remaining battery power of the device is less than a preset value, the power consumption limit instruction carrying the first decoding accuracy and the second decoding accuracy is automatically generated. That is, before receiving the power consumption limit instruction, the method further includes:

[0057] Get the current remaining power of the electronic device;

[0058] Determining whether the remaining power is less than a preset value;

[0059] If it is less than the preset value, a power consumption limiting instruction is generated according to the remaining power.

[0060] Furthermore, after determining whether the remaining battery level is less than a preset value, a first decoding accuracy and a second decoding accuracy may be set based on a preset interval corresponding to the remaining battery level to generate a power consumption limit instruction. For example, if the preset value is set to 30%, and three battery intervals of 0-10%, 11%-20%, and 21%-30% are set below the preset value, then if the current remaining battery level of the electronic device is 25%, which is within the 21%-30% battery interval, a first decoding accuracy of 100% and a second decoding accuracy of 80% corresponding to this interval are obtained, and a power consumption limit instruction is generated based on the first and second decoding accuracies. If the current remaining battery level of the electronic device is 18%, which is within the 11%-20% battery interval, a first decoding accuracy of 100% and a second decoding accuracy of 70% corresponding to this interval are obtained, and a power consumption limit instruction is generated based on the first and second decoding accuracies. If the current remaining battery level of the electronic device is 5%, which is within the 0-10% battery interval, a first decoding accuracy of 90% and a second decoding accuracy of 60% corresponding to this interval are obtained, and a power consumption limit instruction is generated based on the first and second decoding accuracies.

[0061] Step 104 : Decode the blocks to be decoded of the image reference frames in the group of pictures according to the first decoding accuracy, and decode the blocks to be decoded of the non-image reference frames in the group of pictures according to the second decoding accuracy.

[0062] For example, if a power consumption limit instruction is received and the first decoding accuracy is 100% and the second decoding accuracy is 70%, Figure 4 In the embodiment, frames I0, B2, B3, P4, B6, B8, and P12 in the figure can be decoded with 100% accuracy, while frames B1, B5, B7, P9, P10, and P11 can be decoded with 70% accuracy.

[0063] In other embodiments, the non-image reference frame may also be decoded using different decoding accuracies. For example, different second decoding accuracies are used to decode multiple non-image reference frames. Figure 5 , Figure 5 A schematic diagram illustrating different decoding accuracy levels for an image group with and without power consumption limits is provided for an embodiment of the present application. In this embodiment, if no power consumption limit instruction is received, meaning the electronic device currently has sufficient battery life and there is no need to sacrifice decoded image quality to reduce power consumption, all image frames in the image group can be decoded with 100% decoding accuracy when decoding the block to be decoded. If a power consumption limit instruction is received, the image reference frame is decoded with a first decoding accuracy, and multiple non-image reference frames are decoded with their respective second decoding accuracy levels.

[0064] As can be seen from the above, the image processing method provided in the embodiment of the present application can obtain the current video stream, determine the image reference frame based on the picture group structure in the video stream, receive a power consumption limit instruction, and determine a first decoding accuracy and a second decoding accuracy based on the power consumption limit instruction. The decoding block of the image reference frame in the picture group is decoded according to the first decoding accuracy, and the decoding block of the non-image reference frame in the picture group is decoded according to the second decoding accuracy. When power consumption limit is required, the embodiment of the present application can set different decoding accuracies for the image reference frame and non-image reference frame in the picture group to be decoded, thereby reducing the power consumption of the electronic device by sacrificing a small amount of video quality.

[0065] The following will further introduce the image processing method of this application based on the method described in the above embodiment. Figure 6 , Figure 6 Another flowchart of an image processing method provided in an embodiment of the present application is provided, wherein the image processing method includes:

[0066] Step 201: Get the current video stream.

[0067] In an embodiment of the present application, a current video stream is obtained. The video stream may include one or more groups of pictures (GOPs), and a group of pictures includes multiple image frames. In the embodiment of the present application, a current video stream including one group of pictures is used as an example for explanation. The obtained current video stream is an encoded video stream. The image frames in the current video stream may not have been decoded, or some image frames may have been decoded, while other image frames are waiting to be decoded. It should be noted that the decoded image frames can be used as image reference frames when decoding other subsequent image frames.

[0068] Step 202: Obtain decoding dependencies corresponding to the picture groups in the video stream.

[0069] Step 203: Determine the image reference frame according to the decoding dependency relationship.

[0070] The decoding dependency relationship includes a reference relationship between each image frame and other image frames. Specifically, whether the current frame is referenced can be determined in sequence based on the decoding dependency relationship. If so, the current frame is determined to be an image reference frame. If not, the current frame is determined to be a non-image reference frame. That is, the step of determining the image reference frame based on the decoding dependency relationship includes:

[0071] Determining whether the current frame is referenced according to the decoding dependency relationship;

[0072] If so, the current frame is determined to be the image reference frame.

[0073] In one embodiment, VDEC-related hardware or software can determine, after analysis or estimation, that certain image frames will be used when decoding other frames, i.e., the aforementioned decoding dependency. Specifically, the decoding dependency of each image frame can be determined based on the frame header information of each image frame in the current video stream. The data of each image frame can be considered a Network Abstraction Layer (NAL) unit. The frame header information is used to identify the beginning of an image frame. The frame header information can also be considered as NAL unit header information. The frame header information can be used to determine which image frame is being used, i.e., the image reference frame can be determined.

[0074] Step 204: Receive a power consumption limit instruction, and determine a first decoding accuracy and a second decoding accuracy according to the power consumption limit instruction.

[0075] In an embodiment of the present application, after completing the bit rate analysis, the decoder can decide whether to reduce the quality of the decoded image according to the power consumption requirements. If the processor of the electronic device receives a power consumption instruction limit instruction, it can further obtain a first decoding accuracy and a second decoding accuracy, wherein the above-mentioned first decoding accuracy is the accuracy when decoding the image reference frame, and the second decoding accuracy is the accuracy when decoding the non-image reference frame, wherein the above-mentioned first decoding accuracy is greater than the second decoding accuracy, for example, 100% accuracy is used as the first decoding accuracy, and 80% accuracy is used as the second decoding accuracy.

[0076] In other embodiments, the electronic device may also automatically generate a power consumption limit instruction based on the need for long-term video playback and the high power consumption caused by predictable behavior. Furthermore, before generating the power consumption limit instruction, a prompt message may be pre-generated and presented to the user, such as in the form of a pop-up window or voice, and then receive an operation instruction from the user to select whether to confirm. The power consumption limit instruction is then generated when the user confirms to enable power consumption limit.

[0077] Step 205 : When decoding the block to be decoded, determine the corresponding tree diagram according to the preset table of the block to be decoded.

[0078] The bitstream is interpreted during the decoding process. The preset table may be a Transform Tree Syntax Table, which can be obtained from the bitstream. The preset table can be interpreted into a tree diagram, and the tree diagram can be used to quickly infer relevant information about the transform block, such as the size of the transform block and the specific method of inverse transformation of the transform block.

[0079] Step 206: Determine the transform block of the block to be decoded according to the tree diagram.

[0080] In the embodiments of the present application, non-zero values ​​are discarded to reduce the power consumption of the electronic device. This is because, during the decoding process, a greater number of zero values ​​results in fewer togglings in the multiplier and adder circuits. In other words, fewer non-zero values ​​reduce the overall power consumption and energy consumption of the inverse transform. Therefore, before the inverse transform, a corresponding tree diagram can be determined based on a preset table of the block to be decoded. The transform block and inverse transform method can then be inferred from the tree diagram.

[0081] Step 207 : reset the residual coefficients in the transform blocks of the to-be-decoded blocks of the image reference frame in the image group to zero according to the first decoding accuracy, and perform inverse transformation on the transform blocks after the residual coefficients are reset to zero.

[0082] Since people are not sensitive to high-frequency signals but are more sensitive to low-frequency signals during actual use, before the inverse transformation, the embodiment of the present application can adopt a method of discarding high-frequency coefficients to appropriately discard high frequencies, thereby reducing power consumption due to the increase in 0 inputs during the inverse transformation.

[0083] It should be noted that if the above-mentioned first decoding accuracy is 100%, there is no need to reset the residual coefficients in the transform block of the to-be-decoded block of the image reference frame in the picture group to zero, because the first decoding accuracy of 100% indicates that there is no need to reduce power consumption when decoding the image reference frame at this time. If the first decoding accuracy is less than 100%, the step of reseting the residual coefficients in the transform block of the to-be-decoded block of the image reference frame in the picture group to zero according to the first decoding accuracy is performed.

[0084] Specifically, in an embodiment of the present application, different positions in a word block have different corresponding frequencies. For example, in a sub-block, there are five points A, B, C, D, and E. Among these five points, E has the highest frequency component residual. The frequencies decrease in order from E, D, C, B, and A. A has the lowest frequency component residual. Since the high-frequency component residual usually has the smallest visual impact and is therefore the least important, in this embodiment, the residual coefficients can be decreased in order from the most important A to the least important E on the spectrum. Specifically, the residual can be reset to zero in the reverse direction of the wavefront, such as E to D to C to B to A. Resetting the residual coefficients to zero in the reverse direction of the wavefront will cause the picture accuracy to gradually decrease. Among them, the above-mentioned first decoding accuracy can determine the degree to which the above-mentioned residual coefficients are reset to zero, that is, the degree to which the picture accuracy decreases. Therefore, the step of resetting the residual coefficients in the transform block of the to-be-decoded block of the image reference frame in the image group according to the first decoding accuracy includes:

[0085] Obtaining multiple component residuals and corresponding frequencies in a transform block of a block to be decoded in the image reference frame;

[0086] The plurality of component residuals are sorted from high frequency to low frequency, and residual coefficients in a transform block of a to-be-decoded block in the image reference frame are reset to zero according to the first decoding accuracy and the sorting result.

[0087] Step 208 : reset the residual coefficients in the transform blocks of the to-be-decoded blocks of the non-image reference frames in the group of pictures to zero according to the second decoding accuracy, and perform inverse transformation on the transform blocks after the residual coefficients are reset to zero.

[0088] The second decoding accuracy is less than the first second decoding accuracy and is therefore necessarily less than 100%. The step of returning the residual coefficients in the transform block of the to-be-decoded block of the non-image reference frame in the group of pictures to zero can be referred to as step 207. That is, the step of returning the residual coefficients in the transform block of the to-be-decoded block of the non-image reference frame in the group of pictures to zero according to the second decoding accuracy includes:

[0089] Obtaining a plurality of component residuals and corresponding frequencies in a transform block of a block to be decoded in the non-image reference frame;

[0090] The plurality of component residuals are sorted from high frequency to low frequency, and residual coefficients in a transform block of a to-be-decoded block in the non-image reference frame are reset to zero according to the second decoding accuracy and the sorting result.

[0091] Step 209: Decode the block to be decoded in the inverse-transformed block.

[0092] Specifically, the step of decoding the block to be decoded in the inverse-transformed block may include:

[0093] Obtaining a reference motion vector according to the current video stream;

[0094] Acquire a corresponding reference block from an image frame of the current video stream according to the reference motion vector;

[0095] The block to be decoded is decoded according to the reference block.

[0096] In one embodiment, the current video stream is parsed to obtain one or more reference motion vectors. Each reference motion vector corresponds to a reference block, which is used as a reference when decoding the block to be decoded. The relative displacement between the reference block and the block to be encoded can be used as the reference motion vector. Obtaining the corresponding reference block using the reference motion vector allows for refined analysis, and the block to be decoded is then decoded based on the image data of the reference block.

[0097] Specifically, the current video stream can be entropy decoded to obtain one or more motion vector differences (MVDs), and then one or more reference motion vectors can be obtained based on the one or more motion vector differences and the corresponding motion vector prediction values. For example, after entropy decoding the current video stream to obtain one or more motion vector differences, one or more reference motion vectors can be obtained based on the one or more motion vector differences and the corresponding motion vector prediction values, such as adding the motion vector difference to the motion vector prediction value as the reference motion vector. Based on the one or more reference motion vectors, one or more corresponding reference blocks are obtained from one or more image frames of the current video stream, and finally the image data of the required reference blocks are read, and the current block to be decoded or the sub-block in the current block to be decoded is decoded based on the read image data of the reference blocks.

[0098] During decoding, the image frame may be divided into a plurality of non-overlapping blocks, which form a rectangular array, wherein each block is an N×N pixel block, for example, a 4×4 pixel block, a 32×32 pixel block, a 128×128 pixel block, etc. When decoding a block to be decoded in the current image frame to be decoded, it is necessary to read the image data of the required image reference frame from the memory.

[0099] For example, see Figure 7 , Figure 7A schematic diagram of a decoding scenario for a video decoder provided in an embodiment of the present application is provided. The current video stream is illustrated using an H.264 video stream as an example. After entropy decoding of the current video stream, one or more motion vector differences and a quantized first residual are obtained. Based on the motion vector differences and the corresponding motion vector prediction values, a reference motion vector can be obtained, providing a more precise understanding of the reference block used for motion compensation. Entropy decoding can be implemented using a separate hardware design or software. Parsing of the current video stream and image buffering can be implemented in software using a driver or the Open Media Acceleration (OpenMAX) framework. Dequantization and inverse transformation of the first residual yield a second residual. Based on the reference motion vector (the relative displacement between the reference block and the current block to be decoded or a sub-block within the current block to be decoded) and the reference block, a predicted value for the current block to be decoded or a sub-block within the current block to be decoded can be obtained. It should be noted that the predicted value for the current block to be decoded or a sub-block within the current block to be decoded can be obtained using intra-frame prediction or motion compensation. Inverse quantization and inverse transformation, intra / inter mode selection, intra prediction, motion compensation and deblocking filtering in the decoding process can be implemented through an application-specific integrated circuit (ASIC).

[0100] After obtaining the predicted value of the current block to be decoded or the sub-block in the current block to be decoded, the second residual is added to the predicted value of the current block to be decoded or the sub-block in the current block to be decoded to obtain a decoded block of the current block to be decoded or a decoded sub-block of the sub-block in the current block to be decoded (actual value). After performing a blocking effect filter on the decoded block of the current block to be decoded or the decoded sub-block of the sub-block in the current block to be decoded, smoothed video stream decoded data can be obtained.

[0101] If there are other blocks to be decoded or sub-blocks of other blocks to be decoded in the current image frame to be decoded or the current strip to be decoded that need to be decoded, then the other blocks to be decoded or sub-blocks of other blocks to be decoded are decoded. If all blocks to be decoded or sub-blocks of blocks to be decoded in the current image frame to be decoded or the current strip to be decoded have been decoded, then other image frames or strips are decoded until all image frames or strips that need to be decoded are decoded.

[0102] In addition, the method provided in the embodiment of the present application can appropriately reduce the image quality after video decoding. When the picture decoded by the video decoder is reduced, it means that the high-frequency signal will be lost. Usually, at this time, it is possible to reconstruct less high-frequency signals without being detected. Filtering is performed through post-filters such as DBF (Deblocking Filter), SAO (Sample Adaptive Offset), ALF (Adaptive Loop Filter), etc. The above-mentioned methods of filtering through post-filters originally intended to achieve what is not so obvious after reducing the image quality (that is, the original image is filtered by a low-pass filter). In other words, reducing the image can appropriately cover up the image quality loss caused by not performing these in-loop filters. Therefore, this type of method for reducing power consumption is also very suitable for when the video image is reduced in the embodiment of the present application.

[0103] As can be seen from the above, the image processing method provided in the embodiment of the present application can obtain a current video stream, obtain a decoding dependency corresponding to a group of images in the video stream, determine an image reference frame based on the decoding dependency, receive a power consumption limit instruction, and determine a first decoding accuracy and a second decoding accuracy based on the power consumption limit instruction. When decoding a block to be decoded, a corresponding tree diagram is determined based on a preset table of the block to be decoded, a transform block of the block to be decoded is determined based on the tree diagram, residual coefficients in the transform block of the block to be decoded in the image reference frame in the image group are reset to zero based on the first decoding accuracy, and the transform block after the residual coefficients are reset to zero is inversely transformed. Residual coefficients in the transform block of the block to be decoded in the non-image reference frame in the image group are reset to zero based on the second decoding accuracy, and the transform block after the residual coefficients are reset to zero is inversely transformed, and the block to be decoded in the inversely transformed block is decoded. When power consumption limit is required, the embodiment of the present application can set different decoding accuracies for the image reference frame and non-image reference frame in the image group to be decoded, thereby reducing the power consumption of the electronic device by sacrificing a small amount of video quality.

[0104] See also Figure 8 , Figure 8 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application. The image processing device 30 includes:

[0105] Acquisition module 301, used to acquire video stream;

[0106] A determination module 302 is configured to determine an image reference frame according to a GOP structure in the video stream;

[0107] A receiving module 303 is configured to generate a power consumption limit instruction and determine a first decoding accuracy and a second decoding accuracy according to the power consumption limit instruction;

[0108] The decoding module 304 is configured to decode the to-be-decoded blocks of the image reference frame in the image group according to the first decoding accuracy, and decode the to-be-decoded blocks of the non-image reference frame in the image group according to the second decoding accuracy.

[0109] In one embodiment, see Figure 9 In this embodiment, the determining module 302 may include:

[0110] An acquisition submodule 3021 is configured to acquire decoding dependencies corresponding to groups of pictures in the video stream;

[0111] A judgment submodule 3022 is used to determine whether the current frame is referenced according to the decoding dependency relationship;

[0112] The first determining submodule 3023 is configured to determine that the current frame is the image reference frame when the determining submodule determines that the current frame is the image reference frame.

[0113] In one embodiment, the decoding module 304 may include:

[0114] The second determining submodule 3041 is configured to determine a corresponding tree diagram according to a preset table of the block to be decoded when decoding the block to be decoded;

[0115] A third determining submodule 3042 is configured to determine a transform block of the to-be-decoded block according to the tree diagram;

[0116] A first processing submodule 3043 is configured to reset residual coefficients in a transform block of a block to be decoded in an image reference frame in the image group to zero according to the first decoding accuracy, and perform an inverse transform on the transform block after the residual coefficients are reset to zero;

[0117] The second processing submodule 3044 is configured to reset residual coefficients in a transform block of a to-be-decoded block of a non-image reference frame in the image group to zero according to the second decoding accuracy, and perform an inverse transform on the transform block after the residual coefficients are reset to zero.

[0118] As can be seen from the above, the image processing device 30 of the embodiment of the present application can obtain the current video stream, determine the image reference frame based on the picture group structure in the video stream, receive a power consumption limit instruction, and determine a first decoding accuracy and a second decoding accuracy based on the power consumption limit instruction. The decoding block of the image reference frame in the picture group is decoded according to the first decoding accuracy, and the decoding block of the non-image reference frame in the picture group is decoded according to the second decoding accuracy. When power consumption limit is required, the embodiment of the present application can set different decoding accuracies for the image reference frame and non-image reference frame in the picture group to be decoded, thereby reducing the power consumption of the electronic device by sacrificing a small amount of video quality.

[0119] In the embodiment of the present application, the image processing device and the image processing method in the above embodiment belong to the same concept. Any method provided in the image processing method embodiment can be run on the image processing device. The specific implementation process is detailed in the embodiment of the image processing method and will not be repeated here.

[0120] The term "module" as used herein may be considered a software object executed on the computing system. The various components, modules, engines, and services described herein may be considered implementation objects on the computing system. The devices and methods described herein may be implemented in software or hardware, and are all within the scope of protection of this application.

[0121] An embodiment of the present application further provides a storage medium storing a computer program. When the computer program is run on a computer, the computer is caused to execute the above-mentioned image processing method.

[0122] The present application also provides an electronic device, such as a tablet computer, a mobile phone, etc. A processor in the electronic device loads instructions corresponding to one or more application processes into a memory according to the following steps, and the processor runs the application stored in the memory to implement various functions:

[0123] Get the video stream;

[0124] Determining an image reference frame according to a group of pictures structure in the video code stream;

[0125] generating a power consumption limit instruction, and determining a first decoding accuracy and a second decoding accuracy according to the power consumption limit instruction;

[0126] The blocks to be decoded of the picture reference frames in the picture group are decoded according to the first decoding accuracy, and the blocks to be decoded of the non-picture reference frames in the picture group are decoded according to the second decoding accuracy.

[0127] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0128] See also Figure 10 , the electronic device 400 includes a processor 401 and a memory 402. The processor 401 is electrically connected to the memory 402.

[0129] The processor 400 is the control center of the electronic device 400. It uses various interfaces and lines to connect various parts of the entire electronic device. By running or loading computer programs stored in the memory 402 and calling data stored in the memory 402, it executes various functions of the electronic device 400 and processes data, thereby monitoring the electronic device 400 as a whole.

[0130] The memory 402 can be used to store software programs and modules. The processor 401 executes various functional applications and data processing by running the computer programs and modules stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, a computer program required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 402 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage device. Accordingly, the memory 402 may also include a memory controller to provide the processor 401 with access to the memory 402.

[0131] In the embodiment of the present application, the processor 401 in the electronic device 400 loads instructions corresponding to one or more computer program processes into the memory 402 according to the following steps, and the processor 401 runs the computer program stored in the memory 402 to implement various functions as follows:

[0132] Get the video stream;

[0133] Determining an image reference frame according to a group of pictures structure in the video code stream;

[0134] generating a power consumption limit instruction, and determining a first decoding accuracy and a second decoding accuracy according to the power consumption limit instruction;

[0135] The blocks to be decoded of the picture reference frames in the picture group are decoded according to the first decoding accuracy, and the blocks to be decoded of the non-picture reference frames in the picture group are decoded according to the second decoding accuracy.

[0136] Please also refer to Figure 11 In some embodiments, the electronic device 400 may further include: a display 403, a radio frequency circuit 404, an audio circuit 405, and a power supply 406. The display 403, the radio frequency circuit 404, the audio circuit 405, and the power supply 406 are electrically connected to the processor 401, respectively.

[0137] The display 403 can be used to display information input by the user or information provided to the user, as well as various graphical user interfaces. These graphical user interfaces can be composed of graphics, text, icons, videos, or any combination thereof. The display 403 may include a display panel. In some embodiments, the display panel can be configured in the form of a liquid crystal display (LCD) or an organic light-emitting diode (OLED).

[0138] The RF circuit 404 can be used to transmit and receive RF signals to establish wireless communication with network devices or other electronic devices through wireless communication, and to transmit and receive signals between network devices or other electronic devices. Typically, the RF circuit 501 includes but is not limited to an antenna, at least one amplifier, a tuner, one or more oscillators, a subscriber identity module (SIM) card, a transceiver, a coupler, a low noise amplifier (LNA), a duplexer, etc.

[0139] The audio circuit 405 can be used to provide an audio interface between the user and the electronic device through a speaker and a microphone. The audio circuit 506 can convert the received audio data into an electrical signal, transmit it to the speaker, and convert it into a sound signal for output.

[0140] Power supply 406 can be used to power various components of electronic device 400. In some embodiments, power supply 406 can be logically connected to processor 401 via a power management system, thereby enabling the power management system to manage charging, discharging, and power consumption. Power supply 406 can also include one or more DC or AC power supplies, a recharging system, a power failure detection circuit, a power converter or inverter, a power status indicator, and other arbitrary components.

[0141] although Figure 11 Not shown in the figure, the electronic device 400 may further include a camera, a Bluetooth module, etc., which will not be described in detail here.

[0142] In the embodiment of the present application, the storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0143] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0144] It should be noted that, with respect to the image processing method of the embodiment of the present application, a person skilled in the art will understand that all or part of the process of implementing the image processing method of the embodiment of the present application can be accomplished by controlling the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, such as a memory of an electronic device, and executed by at least one processor within the electronic device. During execution, the process may include the process of the embodiment of the image processing method. The storage medium may be a magnetic disk, an optical disk, a read-only memory, a random access memory, or the like.

[0145] For the image processing device of the embodiment of the present application, its various functional modules can be integrated into a single processing chip, or each module can exist physically separately, or two or more modules can be integrated into a single module. The above-mentioned integrated modules can be implemented in the form of hardware or in the form of software functional modules. If the integrated modules are implemented in the form of software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium, such as a read-only memory, a magnetic disk, or an optical disk.

[0146] The above is a detailed introduction to an image processing method, device, storage medium and electronic device provided in the embodiments of the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for those skilled in the art, based on the ideas of the present application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. An image processing method, characterized in that: The method comprises the following steps: Get the video stream; Determining an image reference frame according to a group of pictures structure in the video code stream; generating a power consumption limit instruction, and determining a first decoding accuracy and a second decoding accuracy based on the power consumption limit instruction; and Decoding a block to be decoded of an image reference frame in the image group according to the first decoding accuracy, and decoding a block to be decoded of a non-image reference frame in the image group according to the second decoding accuracy, wherein, when decoding the block to be decoded, determining a corresponding tree diagram according to a preset table of the block to be decoded, and determining a transform block of the block to be decoded according to the tree diagram; obtaining multiple component residuals and corresponding frequencies in the transform block of the block to be decoded in the image reference frame, sorting the multiple component residuals from high frequency to low frequency, and returning residual coefficients in the transform block of the block to be decoded in the image reference frame to zero according to the first decoding accuracy and the sorting result; obtaining multiple component residuals and corresponding frequencies in the transform block of the block to be decoded in the non-image reference frame; sorting the multiple component residuals from high frequency to low frequency, and returning residual coefficients in the transform block of the block to be decoded in the non-image reference frame to zero according to the second decoding accuracy and the sorting result; and decoding the block to be decoded in the inversely transformed block.

2. The image processing method according to claim 1, wherein: The determining of the image reference frame according to the GOP structure in the video stream includes: Obtaining decoding dependencies corresponding to image groups in the video stream; Determining whether the current frame is referenced according to the decoding dependency relationship; If so, the current frame is determined to be the image reference frame.

3. The image processing method according to claim 1, wherein: The decoding of the to-be-decoded block in the inversely transformed block includes: Obtaining a reference motion vector according to the video code stream; Acquiring a corresponding reference block from an image frame of the video stream according to the reference motion vector; The block to be decoded is decoded according to the reference block.

4. The image processing method according to claim 1, wherein: The first decoding accuracy is greater than the second decoding accuracy.

5. The image processing method according to claim 1, wherein: Before generating the power consumption limit instruction, the method further includes: Get the current remaining power of the electronic device; Determining whether the remaining power is less than a preset value; If it is less than the preset value, a power consumption limiting instruction is generated according to the remaining power.

6. An image processing device, characterized in that The device comprises: Acquisition module, used to obtain video stream; A determination module, configured to determine an image reference frame according to a group of pictures structure in the video stream; a receiving module, configured to generate a power consumption limit instruction, and determine a first decoding accuracy and a second decoding accuracy according to the power consumption limit instruction; A decoding module is configured to decode a block to be decoded of an image reference frame in the image group according to the first decoding accuracy, and to decode a block to be decoded of a non-image reference frame in the image group according to the second decoding accuracy, wherein, when decoding the block to be decoded, a corresponding tree diagram is determined according to a preset table of the block to be decoded, and a transform block of the block to be decoded is determined according to the tree diagram; obtain multiple component residuals and corresponding frequencies in the transform block of the block to be decoded in the image reference frame, sort the multiple component residuals from high frequency to low frequency, and reset residual coefficients in the transform block of the block to be decoded in the image reference frame to zero according to the first decoding accuracy and the sorting result; obtain multiple component residuals and corresponding frequencies in the transform block of the block to be decoded in the non-image reference frame; sort the multiple component residuals from high frequency to low frequency, and reset residual coefficients in the transform block of the block to be decoded in the non-image reference frame to zero according to the second decoding accuracy and the sorting result; and decode the block to be decoded in the inversely transformed block.

7. The image processing device according to claim 6, wherein: The determination module includes: An acquisition submodule, configured to acquire decoding dependencies corresponding to image groups in the video stream; A judgment submodule, configured to determine whether the current frame is referenced according to the decoding dependency relationship; The first determining submodule is configured to determine that the current frame is the image reference frame when the judging submodule determines that the current frame is the image reference frame.

8. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed on a computer, the computer is caused to execute the image processing method according to any one of claims 1 to 6.

9. An electronic device comprising a processor and a memory, wherein the memory stores a plurality of instructions, characterized in that: The processor loads the instructions in the memory for performing the following steps: Get the video stream; Determining an image reference frame according to a group of pictures structure in the video code stream; generating a power consumption limit instruction, and determining a first decoding accuracy and a second decoding accuracy according to the power consumption limit instruction; Decoding a block to be decoded of an image reference frame in the image group according to the first decoding accuracy, and decoding a block to be decoded of a non-image reference frame in the image group according to the second decoding accuracy, wherein, when decoding the block to be decoded, determining a corresponding tree diagram according to a preset table of the block to be decoded, and determining a transform block of the block to be decoded according to the tree diagram; obtaining multiple component residuals and corresponding frequencies in the transform block of the block to be decoded in the image reference frame, sorting the multiple component residuals from high frequency to low frequency, and returning residual coefficients in the transform block of the block to be decoded in the image reference frame to zero according to the first decoding accuracy and the sorting result; obtaining multiple component residuals and corresponding frequencies in the transform block of the block to be decoded in the non-image reference frame; sorting the multiple component residuals from high frequency to low frequency, and returning residual coefficients in the transform block of the block to be decoded in the non-image reference frame to zero according to the second decoding accuracy and the sorting result; and decoding the block to be decoded in the inversely transformed block.

Images