Region of Interest Aware Adaptive Resolution Video Coding

Abstract

Methods and systems are provided for supporting resolution adaptive video encoding of output frames for display based on reference frames of different resolutions derived from the same image data, each frame being compressed by a different method based on a region of interest of the source image data, thereby conserving computing resources and improving compression efficiency. The methods and systems described herein provide a video encoder that: generates a group of multiple frames having different resolutions from a source frame; determines to divide each frame of the group into a plurality of different relative regions of visual interest; encodes the reference frame of the group using different encoding modes for at least two different regions of a reference frame; and encodes the output frame of the group using different encoding modes for at least two different regions of an output frame.

Description

Background Art

[0001] In conventional video coding formats such as H.264 / AVC (Advanced Video Coding) and H.265 / HEVC (High Efficiency Video Coding) standards, video frames in a sequence have their size and resolution recorded in the header at the sequence level. Therefore, in order to change the frame resolution, a new video sequence must be generated starting from intra-coded frames, which carry significantly greater transmission bandwidth costs than inter-coded frames. Therefore, although it is desirable to adaptively send downsampled low-resolution video over the network when network bandwidth becomes low, reduced or throttled, it is difficult to achieve bandwidth savings while using conventional video coding formats because the bandwidth cost of adaptive downsampling offsets the bandwidth gain.

[0002] Support for changing resolution while sending coded frames has been studied. In the implementation of the AV1 codec developed by AOM, and in the development of the next generation video codec specification, VVC, new frame types, and new motion prediction coding tools are provided. In general, support for encoding of frames that reference previous frames with different resolutions can be implemented in various ways to achieve the goal of reducing bandwidth costs.

[0003] The result of implementing such techniques will be that the encoder can output frames of various resolutions into a bitstream, and the frames can be encoded for display using motion information of reference frames of any other resolution or many other resolutions. Therefore, it is desirable to optimize the encoding of frame data sent in a bitstream so as to utilize variable resolution bitstream data to facilitate motion prediction and motion compensation, reduce consumption of computing resources, and improve compression efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] The detailed description is set forth with reference to the accompanying drawings. In each figure, the leftmost digit(s) of a reference number identifies the figure in which the reference number first appears. The use of the same reference numbers in different figures indicates similar or identical items or features.

[0005] Figure 1 An example of dividing a frame into regions according to corresponding visual interests according to an exemplary embodiment of the present disclosure is illustrated.

[0006] Figure 2 Illustrate grouping of multiple frames having different resolutions all derived from the same source frame as generated by a video encoder according to an example embodiment of the present disclosure.

[0007] Figure 3 A region of interest perceptual encoding method according to an example embodiment of the present disclosure is illustrated.

[0008] Figure 4An example system for implementing the processes and methods described herein for implementing support for region of interest perceptual coding methods in a video encoder is illustrated. DETAILED DESCRIPTION

[0009] The systems and methods discussed herein are directed to supporting adaptive resolution changes in video encoders, and more particularly, to implementing region-of-interest-aware coding methods that improve computational resource allocation and compression efficiency.

[0010] According to an example embodiment of the present disclosure implemented to be compatible with AVC, HEVC, VP9, ​​and such video coding standards that implement variable resolution frames, a frame may be subdivided into macroblocks ("MBs"), each having a size of 16x16 pixels, which may be further subdivided into partitions. According to an example embodiment of the present disclosure implemented to be compatible with the HEVC standard, a frame may be subdivided into coding tree units ("CTUs"), whose luminance ("luma") and chrominance ("chroma") components may be further subdivided into coding tree blocks ("CTBs"), which may be further subdivided into coding units ("CUs"). According to an example embodiment of the present disclosure implemented as other standards, a frame may be subdivided into units of NxN pixels, which may then be further subdivided into sub-units. For the purposes of the present disclosure, each of these largest subdivision units of a frame may be generally referred to as a "block."

[0011] According to an example embodiment of the present disclosure, a motion prediction coding format may refer to a data format in which image data of a video frame is compressed by encoding motion vector information and prediction information of a frame including one or more references to motion information and prediction units ("PUs") of one or more other frames. Motion information may refer to data describing the motion of a block structure of a frame or a unit or subunit thereof, such as a motion vector and a reference to a block of a current frame or another frame. A PU may refer to a unit or multiple subunits corresponding to a block structure among multiple block structures of a frame (such as an MB or CTU), wherein blocks are partitioned based on frame data and encoded according to an established video codec. The motion information corresponding to a PU may describe a motion prediction as encoded by any motion vector encoding tool, including but not limited to those described herein.

[0012] A video encoder according to motion prediction coding may obtain a frame from a video source and encode the frame to obtain a reconstructed frame that may be ultimately output for display. The reconstructed frame and blocks of the reconstructed frame may be intra-coded, where at least some of the motion information of the reconstructed frame refers to motion information elsewhere in the reconstructed frame, or inter-coded, where at least some of the motion information of the reconstructed frame refers to motion information of another frame. In general, frames and blocks thereof according to example embodiments of the present disclosure may be coded according to motion prediction for intra-coding or inter-coding, unless either is explicitly specified.

[0013] According to adaptive resolution change ("ARC") coding technology as proposed by Alibaba Group, a video encoder can encode frames from a video source in a compressed format in various resolutions and send the encoded frames in a variable resolution bitstream, but a video decoder can then obtain the encoded frames from the variable resolution bitstream and use the variable resolution frames to perform motion compensation to reconstruct the frames for display. Therefore, it should be understood that the video bitstream according to the example embodiments of the present disclosure can be variable resolution through the implementation of such technology, and the details of such technology are not repeated here.

[0014] A variable resolution bitstream according to ARC may include multiple frames of different resolutions, which contain image data all derived from the same source frame. Such a grouping of multiple frames may include an output frame having a resolution that is the same as the resolution of the original frame, and this output frame may be encoded to be output for display. Such a grouping of multiple frames may also include any number of reference frames, each having a resolution different from the resolution of the original frame, and these reference frames may be encoded but may not be ultimately output for display.

[0015] The video encoder according to an example embodiment of the present disclosure can generate a grouping of multiple frames of different resolutions all derived from the same source frame by upsampling or downsampling the image data of the source frame to various different resolutions. The upsampling or downsampling algorithm according to an example embodiment of the present disclosure may include interpolation, averaging, bilinear algorithms, training algorithms, or any other suitable algorithms.

[0016] A decoder according to the ARC implementation can be operated to decode an output frame with reference to any number of reference frames of different resolutions derived from the same original frame. The details of its implementation are not repeated here.

[0017] An exemplary embodiment of the present disclosure provides a method for improving the efficiency of compressing a frame to be output for display by encoding multiple frames of different resolutions all derived from the same source frame as reference frames. According to an exemplary embodiment of the present disclosure, a frame to be encoded can be divided into regions of pixels based on image data of the frame before encoding, thereby becoming multiple corresponding visual interest regions according to the image data in each corresponding region by a human audience. Figure 1 An example of dividing the frame 100 into regions according to corresponding visual interests according to an exemplary embodiment of the present disclosure is illustrated.

[0018] A frame may be divided into any number of multiple regions, which, for descriptive purposes, may be described in terms of their relative visual interest to one another. The visual interest of different regions may be related in terms of the relative frequencies of image content elements that tend to attract the attention of a human viewer. For example, a first region that contains sharp edges in its image content may have more visual interest than a second region that contains blurred edges or no edges in its image content. A first region that contains high-frequency component image content may have more visual interest than a second region that contains low-frequency component image content. A first region that contains foreground image content may have more visual interest than a second region that contains midground or background image content. A first region that contains text in its image content may have more visual interest than a second region that does not contain text in its image content.

[0019] Based on these and any number of other similar criteria for visual interest, the frame can be divided into two regions, three regions, or any other number of regions based on the relative visual interest between each region. Figure 1 As shown, region 102 has the lowest visual interest; region 104 has a higher visual interest than region 102; and region 106 has the highest visual interest.

[0020] The division of frames into regions can be performed based on a source frame that generates a grouping of multiple frames with different resolutions, and each frame of the grouping of multiple frames can be divided into correspondingly shaped and positioned regions proportional to its corresponding resolution. For each frame of such a grouping of frames, the number of regions can be equal to the number of frames in such a grouping of frames, thereby counting the output frames and each reference frame. For each different grouping of frames, the number of regions can be the same or different.

[0021] The determination of the number of regions into which a frame is divided can (as will be described subsequently) be based on the number of multiple frames of different resolutions to be generated; other methods of dividing a frame into regions and determining the number of frames into which a frame is divided other than those specified herein should be outside the scope of the present disclosure.

[0022] Figure 2 Illustrated are groups 200A and 200B of multiple frames having different resolutions all derived from the same source frame as generated by a video encoder according to an example embodiment of the present disclosure. Figure 2 The illustration is drawn around packet 200A and around packet 200B to illustrate boxes for each packet, but these boxes should not be considered to construct transmission of frames, such as in a bitstream. Although the frames of packets 200A and 200B are each shown as a sequence of consecutive frames, they are not necessarily encoded or output into a bitstream in that order or as consecutive frames.

[0023] Figure 2 The relationship between codes among the groups 200A and 200B according to an example embodiment of the present disclosure is further illustrated. Figure 2 The arrows illustrated in the figure may indicate that a frame is encoded with reference to the motion information of another frame, but do not necessarily indicate and limit the ordering of the corresponding frames: for example, in the bitstream, a predicted frame ("P frame") may refer only to the previous frame, but a bidirectional frame ("B frame") may refer to both the previous frame and the subsequent frame.

[0024] Figure 2 Frames 202, 204, and 206 (in order of ascending resolution) in group 200A and frames 208, 210, and 212 (in order of ascending resolution) in group 200B are illustrated. Output frames 206 and 212 are encoded to be output for display, whereas reference frames 202, 204, 208, and 210 are encoded but may not be output for display. Any of the illustrated frames may have a different resolution than any of the other illustrated frames. Figure 2 Frames illustrated as being adjacent to each other in FIG. 1 may be consecutive frames or may have additional frames therebetween that are not illustrated for the purposes of this disclosure.

[0025] According to an example embodiment of the present disclosure, output frames 206 and 212 may each be encoded with reference to each of the reference frames of their respective groups: that is, output frame 206 is encoded with reference to reference frames 202 and 204, while output frame 212 is encoded with reference to reference frames 208 and 210. In addition, each reference frame in the group may be encoded with reference to at least one reference frame having a relatively smaller resolution (except for the reference frame having the smallest resolution). For example, reference frame 204 is encoded with reference to reference frame 202, while reference frame 210 is encoded with reference to reference frame 208.

[0026] Frames in a packet may also be encoded with reference to frames outside the packet. For example, frame 202 may be encoded with reference to an arbitrary frame, or may be intra-coded without reference to any other frame. Similarly, frame 208 may be encoded with reference to an arbitrary frame (e.g., Figure 2As illustrated, frame 206) may be intra-coded, or may be unlimitedly intra-coded without reference to any other frame.

[0027] According to an example embodiment of the present disclosure, a video encoder may generate a grouping of multiple frames having different resolutions that are all derived from the same source frame as described above, and, for each different source frame, a different grouping of multiple frames (of the same number or different numbers) may be generated. The determination of the number of frames of different resolutions to be generated may be based (as will be described subsequently) on the number of regions into which each frame of the grouping is divided; other ways of determining the number of frames of different resolutions to be generated and ways of determining specific different resolutions other than those specified herein shall be outside the scope of the present disclosure.

[0028] The video encoder may then encode each of the groups of multiple frames according to a region of interest aware method as described herein.

[0029] Figure 3 A region of interest perceptual encoding method 300 according to an example embodiment of the present disclosure is illustrated.

[0030] At step 302, a video encoder obtains a source frame from a video source.

[0031] At step 304, the video encoder generates a grouping of a plurality of frames having different resolutions from the source frame. As described above, each frame of the grouping can be derived from the same image data of the source frame, and the number of the plurality of frames can be the same or different for each different grouping of the plurality of frames. Each frame of the grouping can be derived by upsampling or downsampling the image data of the source frame to various different resolutions.

[0032] The grouping of multiple frames includes output frames having the same resolution as the source frames. The grouping of multiple frames further includes any number of reference frames having resolutions different from the source frames. Any number of reference frames having resolutions different from each other can be generated in this manner. The number of reference frames of different resolutions to be generated can be predetermined, can be determined by a configurable setting of the video encoder, or can be determined by dividing each frame of the grouping into multiple regions as described below with respect to step 306.

[0033] In step 306, the video encoder determines a plurality of different relative visual interest regions to apply to each frame of the grouping. As described above, each frame of the grouping can be divided into a plurality of regions based on the respective visual interest of a human viewer to the image data in each respective region. Furthermore, as described above, the number of divisions and the manner of division can be different for each grouping. Furthermore, as described above, the division of the frames into regions can be performed based on the source frames, and each frame of the grouping can be divided into correspondingly shaped and positioned regions proportional to its respective resolution.

[0034] According to an exemplary embodiment of the present disclosure, the number of reference frames of different resolutions to be generated in the grouping may be determined based on the number of regions into which each frame of the grouping is divided, so that an additional reference frame of different resolutions is generated in the grouping for each additional region into which each frame of the grouping is divided. Alternatively, the number of regions into which each frame of the grouping is divided may be determined based on the number of reference frames of different resolutions to be generated in the grouping, so that for each additional reference frame of different resolutions to be generated in the grouping, each frame of the grouping is divided into one additional region.

[0035] Step 304 and step 306 may be performed in any order relative to each other. That is, a grouping of multiple frames may be generated first, followed by determination of regions, wherein the number of regions into which each frame is divided is determined based on the number of reference frames of different resolutions generated in the grouping of multiple frames. Alternatively, determination of the grouping may be performed first, followed by generation of a grouping of multiple frames, wherein the number of reference frames of different resolutions generated is determined based on the number of regions into which each frame is divided.

[0036] At step 308, the video encoder encodes the grouped reference frames using a different coding mode for at least one region of visual interest than for the key region of visual interest.

[0037] According to an example embodiment of the present disclosure, as will be described in further detail below with reference to step 310, an output frame may be encoded with reference to each reference frame of different resolutions. For each reference frame of different resolutions, the output frame may be encoded with reference to a specific visual interest region of the reference frame. The output frame may be encoded with reference to at least one visual interest region of the reference frame. In addition, the output frame may be encoded with reference to a different visual interest region for each reference frame (according to each reference frame being divided into correspondingly shaped and positioned regions, the "same" visual interest region may refer to a correspondingly shaped and positioned region across frames of different resolutions, while the "different" visual interest region may refer to a non-correspondingly shaped and positioned region across frames of different resolutions); these different visual interest regions will be referred to herein as "key regions". Although the output frame may be encoded with reference to more portions of any reference frame than just one visual interest region, the output frame is encoded with reference to at least a different key region of each reference frame.

[0038] For each reference frame, the video encoder may determine which key region to reference when encoding an output frame. This determination may be made based on the relative resolution of each reference frame in sequence and based on the relative visual interest of each region of visual interest in sequence. For example, for a reference frame of lowest resolution, a key region of visual interest of lowest visual interest may be referenced when encoding an output frame; for a reference frame of second lowest resolution, a key region of visual interest of second lowest visual interest may be referenced when encoding an output frame; and so on.

[0039] Therefore, reference Figure 1 and Figure 2 ,assumed Figure 1 The region of interest is applied to the group 200A, then for the reference frame 202, the region 102 of the reference frame 202 can be referenced when encoding the output frame 206; and for the reference frame 204, the region 104 of the reference frame 204 can be referenced when encoding the output frame 206. Alternatively, assuming Figure 1 If the region of interest is applied to group 200B, then for reference frame 208, region 102 of reference frame 208 may be referenced when encoding output frame 212; and for reference frame 210, region 104 of reference frame 210 may be referenced when encoding output frame 212. (However, Figure 1 The region of interest may not apply to both packet 200A and packet 200B.)

[0040] The video encoder may encode the key region using any suitable coding mode as supported by the motion prediction coding format. For example, the key region may be encoded using any suitable algorithm for the following: intra-frame prediction, inter-frame prediction, various forms of motion vector prediction (such as adaptive motion vector prediction ("AMVP"), advanced temporal motion vector prediction ("ATMV"), control point motion vector prediction ("CPMVP"), temporal motion vector prediction ("TMVP"), space-time motion vector prediction ("STMV"), sub-block temporal motion vector prediction ("SbTMVP"), etc.), intra-block copy ("IBC"), etc. The encoding of the key region may generally not be limited to any one of the coding modes according to the example embodiments of the present disclosure.

[0041] However, according to an exemplary embodiment of the present disclosure, the video encoder may encode each reference frame using a coding mode different from the coding mode used to encode the key region for at least one visual interest region or a coding mode different from the coding mode used to encode the key region for each visual interest region other than the key region. In particular, those other visual interest regions may be encoded by a minimal computationally intensive coding mode. For example, blocks of those other visual interest regions may be encoded by a skip mode or a merge mode, where the motion information of the region is copied from the motion information of the blocks of another reference frame. The copied motion information may be further resized and resampled in proportion to the difference between the resolution of the current reference frame and the resolution of the other reference frame.

[0042] Since it is known that the output frame is encoded with reference to the key region, any other visual interest region not used to encode the output frame can be encoded by the least computationally intensive encoding mode to minimize the computational overhead of the encoding process. According to an example embodiment of the present disclosure, each visual interest region can be encoded by the least computationally intensive encoding mode.

[0043] Therefore, reference Figure 1 and Figure 2 ,assumed Figure 1 The region of interest is applied to group 200A, then for reference frame 202, region 102 of reference frame 202 may be a key region and may be encoded by any suitable coding mode without restriction, while regions 104 and 106 of reference frame 202 may be encoded by the least computationally intensive coding mode; and for reference frame 204, region 104 of reference frame 204 may be a key region and may be encoded by any suitable coding mode without restriction, while regions 102 and 106 of reference frame 204 may be encoded by the least computationally intensive coding mode. Alternatively, assuming Figure 2The region of interest is applied to group 200B, then for reference frame 208, region 102 of reference frame 208 may be a key region and may be encoded by any suitable coding mode without restriction, while regions 104, 106 and 106 of reference frame 208 may be encoded by the least computationally intensive coding mode; and for reference frame 210, region 104 of reference frame 210 may be a key region and may be encoded by any suitable coding mode without restriction, while regions 102 and 106 of reference frame 210 may be encoded by the least computationally intensive coding mode. (However, Figure 1 The region of interest may not apply to both packet 200A and packet 200B.)

[0044] At step 310, the video encoder encodes the grouped output frames using a different coding mode for at least one non-highest visual interest region than for the highest visual interest region.

[0045] The video encoder may encode the highest visual interest region using any suitable coding mode as supported by the motion prediction coding format. For example, the highest visual interest region may be encoded using any suitable algorithm for the following: intra-frame prediction, inter-frame prediction, various forms of motion vector prediction (such as adaptive motion vector prediction ("AMVP"), advanced temporal motion vector prediction ("ATMV"), control point motion vector prediction ("CPMVP"), temporal motion vector prediction ("TMVP"), space-time motion vector prediction ("STMV"), sub-block temporal motion vector prediction ("SbTMVP"), etc.), intra-block copy ("IBC"), etc. The encoding of the highest visual interest region may generally not be limited to any one of the coding modes according to the example embodiments of the present disclosure.

[0046] However, according to an exemplary embodiment of the present disclosure, the video encoder may encode the output frame using a different encoding mode for at least one non-highest visual interest region than the encoding mode used to encode the highest visual interest region, or using a different encoding mode for each non-highest visual interest region than the encoding mode used for the highest visual interest region. In particular, those non-highest visual interest regions may be encoded by a less computationally intensive encoding mode than the encoding mode used for the highest visual interest region.

[0047] For example, blocks of non-highest visual interest regions of an output frame may be encoded by simplifying the inter-frame coded motion prediction mode based on blocks of key regions of one of the grouped reference frames. As described above, the output frame is encoded with reference to different key regions of at least each reference frame; therefore, each region of the output frame is encoded with reference to different key regions of different reference frames. In addition, since each reference frame is derived from the same source frame as the output frame, it can be expected that no residual information (generated from calculating the difference between the output frame and the reference frame) is required to transform the motion information of the reference frame to the motion information of the source frame. Therefore, the simplification of the inter-frame coded motion prediction mode can be characterized by not calculating the residual information between the output frame and the reference frame when encoding the output frame and then not sending it with the output frame. In addition, the coded motion information size can also be adjusted and resampled in proportion to the difference between the resolution of the output frame and the resolution of the reference frame. In other aspects, the inter-frame coded motion prediction mode can be implemented according to the knowledge of those skilled in the art.

[0048] At step 312, the video encoder outputs the grouped coded frames to a bitstream. The coded frames may include output frames and reference frames. The coded frames do not retain information about the division of each frame into regions.

[0049] Figure 4 An example system 400 is illustrated for implementing the above-described processes and methods for implementing support for region of interest perceptual coding methods in a video encoder.

[0050] The techniques and mechanisms described herein may be implemented by multiple instances of the system 400, as well as by any other computing devices, systems, and / or environments. Figure 4 The system 400 shown is only one example of a system and is not intended to suggest any limitation on the scope of use or functionality of any computing device utilized to perform the above-described processes and / or procedures. Other well-known computing devices, systems, environments, and / or configurations that may be suitable for use with embodiments include, but are not limited to, personal computers, server computers, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set-top boxes, game consoles, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above-described systems or devices, implementations using field programmable gate arrays ("FPGAs") and application specific integrated circuits ("ASICs"), and / or the like.

[0051] System 400 may include one or more processors 402 and a system memory 404 communicatively coupled to processor 402. Processor 402 may run one or more modules and / or processes to enable processor 402 to perform various functions. In some embodiments, processor 402 may include a central processing unit ("CPU"), a graphics processing unit ("GPU"), both a CPU and a GPU, or other processing units or components known in the art. Additionally, each processor 402 may have its own local memory, which may also store program modules, program data, and / or one or more operating systems.

[0052] System memory 404 may be volatile, such as RAM, non-volatile, such as ROM, flash memory, micro hard drive, memory card, etc., or some combination thereof, depending on the exact configuration and type of system 400. System memory 404 may include one or more computer executable modules 406 that may be executed by processor 402.

[0053] Module 406 may include but is not limited to encoder module 408. Encoder module 408 also includes source frame acquisition submodule 410, frame generation submodule 412, region determination submodule 414, reference frame encoding submodule 416, output frame encoding submodule 418 and frame output submodule 420.

[0054] The encoder module 408 may be configured to perform motion prediction encoding on frames from a video source through any of the algorithms and processes described herein, including the functionality of each of the sub-modules described herein.

[0055] The source frame acquisition submodule 410 may be configured to obtain a source frame from a video source, as described above with reference to Figure 3 As described.

[0056] The frame generation submodule 412 may be configured to generate a group of multiple frames having different resolutions from a source frame, as described above with reference to Figure 3 As described.

[0057] The region determination submodule 414 may be configured to determine each frame of the grouping into a plurality of different relative visual interest regions, as described above with reference to Figure 3 As described.

[0058] The reference frame encoding submodule 416 may be configured to encode the grouped reference frames using a different encoding mode for at least one visual interest region than for the key visual interest region, as described above with reference to Figure 3 As described.

[0059] The output frame encoding submodule 418 may be configured to encode the grouped output frames using a different encoding mode for at least one non-highest visual interest region than for the highest visual interest region, as described above with reference to Figure 3 As described.

[0060] The frame output submodule 420 may be configured to output the packetized encoded frames into a bitstream as described above with reference to Figure 3 As described.

[0061] The system 400 may additionally include an input / output ("I / O") interface 440 for receiving a sequence of frames from the video source data, and for outputting reconstructed frames to a reference frame buffer and / or a transmission buffer. The system 400 may also include a communication module 450, allowing the system 400 to communicate with other devices (not shown) over a network (not shown). The network may include the Internet, wired media such as a wired network or a direct wired connection, and wireless media such as acoustic, radio frequency ("RF"), infrared, and other wireless media.

[0062] Some or all of the operations of the above methods can be performed by executing computer readable instructions stored on a computer readable storage medium as defined below. The term "computer readable instructions" as used in the specification and claims includes routines, applications, application modules, program modules, programs, components, data structures, algorithms, etc. The computer readable instructions can be implemented on a variety of system configurations, including single-processor or multi-processor systems, minicomputers, mainframe computers, personal computers, handheld computing devices, microprocessor-based systems, programmable consumer electronics, combinations thereof, etc.

[0063] Computer-readable storage media may include volatile memory (such as random access memory ("RAM")) and / or non-volatile memory (such as read-only memory ("ROM"), flash memory, etc.). Computer-readable storage media may also include additional removable and / or non-removable storage devices, including but not limited to flash memory, magnetic storage devices, optical storage devices, and / or tape storage devices that can provide non-volatile storage of computer-readable instructions, data structures, program modules, etc.

[0064] Non-transient computer-readable storage media are examples of computer-readable media. Computer-readable media include at least two types of computer-readable media, namely computer-readable storage media and communication media. Computer-readable storage media include volatile and non-volatile, removable and non-removable media implemented in any process or technology for storing information such as computer-readable instructions, data structures, program modules or other data. Computer-readable storage media include, but are not limited to, phase change memory ("PRAM"), static random access memory ("SRAM"), dynamic random access memory ("DRAM"), other types of random access memory ("RAM"), read-only memory ("ROM"), electrically erasable programmable read-only memory ("EEPROM"), flash memory or other memory technology, compact disk read-only memory ("CD-ROM"), digital versatile disk ("DVD") or other optical storage device, magnetic cassette, tape, magnetic disk storage device or other magnetic storage device, or any other non-transmission medium capable of storing information for access by a computing device. In contrast, communication media can embody computer-readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier or other transmission mechanism. The computer-readable storage medium employed herein should not be construed as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (such as light pulses through a fiber optic cable), or an electrical signal propagating through wires.

[0065] The computer readable instructions stored on one or more non-transitory computer readable storage media, when executed by one or more processors, may perform the above-referenced Figure 1-4 Typically, computer-readable instructions include routines, programs, objects, components, data structures, etc. that perform specific functions or implement specific abstract data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations can be combined in any order and / or in parallel to implement the process.

[0066] Through the above technical solution, the present disclosure provides resolution adaptive video coding supported by the region of interest perception coding mode of the video encoder, so that the encoder can encode the motion information of the output frame for display based on reference frames of different resolutions derived from the same image data, and each frame is compressed by different methods based on the region of interest of the source image data, so as to save computing resources and improve compression efficiency. The method and system described in this article provide a video encoder, which: obtains a source frame from a video source; generates a group of multiple frames with different resolutions from the source frame; determines to divide each frame of the group into multiple different relative visual interest regions; encodes the reference frame of the group using a different coding mode for at least one visual interest region than for a key visual interest region; encodes the output frame of the group using a different coding mode for at least one non-highest visual interest region than for a highest visual interest region; and outputs the encoded frame of the group to a bitstream.

[0067] Sample Clauses

[0068] A. A method, the method comprising: obtaining a source frame from a video source; generating a group of multiple frames with different resolutions from the source frame; determining multiple different relative visual interest regions, the multiple regions being applied to each frame of the group; encoding a reference frame of the group using a different encoding mode for at least one visual interest region among the multiple regions than for a key visual interest region among the multiple regions; and encoding an output frame of the group using a different encoding mode for at least one non-highest visual interest region among the multiple regions than for a highest visual interest region among the multiple regions.

[0069] B. The method of paragraph A, wherein each frame of the group is derived by upsampling or downsampling the image data of the source frame to a different resolution.

[0070] C. The method of paragraph A, wherein the grouping includes an output frame having the same resolution as the source frame and at least one reference frame, each reference frame having a different resolution.

[0071] D. A method as described in paragraph C, wherein the resolution of each reference frame is different from the resolution of the source frame.

[0072] E. The method of paragraph C, wherein the number of reference frames of different resolutions to be generated in the group is determined based on the number of the multiple regions applied to each frame of the group.

[0073] F. The method of paragraph C, wherein the number of the plurality of regions applied to each frame of the group is determined based on the number of reference frames of different resolutions to be generated in the group.

[0074] G. The method of paragraph A, wherein the relative visual interest of each of the plurality of regions is based on the respective visual interest of the image data of the source frame in each respective region.

[0075] H. A method as described in paragraph A, wherein the output frame is encoded with reference to at least one visual interest region among the multiple regions applied to each reference frame of different resolutions.

[0076] I. A method as described in paragraph H, wherein the output frame is encoded with reference to different key areas among the multiple areas applied to each reference frame of different resolutions.

[0077] J. A method as described in paragraph I, wherein, for each reference frame of different resolution, the key area to be referenced when encoding the output frame is determined in sequence based on the relative resolution of each reference frame and in sequence based on the relative visual interest of each visual interest region.

[0078] K. A method as described in paragraph I, wherein each reference frame is encoded using a coding mode different from the coding mode used to encode the key area for at least one visual interest area among the multiple areas.

[0079] L. The method of paragraph K, wherein each reference frame is encoded using a coding mode different from the coding mode used to encode the key region for each region of visual interest among the plurality of regions other than the key region.

[0080] M. A method as described in paragraph K, wherein the different encoding mode is the least computationally intensive encoding mode.

[0081] N. A method as described in paragraph M, wherein the different encoding mode is a skip mode or a merge mode.

[0082] O. A method as described in paragraph I, wherein the output frame is encoded using a coding mode different from the coding mode used to encode the highest visual interest region for at least one non-highest visual interest region among the multiple regions.

[0083] P. A method as described in paragraph O, wherein the output frame is encoded using a coding mode different from the coding mode used to encode the highest visual interest region among the multiple regions.

[0084] Q. A method as described in paragraph O, wherein the different encoding mode is a less computationally intensive encoding mode than the encoding mode used for the highest visual interest area.

[0085] R. The method as described in paragraph Q, wherein the different coding mode is an inter-frame coding motion prediction mode that is simplified by not calculating residual information between the output frame and the reference frame.

[0086] S. A system, the system comprising: one or more processors; and a memory, the memory being communicatively coupled to the one or more processors, the memory storing computer executable modules executable by the one or more processors, the computer executable modules performing related operations when executed by the one or more processors, the computer executable modules comprising: an encoder module, the encoder module further comprising: a frame acquisition submodule, the frame acquisition submodule being configured to obtain a source frame from a video source; a frame generation submodule, the frame generation submodule being configured to generate a group of multiple frames with different resolutions from the source frame; an area determination submodule, the area determination submodule A stator module is configured to determine a plurality of different relative visual interest regions, the plurality of regions being applied to each frame of the group; a reference frame encoding submodule, the reference frame encoding submodule being configured to encode the reference frame of the group using a different encoding mode for at least one visual interest region among the plurality of regions and for a key visual interest region among the plurality of regions; and an output frame encoding submodule, the output frame encoding submodule being configured to encode the output frame of the group using a different encoding mode for at least one non-highest visual interest region among the plurality of regions and for a highest visual interest region among the plurality of regions.

[0087] T. A system as described in paragraph S, wherein the frame generation submodule is configured to derive each frame of the group by upsampling or downsampling the image data of the source frame to a different resolution.

[0088] U. A system as described in paragraph S, wherein the group includes an output frame having the same resolution as the source frame and at least one reference frame, each reference frame having a different resolution.

[0089] V. A system as described in paragraph U, wherein the resolution of each reference frame is different from the resolution of the source frame.

[0090] W. A system as described in paragraph U, wherein the region determination submodule is configured to determine a number of reference frames of different resolutions to be generated in the group based on the number of the multiple regions applied to each frame of the group.

[0091] X. A system as described in paragraph U, wherein the region determination submodule is configured to determine the number of the multiple regions applied to each frame of the group based on the number of reference frames of different resolutions to be generated in the group.

[0092] Y. A system as described in paragraph S, wherein the relative visual interest of each of the plurality of regions is based on the respective visual interest of the image data of the source frame in each respective region.

[0093] Z. A system as described in paragraph S, wherein the output frame encoding submodule is configured to encode the output frame with reference to at least one visual interest region among the multiple regions applied to each reference frame of different resolutions.

[0094] AA. The system of paragraph Z, wherein the output frame encoding submodule is configured to encode the output frame with reference to different key areas among the multiple areas applied to at least one reference frame of different resolutions.

[0095] BB. A system as described in paragraph AA, wherein the output frame encoding submodule is configured to determine, for each reference frame of different resolutions, a key area to be referenced when encoding the output frame based on the relative resolution of each reference frame and based on the relative visual interest of each visual interest area in sequence.

[0096] CC. A system as described in paragraph AA, wherein the reference frame encoding submodule is configured to encode each reference frame using a coding mode different from a coding mode used to encode the key area for at least one visual interest area among the multiple areas.

[0097] DD. A system as described in paragraph CC, wherein the reference frame encoding submodule is configured to encode each reference frame using a coding mode different from the coding mode used to encode the key area for each visual interest area among the multiple areas other than the key area.

[0098] EE. A system as described in paragraph CC, wherein the different encoding mode is a least computationally intensive encoding mode.

[0099] FF. A system as described in paragraph EE, wherein the different encoding mode is a skip mode or a merge mode.

[0100] GG. A system as described in paragraph Z, wherein the output frame encoding submodule is configured to encode the output frame using a coding mode different from the coding mode used to encode the highest visual interest region among the multiple regions.

[0101] HH. A system as described in paragraph GG, wherein the output frame encoding submodule is configured to encode the output frame using a coding mode different from the coding mode used to encode the highest visual interest region among the multiple regions.

[0102] II. A system as described in paragraph GG, wherein the different encoding mode is a less computationally intensive encoding mode than the encoding mode used for the highest visual interest area.

[0103] JJ. A system as described in paragraph II, wherein the different coding mode is an inter-frame coding motion prediction mode that is simplified by not calculating residual information between the output frame and the reference frame.

[0104] KK. A computer-readable storage medium storing computer-readable instructions executable by one or more processors, wherein when executed by the one or more processors, the computer-readable instructions cause the one or more processors to perform operations, the operations comprising: obtaining a source frame from a video source; generating a group of multiple frames with different resolutions from the source frame; determining multiple different relative visual interest regions, the multiple regions being applied to each frame of the group; encoding a reference frame of the group using a different encoding mode for at least one visual interest region among the multiple regions than for a key visual interest region among the multiple regions; and encoding an output frame of the group using a different encoding mode for at least one non-highest visual interest region among the multiple regions than for a highest visual interest region among the multiple regions.

[0105] LL. A computer-readable storage medium as described in paragraph KK, wherein each frame of the group is derived by upsampling or downsampling image data of the source frame to a different resolution.

[0106] MM. A computer-readable storage medium as described in paragraph KK, wherein the grouping includes an output frame having the same resolution as the source frame and at least one reference frame, each reference frame having a different resolution.

[0107] NN. A computer-readable storage medium as described in paragraph MM, wherein the resolution of each reference frame is different from the resolution of the source frame.

[0108] OO. A computer-readable storage medium as described in paragraph MM, wherein the number of reference frames of different resolutions to be generated in the group is determined based on the number of the multiple regions applied to each frame of the group.

[0109] PP. A computer-readable storage medium as described in paragraph MM, wherein the number of the multiple regions applied to each frame of the group is determined based on the number of reference frames of different resolutions to be generated in the group.

[0110] QQ. A computer-readable storage medium as described in paragraph KK, wherein the relative visual interest of each of the multiple regions is based on the corresponding visual interest of the image data of the source frame in each corresponding region.

[0111] RR. A computer-readable storage medium as described in paragraph KK, wherein the output frame is encoded with reference to at least one visual interest region among the multiple regions applied to each reference frame of different resolutions.

[0112] SS. A computer-readable storage medium as described in paragraph RR, wherein the output frame is encoded with reference to a different key area among the multiple areas applied to at least one reference frame of different resolutions.

[0113] TT. A computer-readable storage medium as described in paragraph SS, wherein, for each reference frame of different resolutions, the key regions to be referenced when encoding the output frame are determined sequentially based on the relative resolution of each reference frame and sequentially based on the relative visual interest of each visual interest region.

[0114] UU. A computer-readable storage medium as described in paragraph SS, wherein each reference frame is encoded using a coding mode different from a coding mode used to encode the key region for at least one visual interest region among the plurality of regions.

[0115] VV. A computer-readable storage medium as described in paragraph UU, wherein each reference frame is encoded using a coding mode different from the coding mode used to encode the key region for each visual interest region among the multiple regions other than the key region.

[0116] WW. A computer-readable storage medium as described in paragraph UU, wherein the different encoding mode is a least computationally intensive encoding mode.

[0117] XX. A computer-readable storage medium as described in paragraph WW, wherein the different encoding mode is a skip mode or a merge mode.

[0118] YY. A computer-readable storage medium as described in paragraph SS, wherein the output frame is encoded using a coding mode different from the coding mode used to encode the highest visual interest region for at least one non-highest visual interest region among the multiple regions.

[0119] ZZ. A computer-readable storage medium as described in paragraph YY, wherein the output frame is encoded using a different encoding mode for each non-highest visual interest region among the multiple regions than the encoding mode used to encode the highest visual interest region.

[0120] AAA. The computer-readable storage medium of paragraph YY, wherein the different encoding mode is a less computationally intensive encoding mode than the encoding mode used for the region of highest visual interest.

[0121] BBB. A computer-readable storage medium as described in paragraph AAA, wherein the different encoding mode is an inter-frame coding motion prediction mode that is simplified by not calculating residual information between the output frame and the reference frame.

[0122] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described. Rather, the specific features and acts are disclosed as example forms of implementing the claims.

Claims

1. A coding method, the method comprising: generating a grouping of a plurality of frames having different resolutions; determining a plurality of different relative visual interest regions, the plurality of regions being applicable to each frame of the grouping; encoding the grouped reference frames using, for at least one region of visual interest, a coding mode different from a coding mode used to encode the key region; encoding an output frame of the group with reference to, for each reference frame of a different resolution in the group, at least a key region of the reference frame that is different from a key region referenced for each other reference frame; as well as The grouped output frames are encoded using different encoding modes for at least one non-highest visual interest region and the highest visual interest region. 2 . The method of claim 1 , wherein the grouping comprises at least one reference frame, each reference frame having a different resolution from one another. 3 . The method according to claim 1 , wherein the different encoding mode is an inter-frame coding motion prediction mode that is simplified by not calculating residual information between the output frame and the reference frame.

4. A coding system, the system comprising: one or more processors; as well as A memory, the memory being communicatively coupled to the one or more processors, the memory storing computer executable modules executable by the one or more processors, the computer executable modules performing related operations when executed by the one or more processors, the computer executable modules comprising: The encoder module further comprises: a frame generation submodule, the frame generation submodule being configured to generate a group of a plurality of frames having different resolutions; a region determination submodule, the region determination submodule being configured to determine a plurality of different relative visual interest regions, the plurality of regions being applied to each frame of the group; a reference frame encoding submodule configured to encode the grouped reference frames using a coding mode different from a coding mode used to encode the key region for at least one visual region of interest; An output frame encoding submodule is configured to, for each reference frame of different resolutions in the group, encode the output frame of the group with reference to a key area of ​​the reference frame that is at least different from the key area referenced for each other reference frame; and encode the output frame of the group using different encoding modes for at least one non-highest visual interest region and the highest visual interest region.

5. The system of claim 4, wherein the grouping includes at least one reference frame, each reference frame having a different resolution from one another.

6. The system of claim 4, wherein the different encoding mode is an inter-frame coding motion prediction mode that is simplified by not calculating residual information between the output frame and the reference frame.

7. A computer-readable storage medium storing computer-readable instructions executable by one or more processors, wherein when executed by the one or more processors, the computer-readable instructions cause the one or more processors to perform operations comprising: generating a grouping of a plurality of frames having different resolutions; determining a plurality of different relative visual interest regions, the plurality of regions being applicable to each frame of the grouping; encoding the grouped reference frames using, for at least one region of visual interest, a coding mode different from a coding mode used to encode the key region; encoding an output frame of the group with reference to, for each reference frame of a different resolution in the group, at least a key region of the reference frame that is different from a key region referenced for each other reference frame; as well as The grouped output frames are encoded using different encoding modes for at least one non-highest visual interest region and the highest visual interest region.

8. The computer-readable storage medium of claim 7, wherein the output frame is encoded with reference to at least one region among a plurality of regions of each reference frame of different resolutions.

9. The computer-readable storage medium of claim 7, wherein the different encoding mode is an inter-frame coding motion prediction mode that is simplified by not calculating residual information between the output frame and the reference frame.

Images