METHOD AND SYSTEM FOR VIDEO CODING USING REFERENCE REGIONS - Patent application

Abstract

The decoder includes circuitry configured to receive a bitstream, identify a first frame, find a first independent reference region in the first frame, extract the first independent reference region from the first frame, and decode the second frame using the first independent reference region as a reference for the second frame.

Description

[Technical Field]

[0001] (Related Applications) This application claims the benefit of priority to U.S. Non-Provisional Application No. 17 / 229,957, filed April 14, 2021, entitled "METHODS AND SYSTEMS OF VIDEO CODING USING REFERENCE REGIONS," which is incorporated herein by reference in its entirety, and to U.S. Provisional Application No. 63 / 009,978, filed April 14, 2020, entitled "METHODS AND SYSTEMS OF VIDEO CODING USING REFERENCE REGIONS," which is incorporated herein by reference in its entirety.

[0002] The present invention relates generally to the field of video compression, and more particularly to a method and system for video coding using reference domains.

[0003] A video codec may include electronic circuitry or software that compresses or decompresses digital video. It may convert uncompressed video into a compressed format, or vice versa. In the context of video compression, a device that compresses video (and / or performs some of the functions) may typically be called an encoder, and a device that decompresses video (and / or performs some of the functions) may be called a decoder.

[0004] The format of the compressed data may conform to standard video compression specifications. The compression may be lossy, in that the compressed video lacks some information present in the original video. Consequences of this may include that the decompressed video may have lower quality than the original uncompressed video, because there is insufficient information to accurately reconstruct the original video.

[0005] There can be a complex relationship between video quality, the amount of data used to represent the video (e.g., as determined by bitrate), the complexity of the encoding and decoding algorithms, sensitivity to data loss and errors, ease of editing, random access, end-to-end delay (e.g., latency), etc.

[0006] Motion compensation may involve an approach to predicting a video frame or a portion thereof given a reference frame, such as a previous and / or future frame, by considering the motion of the camera and / or objects in the video. It may be employed in encoding and decoding video data for video compression, such as encoding and decoding using the Motion Picture Experts Group (MPEG) advanced video coding (AVC) standard (also known as H.264). Motion compensation may describe a picture in terms of a transformation from a reference picture to a current picture. The reference picture may be temporally earlier than the current picture or may be from a future date compared to the current picture. Compression efficiency may be improved when an image can be accurately synthesized from previously transmitted and / or stored images. Summary of the Invention

[0007] In one aspect, a decoder includes circuitry configured to: receive an encoded video bitstream, the encoded video stream including an encoded reference picture and an encoded current picture having a first size; decode the reference picture; identify a sub-region of the reference picture from the bitstream, the sub-region having a second size, the second size different from the first size; rescale the sub-region to a third size to form a rescaled reference picture, the third size equal to the first size; and decode the current picture using the rescaled reference picture.

[0008] In another aspect, a decoder includes circuitry configured to receive an encoded video bitstream including an encoded first reference picture and an encoded current picture, decode the reference picture, identify a first sub-region of the reference picture from the bitstream, transform the first sub-region to form a second reference picture, and decode the current picture using the second reference picture.

[0009] In another aspect, a method of video coding using reference regions includes receiving, by a decoder, an encoded video bitstream, the encoded video stream including an encoded reference picture and an encoded current picture having a first size; decoding, by the decoder, the reference picture; identifying, by the decoder and from the bitstream, a sub-region of the reference picture, the sub-region having a second size, the second size different from the first size; rescaling, by the decoder, the sub-region to a third size to form a rescaled reference picture, the third size equal to the first size; and decoding, by the decoder, the current picture using the rescaled reference picture.

[0010] In another aspect, a decoder includes circuitry configured to receive a bitstream, identify a first frame, find a first independent reference region in the first frame, extract the first independent reference region from the first frame, and decode the second frame using the first independent reference region as a reference for the second frame.

[0011] In another aspect, a method for video encoding using reference regions includes receiving a bitstream, identifying a first frame, locating a first independent reference region in the first frame, extracting the first independent reference region from the first frame, and decoding the second frame using the first independent reference region as a reference for the second frame.

[0012] These and other aspects and features of non-limiting embodiments of the present invention will become apparent to those of ordinary skill in the art upon review of the following description of specific non-limiting embodiments of the present invention in conjunction with the accompanying drawings. [Brief explanation of the drawings]

[0013] For the purpose of illustrating the invention, the drawings show aspects of one or more embodiments of the invention, it being understood, however, that the invention is not limited to the precise arrangements and instrumentalities shown in the drawings. [Figure 1] FIG. 1 is a block diagram illustrating one embodiment of decoding using reference frames. [Figure 2] FIG. 2 is an illustration of an exemplary embodiment of a reference frame with an independent reference region. [Figure 3] FIG. 3 is an illustration of an example embodiment of independent reference regions and predicted frames. [Figure 4] FIG. 4 is an illustration of an example embodiment of independent reference regions and predicted frames. [Figure 5] FIG. 5 is an illustration of an example embodiment of independent reference regions and predicted frames. [Figure 6] FIG. 6 is an illustration of an exemplary embodiment of an LTR buffer. [Figure 7] FIG. 7 is a process flow diagram illustrating an exemplary process for decoding video according to some implementations of the present subject matter. [Figure 8]FIG. 8 is a system block diagram illustrating an exemplary decoder capable of decoding a bitstream in accordance with some implementations of the present subject matter. [Figure 9] FIG. 9 is a process flow diagram illustrating an exemplary process for encoding video, consistent with some implementations of the present subject matter. [Figure 10] FIG. 10 is a system block diagram illustrating an exemplary video encoder according to some implementations of the present subject matter. [Figure 11] FIG. 11 is a block diagram of a computing system that may be used to implement any one or more of the methodologies disclosed herein, and any one or more portions thereof.

[0014] The drawings are not necessarily to scale and may be illustrated by phantom lines, diagrammatic representations, and partial views. In some cases, details that are not necessary for understanding the embodiments or that make other details difficult to grasp may be omitted. DETAILED DESCRIPTION OF THE INVENTION

[0015] In conventional video coding schemes, a video sequence is divided into Group-of-Pictures (GOPs). Each GOP is self-contained in terms of temporal and spatial prediction. Typically, the first picture in the group is used as a reference picture for subsequent pictures. The temporal and spatial relationships between pictures allow for very efficient compression using predictive coding.

[0016] 1, each GOP may include reference frames 104 or intra-frames (I-frames) used as references and information usable to predict other frames 108 from the references. The information usable for prediction may include, but is not limited to, global and / or local motion vectors and / or transforms, as well as residuals, as further described. Transmission of the reference frames 104 or I-frames may represent a substantial portion of the bandwidth used to transmit the GOP.

[0017] In some embodiments, long-term reference (LTR) frames may be used to reduce transmission bandwidth and / or improve decoding and / or encoding efficiency. As used in this disclosure, an LTR frame is a frame and / or picture in one or more groups of pictures (GOPs) that is used to create predicted frames and / or pictures, but which may not itself appear in video pictures. A frame marked as an LTR frame in a video bitstream may be available as a reference until explicitly deleted by bitstream signaling. LTR frames may improve prediction and compression efficiency in scenes with static backgrounds over long periods of time (e.g., video conference backgrounds or parking lot surveillance videos).

[0018] Current standards such as H.264 and H.265 allow updating of similar frames, such as LTR frames, by signaling that a newly decoded frame is stored and made available as a reference frame 104. Such updates are signaled by the encoder, and the entire frame is updated. However, updating the entire frame can be costly, especially when only a small portion of a static background has changed.

[0019] Referring now to FIG. 2 , embodiments disclosed herein improve the efficiency and flexibility of the prediction process described above by performing prediction using at least a reference region of the reference frame 104 as a reference for the current frame. The reference region or “sub-region” has a size, which may include an area smaller than the area of ​​the reference frame 104, e.g., defined in pixels. In contrast to current encoding standards in which a predicted frame is generated from the entire reference frame 104, the above-described approach may enable a decoder to perform decoding operations more efficiently and with greater variation. At least the sub-region 204 may be used at any position within a GOP and for any number of frames, thus eliminating the requirement for re-encoding and / or re-transmission of I-frames.

[0020] 2, an example embodiment of a reference frame 104 having a sub-region 204 representing a cropped portion of a video view is shown. The sub-region 204 may be identified within the reference frame 104, for example, by a decoder, as described in further detail below. The sub-region 204 may be used as a reference region in a manner similar to the use of the cropped reference frame 104, without requiring that the cropped reference frame 104 be transmitted separately.

[0021] Also referring to FIG. 2 , as one non-limiting, illustrative example, the reference frame 104 may have a first resolution defined by a first width D1 and a first height D2, where D1 and D2 may be numbers of units of measurement, such as, but not limited to, pixels and / or fractions of pixels. The area of ​​the reference frame 104 may be defined as the area of ​​a rectangular array of units of measurement D1 and D2, defined as the resolution D1×D2. The subregion 204 may have a width W and a height H, defining an area or resolution of W×H. The subregion 204 may define a subpicture within the reference picture having the same or smaller dimensions, where “smaller dimensions” means that at least one of W and H is smaller than the corresponding dimension of the reference frame. In other words, either W is smaller than D1 or H is smaller than D2. As a result, the resolution or area W×H may be smaller than the resolution or area D1×D2. The subregion 204 may be defined by a 4-tuple (X, Y, W, H), where X and Y are the coordinates of the upper left corner of the subregion 204 relative to the upper left corner of the reference picture, and W and H are the width and height of the subregion 204 expressed in units of measure. Note that alternative 4-tuple may be selected to define the subregion 204, such as, but not limited to, alternative corner coordinates of the subregion 204, a set of two diagonally opposite vertices, and / or a vector to any defined point. The data defining the subregion 204 may be static across a GOP. For example, the 4-tuple (X, Y, W, H) or equivalent may be static across a GOP. Alternatively, or additionally, the data defining the subregion 204 may be dynamic. For example, but not limited to, the subregion 204 changes between subsequent pictures of a GOP to follow the movement of an object and / or person of interest in the video picture. This may be coded similarly to the motion vectors and / or transforms generally used in video coding. Data defining a sub-region 204 may be provided for each picture in the group of pictures.This may be achieved, for example and without limitation, by a set of data defining sub-regions 204 in one picture, by further data describing the movement of sub-regions 204 from one picture to a previous or subsequent picture, or by a set of data defining sub-regions 204 for each picture of a group of pictures, as described above. The data defining sub-regions 204 may be specified and / or signaled in a sequence parameter set (SPS). Updated data defining sub-regions 204 may be provided in a picture parameter set (PPS) for one or more selected pictures and / or frames of the GOP.

[0022] Continuing with reference to FIG. 2, the decoder may be receiving, about to receive, or may have already received a reference frame at resolution D1×D2 and may select a subregion 204 using a 4-tuple, as described above. In some implementations, the encoder may use extra bits in the bitstream to signal geometric characteristics of the subregion 204 to the decoder. The signaling bits may indicate an index identifying the reference frame 104 index and / or GOP in a buffer, such as an LTR buffer and / or reference buffer, a picture index identifying the decoder, and a 4-tuple for the subregion 204, as described in further detail below. The decoder may then extract the subregion 204 as an independent reference region. Subsequent frames may be predicted from the extracted independent reference region. If the data defining the subregion 204 is dynamic, as described above, subsequent frames may also be predicted using such data and the reference region. Advantageously, a single reference region may be used for a subregion 204 that moves relative to the picture without requiring retransmission of the reference region. Alternatively or additionally, the size and / or position of the sub-region 204, reference frame 104, etc. may be characterized using parameters, such as height offset, height, length offset, and / or length, which may be signaled in the bitstream.

[0023] 2, the sub-region 204 may be signaled using at least one vertical offset and at least one horizontal offset. For example, without limitation, as described above, a 4-tuple may specify a vertical offset from the top of the frame, a vertical offset from the bottom of the frame, a horizontal offset from the left edge of the frame, and a horizontal offset from the right edge of the frame, where the offsets may be measured in pixels of the frame either before or after rescaling, as described in further detail below. As one non-limiting example, the at least one vertical offset may include sps_conf_win_top_offset and sps_conf_win_bottom_offset, which may be signaled in the SPS and may specify a vertical offset from the top of the frame and a vertical offset from the bottom of the frame, respectively. As a further non-limiting example, the at least one horizontal offset may include sps_conf_win_left_offset and sps_conf_win_right_offset, which may be signaled in the SPS and may specify a horizontal offset from the left edge of the frame and a horizontal offset from the right edge of the frame, respectively.

[0024] Continuing with reference to FIG. 2, alternatively or additionally, the subregion 204 may be identified by designating one or more tiles or slices to be included in and / or excluded from the subregion 204. The number and location of tiles within a frame may be signaled in the picture header. In one embodiment, the signaling may be explicit. Alternatively or additionally, the PPS may signal any or all of the tile rows, columns, row heights, and / or column widths, which may be combined and / or utilized by a decoder to determine the tile count and / or number. For example, without limitation, a PPS parameter denoted as pps_num_exp_tile_columns_minus1, incremented by 1, may explicitly specify the number of tile column widths provided. As a further non-limiting example, the parameter pps_tile_column_width_minus1[i] incremented by 1 may specify the width of the i-th tile column in units of coding tree blocks (CTBs), for i in the range from 0 to pps_num_exp_tile_columns_minus1. The parameter pps_tile_row_height_minus1[i] incremented by 1 may specify the height of the i-th tile row in units of CTBs, for i. Alternatively or additionally, the signaled parameters may specify the number and / or dimensions of slices in one or more tiles. For example, the parameter pps_num_exp_slices_in_tile[i] may specify the number of explicitly provided slice heights for slices in the tile containing the i-th slice. The parameter pps_slice_width_in_tiles_minus1[i] incremented by 1 may specify the width of the i-th rectangular slice in units of tile columns. The parameter denoted pps_slice_height_in_tiles_minus1[i] incremented by 1 may specify the height of the i-th rectangular slice in units of tile rows, for example, when pps_num_exp_slices_in_tile[i] is equal to 0.Those skilled in the art will recognize, upon reviewing this disclosure in its entirety, various alternative or additional ways in which tile and / or slice parameters may be signaled and / or determined, whether implicitly or explicitly, in and / or from the bitstream and / or header parameters.

[0025] Also referring to FIG. 2 , if transforming a subregion 204 includes rescaling the subregion 204, the width and height of the smaller and / or larger subregions may be obtained by multiplying the width and height of the subregion 204 by an arbitrary rescaling constant (Rc), which may also be referred to as a scaling factor and / or constant and may alternatively or additionally be referred to by a variable name such as RefPicScale. For smaller subregions 204, Rc may have a value between 0 and 1. For larger frames, Rc may have a value greater than 1. For example, Rc may have a value between 1 and 4, or other values. The rescaling constant may be different from one resolution dimension to another. For example, a rescaling constant Rch may be used to rescale the height, and another rescaling constant Rcw may be used to rescale the width.

[0026] Also, referring to Figure 2, rescaling may be implemented as a mode. In some implementations, the encoder may signal to the decoder the rescaling constant to use as a function of picture parameters, such as, for example, the pps_pic_width_in_luma_samples parameter, the pps_scaling_win_right_offset parameter, and / or the pps_scaling_win_left_offset parameter. The signaling may be performed in the sequence parameter set (SPS) corresponding to the GOP that includes the current picture and / or the picture parameter set (PPS) corresponding to the current picture. For example, without limitation, an encoder may signal rescaled parameters using fields such as pps_pic_width_in_luma_samples, pps_pic_height_in_luma_samples, pps_scaling_win_left_offset, pps_scaling_win_right_offset, pps_scaling_win_top_offset, pps_scaling_win_bottom_offset, and / or sps_num_subpics_minus1. A parameter such as pps_scaling_window_explicit_signalling_flag equal to 1 may specify that a scaling window offset parameter is present in the PPS. A pps_scaling_window_explicit_signalling_flag equal to 0 may indicate that a scaling window offset parameter is not present in the PPS. When sps_ref_pic_resampling_enabled_flag is equal to 0, the value of pps_scaling_window_explicit_signalling_flag may be equal to 0.pps_scaling_win_left_offset, pps_scaling_win_right_offset, pps_scaling_win_top_offset, and pps_scaling_win_bottom_offset may specify offsets to be applied to the picture size for scaling ratio calculation. When absent, the values ​​of pps_scaling_win_left_offset, pps_scaling_win_right_offset, pps_scaling_win_top_offset, and pps_scaling_win_bottom_offset may be inferred to be equal to pps_conf_win_left_offset, pps_conf_win_right_offset, pps_conf_win_top_offset, and pps_conf_win_bottom_offset, respectively.

[0027] 2 , as described above, the W and H parameters may be represented using, but are not limited to, the variables CurrPicScalWinWidthL and CurrPicScalWinHeightL, respectively. These variables may be derived from the signaled parameters, as described above, using one or more mathematical relationships between the signaled parameters and the variables. For example, but not limited to, CurrPicScalWinWidthL may be derived according to the following equation: CurrPicScalWinWidthL = pps_pic_width_in_luma_samples - SubWidthC * (pps_scaling_win_right_offset + pps_scaling_win_left_offset)

[0028] As a further non-limiting example, CurrPicScalWinHeightL may be derived according to the following formula: CurrPicScalWinWidthL = pps_pic_width_in_luma_samples - SubWidthC * (pps_scaling_win_right_offset + pps_scaling_win_left_offset) The rescaling operation may be performed at the block level of the coded frame and / or sub-region 204. For example, the sub-region 204 to be used as the reference frame 104 may be first rescaled, and then prediction may be performed. The block prediction process may be performed on the scaled reference frame 104 (having the scaled resolution) rather than the original reference frame 104. Rescaling the reference frame 104 and / or sub-region 204 may include rescaling according to any parameters signaled by the encoder, as described above. For example, if the reference frame 104 to be used with the current picture is signaled, such as, but not limited to, via reference to an index value associated with the reference frame 104, the signaled reference frame 104 may be rescaled prior to prediction according to any of the rescaling methods described above. The rescaled reference frame 104 may be stored in memory and / or a buffer, including, but not limited to, a buffer that identifies the frame contained therein by an index upon which a frame search may be performed. The buffer may include a decoded picture buffer (DCB) and / or one or more additional buffers implemented by the decoder. For example, the prediction process may include inter-picture prediction, which includes motion compensation.

[0029] Also, referring to FIG. 2, some implementations of block-based rescaling may allow the flexibility to apply an optimal filter to each block instead of applying the same filter to the entire frame. Some implementations may allow some blocks (e.g., based on pixel uniformity and bitrate cost) to be in skip-rescaling mode (so that rescaling does not change the bitrate). The skip-rescaling mode may be signaled in the bitstream. For example, without limitation, the skip-rescaling mode may be signaled in a PPS parameter. Alternatively or additionally, the decoder may determine that the skip-rescaling mode is active based on one or more parameters set by the decoder and / or signaled in the bitstream.

[0030] Also, referring to FIG. 2 , rescaling may include upsampling or other methods using a spatial filter. Spatial filters used in rescaling may include, but are not limited to, a bicubic spatial filter that applies bicubic interpolation, a bilinear spatial filter that applies bilinear interpretation, a sinc filter, a Lanczos filter that uses Lanczos filtering and / or Lanczos resampling using a combination of sinc function interpolation and / or signal reconstruction techniques, etc. Those skilled in the art will recognize various filters that may be used for interpolation consistent with this disclosure upon reviewing the entirety of this disclosure. As a non-limiting example, the interpolation filter may include any of the filters described above, a low-pass filter, which may be used as an upsampling process, but is not limited to, in which pixels between pixels of a block and / or frame before scaling may be initialized to zero, and the output of the low-pass filter may then be input. Alternatively or additionally, any luma sample interpolation filtering process may be used. Luma sample interpretation may include calculating an interpolated value at a half-sample interpolation filter index located between two consecutive sample values ​​of an unscaled sample array. Calculation of the interpolated values ​​may be performed by, but is not limited to, retrieving coefficients and / or weights from a lookup table. The selection of the lookup table may be performed as a function of the motion model of the coding unit and / or scaling ratio, for example, as determined using a scaling constant, as described above. The calculation may include, but is not limited to, performing a weighted sum of neighboring pixel values, with the weights retrieved from a lookup table. Alternatively or additionally, the calculated values ​​may be shifted. For example, but not limited to, the values ​​may be shifted by Min(4, BitDepth-8), 6, Max(2, 14-BitDepth), etc. Those skilled in the art will recognize various alternative or additional implementations that may be used for the interpolation filters upon reviewing this disclosure in its entirety.

[0031] Referring now to FIG. 3 , the predicted picture 108 may have the same or similar resolution and / or size as the extracted independent reference region 304. This approach may be used to downscale video resolution and thus reduce bitrate, to focus on regions of interest to the viewer, and / or to focus on regions identified by automatic or user-friendly detection as containing visual data that is more relevant to some purpose and / or task. Alternatively or additionally, this approach may allow for continued display of video when network speeds slow down. Advantages provided by this approach may include saving bandwidth used for video transmission, saving resources used for video encoding, and / or saving the time required to decode and play back the video. Devices and / or networks implementing the disclosed embodiments may result in a better user experience as well as more efficient use of resources.

[0032] 3, the predicted picture 108 may then be rescaled to a smaller or larger picture. The width and height of the smaller or larger picture may be obtained by multiplying W and H by an arbitrary rescaling constant (Rc) (also referred to as a scaling factor). As a non-limiting example, for smaller pictures, Rc may have a value between 0 and 1. As a further non-limiting example, for larger frames, Rc may have a value between 1 and 4. Other values ​​are also possible. The rescaling operation may be left as an option for the end user and / or additional programs and / or modules operating on the computing device displaying the video to the end user, and in one example, the image may be rescaled to fit the display resolution.

[0033] 4, the decoder may rescale the independent reference region 304, e.g., using a rescaling constant as described above, to generate a rescaled region 404 to match the full resolution and / or target resolution of the original video picture. For example, W and H may each be multiplied by a selected Rc to scale W and H to the same size as D1 and D2 as described above, such as, but not limited to, Rc = D1 / W. Prediction and other operations may be performed to obtain a predicted picture using the rescaled sub-regions.

[0034] Referring now to FIG. 5 , the independent reference region 204 may be used to predict a portion of a picture rather than the entire picture. For example, a picture may extend beyond a user's field of view, such as, but not limited to, a 360-degree video picture and / or a video picture used in virtual reality. In such a situation, a given frame of a video picture may be rendered with the independent reference region 204 corresponding to a predicted and / or detected user's current field of view. In other embodiments, the independent reference region may correspond to an important, highly detailed, and / or high-motion portion of a first frame. The residual of the predicted frame may be generated using any other suitable prediction and / or decoding method. Pixels may be uncoded, coded with a default color such as, but not limited to, black, and / or given the chroma and / or luma values ​​of neighboring pixels, e.g., extending the chroma and luma values ​​from the edge of the independent reference region to fill the screen. Alternatively or additionally, portions may be predicted from other portions, such as reference frames, residuals, motion vectors, etc.

[0035] 5, the decoder may decode all or a portion of the second frame by transforming the first independent reference region 204, denoted here as "1" for illustrative purposes. Transforming the first independent reference region 204 may include, for example, scaling the first independent reference region 204, as described above. Alternatively or additionally, transforming the first independent reference region 204 may include moving the first independent reference region 204 relative to a position in the video picture, which may include an edge and / or any coordinate in the video picture. As one non-limiting example, as shown for illustrative purposes in FIG. 5 , the first independent reference region 204 may be displaced from its original position in the video picture coordinate system and / or displaced to a new position relative to edges and / or pixels using a linear transformation, such as an affine transformation; as used in this disclosure, an “affine motion transformation” is a transformation, such as a matrix and / or vector, that describes the uniform displacement of a video picture and / or a set of pixels or points represented in a picture, such as a set of pixels representing an object that moves across a view in a video without changing its apparent shape during the movement. Any transformation, including any transformation that can be described using a matrix or other mathematical descriptor, may be used consistently with this disclosure to move or otherwise transform the first independent reference region. For example, without limitation, transforming the first independent reference region may include rotating the first independent reference region relative to its position in the video picture, flipping the first independent reference region, etc.

[0036] Also, referring to FIG. 5 , decoding may include use of a second independent reference region 204, shown here for illustrative purposes as “2.” In one embodiment, the decoder may find the second independent reference region 204 in the first frame, which may be performed in any manner described above with respect to the first independent reference region 204. Alternatively or additionally, the second independent reference region 204 may be extracted from another reference frame and / or retrieved from a buffer, such as a reference buffer and / or an LTR buffer, as described in more detail below. Decoding from the second independent reference region 204 may be performed using any method and / or method steps, as described above with respect to the first independent reference region. The combination of the first independent reference region 204 and the second independent reference region 204 may be used in various ways; for example, the first independent reference region 204 may depict a first view for a user of a picture having a size that exceeds the user's field of view, and the second independent reference region 204 may depict another view, which may be contiguous. Further independent reference regions 204 may also be used to provide further portions of the decoded frame. Multiple independent reference regions may be extracted and / or retrieved to decode a picture, may be contiguous, may be connected by pixels predicted using any of the methods described above, or may be otherwise combined. Alternatively or additionally, multiple independent reference regions 204 may be used sequentially for a sequence of frames.

[0037] 6, one or more independent reference regions 204 may be stored in a buffer, such as a reference buffer and / or LTR buffer 604. The LTR buffer 604 may include multiple frames. In one embodiment, the LTR buffer 604 may include multiple frames and / or independent reference regions 204. Each of the multiple frames and / or independent reference regions may have a corresponding index and / or signaling for searching that allows searching, for example, as described in further detail below. The reference buffer and / or LTR buffer 604 may be periodically updated and / or modified, for example, by adding and / or removing frames and / or independent reference regions.

[0038] Also, referring to FIG. 6 , the use of independent reference regions 204 and / or reference frames 104 may be signaled in a bitstream, for example, by an encoder. For example, but not limited to, the use of independent reference regions, the presence of independent reference regions in a picture, may be signaled by an encoder in a header of a video sequence, for example, in a sequence parameter set. A single flag may be used to indicate the presence of independent regions. The absence of a flag may be interpreted as the absence of any independent regions. The total number of independent regions may also be signaled in the sequence header. For example, as described above, geometric characteristics of the independent reference regions, identifiers of the independent reference regions, for retrieval from a buffer, may also be signaled in the sequence header. Alternatively or additionally, one or more signals may be provided in a picture header, for example, in a picture parameter set. In one embodiment, signaling in a picture header may enhance decoder flexibility and enable picture-level decisions. The list of region IDs may include a sequence of consecutive numbers representing the region IDs in a predetermined order. The decoder may use the signaled list to rearrange and reconstruct the independent regions and the picture regions predicted from the independent regions.

[0039] Referring now to FIG. 7, an exemplary embodiment of a method 700 of video encoding using reference regions is shown. At step 705, a decoder receives a bitstream, e.g., as described in more detail below. The bitstream may include an encoded video bitstream. The bitstream may include at least one encoded reference picture and / or LTR frame, which may alternatively be referred to as a "reference picture" and / or an "LTR picture," and at least one encoded current picture. The encoded current picture may have a first size, which may include any size as described above, including an area. At step 710, the decoder decodes the reference picture and / or LTR frame. This may be performed according to any process for decoding as described in this disclosure. The decoder may identify the reference frame and / or LTR frame in the bitstream. Alternatively, the reference frame and / or LTR frame may not be decoded, and only the independent reference region may be decoded.

[0040] Also referring to FIG. 7 , in step 715, the decoder locates a first sub-region within the reference frame and / or LTR frame. This may be accomplished, without limitation, as described above with reference to FIGS. 1-6 . For example, without limitation, locating the first sub-region may include identifying, in a bitstream, a geometric characterization of an independent reference region within the reference frame and / or LTR frame. The bitstream may be signaled by the encoder as described above. As a non-limiting example, the first sub-region may be rectangular, and the geometric characterization may include a 4-tuple of values ​​characterizing the vertices of the first sub-region. As a further non-limiting example, the geometric characterization may include a height offset, a height, a length offset, and a length, and / or the sub-region 204 may be characterized by a height offset, a height, a length offset, and a length. The first sub-region has a second size. The second size may be different from the first size, or in other words, may be either larger or smaller than the first size. Identifying the first sub-region may include receiving an indication in the bitstream that the first sub-region is present. In one embodiment, traditional prediction using reference frames may still be supported by either signaling that there is a zero region in the picture or by defining one region having the same size as the original picture. Flexibility may be provided by allowing the specification of one or more regions that are extracted and, as such, considered as independent reference pictures for future prediction.

[0041] Continuing to refer to FIG. 7, at step 720, the decoder transforms the first sub-region 204. The transformation may generate a second and / or rescaled reference picture and / or portion thereof. Transforming the first sub-region may include any transformation and / or modification to any sub-region, as described in this disclosure. Transforming the first sub-region may include, but is not limited to, moving the first sub-region. As a further example, the decoder may be configured to transform the first sub-region by applying an affine transform, which may include any affine transform, as described above. As a further non-limiting example, the decoder may rescale the first sub-region to a third size to form a rescaled reference picture. The third size may be equal to the first size. In other words, the decoder may rescale the sub-region to match the current and / or signaled size of the current frame. Alternatively or additionally, the first sub-region may remain at its current size, and the decoder may not transform the first sub-region. The decoder may extract the first sub-region from the reference frame and / or the LTR frame. This may be implemented, without limitation, as described above with reference to FIGS. 1-6. In step 725, the decoder decodes the current frame using the first sub-region as a reference for the current frame. This may be implemented, without limitation, as described above with reference to FIGS. 1-6. For example, decoding the current frame may include decoding a current frame having the same size as the first sub-region. Decoding the second frame may include transforming the first sub-region. Transforming the first sub-region may include scaling the first sub-region, flipping the first sub-region, moving the first sub-region relative to its position in the video picture, and / or rotating the first sub-region relative to its position in the video picture.

[0042] Also, referring to FIG. 7 , the decoder may store the reference frame and / or the LTR frame in a buffer. The buffer may include a long-term reference buffer and / or a reference picture buffer. The decoder may be further configured to find a second sub-region in the reference frame and / or the LTR frame. The decoder may decode a second current frame using the first sub-region and / or the second sub-region. The decoder may store a second independent reference region in the buffer. The decoder may decode the second current frame using the first sub-region and / or the second sub-region and / or the reference frame, where the first sub-region and / or the second sub-region and / or the reference frame may be retrieved from the buffer, extracted from another frame, etc.

[0043] 8 is a system block diagram illustrating an example decoder 700 capable of decoding a bitstream containing global motion vector candidates utilized by neighboring blocks by constructing a motion vector candidate list. The decoder 700 may include an entropy decoder processor 704, an inverse quantization and inverse transform processor 708, a deblocking filter 712, a frame buffer 716, a motion compensation processor 720, and / or an intra-prediction processor 724.

[0044] Also referring to FIG. 8 , in operation, a bitstream 728 may be received by the decoder 700 and input to the entropy decoder processor 704, which may entropy decode portions of the bitstream into quantized coefficients. The quantized coefficients may be provided to the inverse quantization and inverse transform processor 708, which may perform inverse quantization and inverse transform to generate a residual signal, which may be added to the output of the motion compensation processor 720 or the intra-prediction processor 724, depending on the processing mode. The output of the motion compensation processor 720 and the intra-prediction processor 724 may include a block prediction value based on a previously decoded block. The sum of the prediction value and the residual may be processed by the deblocking filter 712 and stored in the frame buffer 716.

[0045] Also, referring to FIG. 8 , in one embodiment, decoder 700 may include circuitry configured to implement any of the operations as described above in any embodiment, in any order and with any degree of repetition. For example, decoder 700 may be configured to repeatedly perform a single step or sequence until a desired or commanded result is achieved. The repetition of a step or sequence of steps may be performed iteratively and / or recursively, using the output of a previous iteration as input to a subsequent iteration, aggregating the inputs and / or outputs of an iteration to generate an aggregate result, reducing or decrementing one or more variables, such as a global variable, and / or dividing a larger processing task into a set of smaller processing tasks that are addressed iteratively. The decoder may perform any step or sequence of steps in parallel as described in this disclosure, such as performing a step two or more times simultaneously and / or substantially simultaneously using two or more parallel threads, processor cores, etc. The division of tasks among parallel threads and / or processes may be performed according to any protocol suitable for dividing tasks among iterations. Those skilled in the art will recognize, upon reviewing this disclosure in its entirety, the various ways in which steps, sequences of steps, processing tasks, and / or data can be subdivided, shared, or otherwise handled using iterative, recursive, and / or parallel processing.

[0046] Continuing with reference to FIG. 8 , decoder 700 and / or its circuitry may be designed and / or configured to perform any method, method step, or sequence of method steps in any embodiment described in this disclosure in any order, and with any degree of repetition. For example, decoder 700 and / or its circuitry may be configured to repeatedly perform a single step or sequence until a desired or commanded result is achieved. The repetition of a step or sequence of steps may be performed iteratively and / or recursively, using the output of a previous iteration as input to a subsequent iteration, aggregating the inputs and / or outputs of an iteration to produce an aggregate result, reducing or decrementing one or more variables, such as a global variable, and / or dividing a larger processing task into a set of smaller processing tasks that are addressed iteratively. Decoder 700 and / or its circuitry may perform any step or sequence of steps in parallel as described in this disclosure, such as performing a step two or more times simultaneously and / or substantially simultaneously using two or more parallel threads, processor cores, etc. The division of tasks among parallel threads and / or processes may be performed according to any protocol suitable for dividing tasks among iterations. Those skilled in the art will recognize, upon reviewing this disclosure in its entirety, various ways in which steps, sequences of steps, processing tasks, and / or data may be subdivided, shared, or otherwise handled using iterations, recursion, and / or parallel processing.

[0047] 9 is a process flow diagram illustrating an example process 800 for encoding video with adaptive cropping, which may allow more flexibility for video encoders / decoders, enabling bitrate savings in various use cases. At step 805, the video frame may undergo initial block division, for example, using a tree-structured macroblock partitioning scheme, which may include dividing the picture frame into CTUs and CUs.

[0048] Also, referring to FIG. 9 , at step 810, identification of a first reference region may be performed, including selection of a subregion of a frame or portion thereof. The region may be selected by means of automatic input or expert input. As a non-limiting example, automatic selection may be achieved by a computer vision algorithm that detects a particular object. Object detection may include further processing, such as object classification. Expert input selection may be achieved using manual human intervention, such as, but not limited to, selecting a close-up of a person and / or object of interest in a video, such as a person in a surveillance video. Another possible use case may be selecting a maximum attention region that contributes most to bitrate reduction. Adaptive cropping may further include selection of geometric characterizations of the subregion. For example, but not limited to, selection of geometric characterizations of the subregion may include selection of a 4-tuple as described above, such as, but not limited to, (X, Y, W, H). Selection of geometric characterizations of the subregion may include update information and / or information indicating changes to the data defining the subregion from one frame to another, as described above with respect to dynamic data defining the subregion.

[0049] 9, at step 815, the block may be coded and included in a bitstream. For example, coding may include utilizing inter-prediction and intra-prediction modes. For example, coding may include adding bits to characterizations (X, Y, W, H) in the bitstream, specifying adaptive cropping modes, etc., as described above. Coding may also include coding update information and / or information indicating changes to data defining the subregion from one frame to another, as described above with respect to dynamic data defining the subregion.

[0050] 10 is a system block diagram illustrating an example video encoder 1000 capable of adaptive cropping, which may enable additional flexibility for video encoders / decoders, enabling bitrate savings in various use cases. The example video encoder 1000 receives an input video 1005, which may be initially segmented or divided according to a processing scheme, such as a tree-structured macroblock partition scheme (e.g., a quadtree plus binary tree). One example of a tree-structured macroblock partition scheme may include dividing a picture frame into large block elements called coding tree units (CTUs). In some implementations, each CTU may be further divided one or more times into multiple sub-blocks called coding units (CUs). The end result of this division may include groups of sub-blocks, which may be called prediction units (PUs). Transform units (TUs) may also be utilized.

[0051] 10, the exemplary video encoder 1000 includes an intra-prediction processor 1015, a motion estimation / compensation processor 1020 (also called an inter-prediction processor) that can support adaptive cropping, a transform / quantization processor 1025, an inverse quantization / inverse transform processor 1030, an in-loop filter 1035, a decoded picture buffer 1040, and an entropy coding processor 1045. Bitstream parameters may be input to the entropy coding processor 1045 and included in the output bitstream 1050.

[0052] Continuing to refer to Figure 10, in operation, for each block of a frame of input video 1005, it may be determined whether to process the block via intra-picture prediction or to process the block using motion estimation / compensation. The block may be provided to an intra-prediction processor 1010 or a motion estimation / compensation processor 1020. If the block is to be processed via intra-prediction, the intra-prediction processor 1010 may perform processing and output a predictor. If the block is to be processed via motion estimation / compensation, the motion estimation / compensation processor 1020 may perform processing including using adaptive cropping, if applicable.

[0053] Also referring to FIG. 10 , a residual may be formed by subtracting a predictor from the input video. The residual may be received by a transform / quantization processor 1025, which may perform a transform operation (e.g., a discrete cosine transform (DCT)) to generate coefficients that may be quantized. The quantized coefficients and any associated signaling information may be provided to an entropy coding processor 1045 for entropy coding and included in the output bitstream 1050. The entropy coding processor 1045 may support encoding of signaling information related to encoding the current block. Furthermore, the quantized coefficients may be provided to an inverse quantization / inverse transform processor 1030, which may regenerate pixels that may be combined with the predictor and processed by an in-loop filter 1035, the output of which is stored in a decoded picture buffer 1040 for use by a motion estimation / compensation processor 1020, which is capable of adaptive cropping.

[0054] 10, although several variations have been described in detail above, other modifications or additions are possible. For example, in some implementations, the current block may include any asymmetric block (8x4, 16x8, etc.), as well as any symmetric block (8x8, 16x16, 32x32, 64x64, 128x128, etc.).

[0055] Also, referring to FIG. 10, in some implementations, a quad-tree plus binary decision tree (QTBT) may be implemented. In QTBT, at the coding tree unit level, the partition parameters of the QTBT are dynamically derived to adapt to local characteristics without transmitting any overhead. Then, at the coding unit level, a joint classifier decision tree structure may eliminate unnecessary repetitions and control the risk of erroneous predictions. In some implementations, an LTR frame block update mode may be available as an additional selection available at each leaf node of the QTBT.

[0056] Continuing with reference to FIG. 10, in some implementations, additional syntax elements may be signaled at different hierarchical levels of the bitstream. For example, a flag may be valid for the entire sequence by including an enable flag coded in a sequence parameter set (SPS). Furthermore, a CTU flag may be coded at the coding tree unit (CTU) level.

[0057] Also, with reference to FIG. 10 , encoder 1000 may include circuitry configured to implement any of the operations described above with reference to FIG. 8 or FIG. 10 in any embodiment, in any order, and with any degree of repetition. For example, encoder 1000 may be configured to repeatedly perform a single step or sequence until a desired or commanded result is achieved. The repetition of a step or sequence of steps may be performed iteratively and / or recursively, using the output of a previous iteration as input to a subsequent iteration, aggregating the inputs and / or outputs of an iteration to generate an aggregate result, reducing or decrementing one or more variables, such as a global variable, and / or dividing a larger processing task into a set of smaller processing tasks that are addressed iteratively. Encoder 1000 may perform any step or sequence of steps in parallel as described in this disclosure, such as performing a step two or more times, simultaneously and / or substantially simultaneously, using two or more parallel threads, processor cores, etc. The division of tasks among parallel threads and / or processes may be performed according to any protocol suitable for dividing tasks among iterations. Those skilled in the art will recognize, upon reviewing this disclosure in its entirety, the various ways in which steps, sequences of steps, processing tasks, and / or data can be subdivided, shared, or otherwise handled using iterative, recursive, and / or parallel processing.

[0058] Continuing with reference to FIG. 10 , a non-transitory computer program product (i.e., a physically embodied computer program product) may store instructions that, when executed by one or more data processors of one or more computing systems, cause at least one data processor to perform the operations and / or steps thereof described herein, including, but not limited to, may be configured to perform any of the operations described above and / or any of the operations of decoder 700 and / or encoder 1000. Similarly, a computer system is also described that may include one or more data processors and a memory coupled to the one or more data processors. The memory may temporarily or permanently store instructions that cause at least one processor to perform one or more operations described herein. Furthermore, the methods may be implemented either by one or more data processors in a single computing system or by one or more data processors distributed among two or more computing systems. Such computing systems may be connected to exchange data and / or commands or other instructions, etc., via one or more connections, including, for example, via a direct connection between one or more of the computing systems, or via a network (e.g., the Internet, a wireless wide area network, a local area network, a wide area network, a wired network, etc.).

[0059] Continuing with reference to FIG. 10 , encoder 1000 and / or its circuitry may be designed and / or configured to perform any method, method step, or sequence of method steps in any embodiment described in this disclosure in any order and with any degree of repetition. For example, encoder 1000 and / or its circuitry may be configured to repeatedly perform a single step or sequence until a desired or commanded result is achieved. The repetition of a step or sequence of steps may be performed iteratively and / or recursively, using the output of a previous iteration as input to a subsequent iteration, aggregating the inputs and / or outputs of an iteration to generate an aggregate result, reducing or decrementing one or more variables, such as a global variable, and / or dividing a larger processing task into a set of smaller processing tasks that are addressed iteratively. Encoder 1000 and / or its circuitry may perform any step or sequence of steps in parallel as described in this disclosure, such as performing a step two or more times simultaneously and / or substantially simultaneously using two or more parallel threads, processor cores, etc. The division of tasks among parallel threads and / or processes may be performed according to any protocol suitable for dividing tasks among iterations. Those skilled in the art will recognize, upon reviewing this disclosure in its entirety, various ways in which steps, sequences of steps, processing tasks, and / or data may be subdivided, shared, or otherwise handled using iterations, recursion, and / or parallel processing.

[0060] In some embodiments, a decoder includes circuitry configured to: receive an encoded video bitstream, the encoded video stream including an encoded reference picture and an encoded current picture having a first size; decode the reference picture; identify a sub-region of the reference picture from the bitstream, the sub-region having a second size, the second size different from the first size; rescale the sub-region to a third size to form a rescaled reference picture, the third size equal to the first size; and decode the current picture using the rescaled reference picture.

[0061] A sub-region may be characterized by a top offset, a bottom offset, a right offset, and a left offset. Identifying a sub-region may include receiving an indication in the bitstream that the sub-region exists.

[0062] In some embodiments, the decoder includes circuitry configured to receive an encoded video bitstream including an encoded first reference picture and an encoded current picture, decode the reference picture, identify a first sub-region of the reference picture from the bitstream, transform the first sub-region to form a second reference picture, and decode the current picture using the second reference picture.

[0063] The current picture may have a first size, the first sub-region may have a second size different from the first size, and the decoder may be configured to transform the sub-region by scaling the first sub-region to a third size equal to the first size. The decoder may be configured to transform the first sub-region by moving the first sub-region. The decoder may be configured to transform the first sub-region by applying an affine transformation. The decoder may be further configured to store the first reference picture in a buffer. The buffer may include a long-term reference buffer. The buffer may include a reference picture buffer. The decoder may be further configured to find the second sub-region in the first reference picture. The decoder may be further configured to decode the second frame using the first reference region and the second independent reference region. The decoder may be further configured to store the second independent reference region in a buffer. The current picture may be a first picture, and the decoder may be further configured to decode a second picture using the first sub-region and the second sub-region.

[0064] In some embodiments, a method of video coding using reference regions includes receiving an encoded video bitstream, the encoded video stream including an encoded reference picture and an encoded current picture having a first size; decoding, by a decoder, the reference picture; identifying, by the decoder and from the bitstream, a sub-region of the reference picture, the sub-region having a second size, the second size different from the first size; rescaling, by the decoder, the sub-region to a third size to form a rescaled reference picture, the third size equal to the first size; and decoding, by the decoder, the current picture using the rescaled reference picture.

[0065] The sub-region may be characterized by a height offset, a height, a length offset, and a length. Identifying the sub-region may include receiving an indication in the bitstream that the sub-region exists. The method may include storing the reference frame in a buffer. The buffer may include a long-term reference buffer. The buffer may include a reference picture buffer.

[0066] In some embodiments, the decoder includes circuitry configured to receive a bitstream, identify a first frame, find a first independent reference region in the first frame, extract the first independent reference region from the first frame, and decode the second frame using the first independent reference region as a reference for the second frame.

[0067] The decoder may be further configured to locate the first independent reference region by identifying, in the bitstream, a geometric characterization of the independent reference region in the first frame. The first independent reference region may be rectangular, and the geometric characterization may include a 4-tuple of numerical values ​​characterizing vertices of the first independent reference region. Identifying the first independent reference region may include receiving, in the bitstream, an indication that the first independent reference region is present. The first independent reference region may have a size, and the decoder may be configured to decode the second frame by decoding the second frame having the same size as the first independent reference region. The decoder may be configured to decode the second frame by transforming the first independent reference region. Transforming the first independent reference region may include scaling the first independent reference region. Transforming the first independent reference region may include flipping the first independent reference region. Transforming the first independent reference region may include moving the first independent reference region relative to a position in the video picture. Transforming the first independent reference region may include rotating the first independent reference region relative to a position in the video picture.

[0068] The decoder may be further configured to store the first frame in a buffer. The buffer may include a long-term reference buffer. The buffer may include a reference picture buffer. The decoder may be further configured to find a second reference region in the first frame. The decoder may be further configured to decode the second frame using the first reference region and the second independent reference region. The decoder may be further configured to store the second independent reference region in a buffer. The decoder may be further configured to decode the second frame using the first reference region and the second reference region.

[0069] In some embodiments, a method for video encoding using reference regions includes receiving a bitstream, identifying a first frame, locating a first independent reference region in the first frame, extracting the first independent reference region from the first frame, and decoding the second frame using the first independent reference region as a reference for the second frame.

[0070] Locating the first independent reference region may include identifying, in the bitstream, a geometric characterization of the independent reference region in the first frame. The first independent reference region may be rectangular, and the geometric characterization may include a 4-tuple of numerical values ​​characterizing vertices of the first independent reference region. Identifying the first independent reference region may include receiving, in the bitstream, an indication that the first independent reference region exists. The first independent reference region may have a size, and decoding the second frame may include decoding the second frame having the same size as the first independent reference region. The method may include decoding the second frame by transforming the first independent reference region. Transforming the first independent reference region may include scaling the first independent reference region. Transforming the first independent reference region may include flipping the first independent reference region. Transforming the first independent reference region may include moving the first independent reference region relative to a position in the video picture. Transforming the first independent reference region may include rotating the first independent reference region relative to a position in the video picture.

[0071] The method may include storing the first frame in a buffer. The buffer may include a long-term reference buffer. The buffer may include a reference picture buffer. The decoder may be further configured to locate a second reference region in the first frame. The method may further include decoding the second frame using the first reference region and the second independent reference region. The method may include storing the second independent reference region in a buffer. The method may include decoding the second frame using the first reference region and the second reference region.

[0072] It should be noted that any one or more aspects and embodiments described herein may be conveniently implemented using one or more machines (e.g., one or more computing devices utilized as user computing devices for electronic documents, one or more server devices, such as document servers, etc.) programmed in accordance with the teachings herein, as would be apparent to those skilled in the computer arts. Appropriate software coding may be readily produced by skilled programmers based on the teachings of the present disclosure, as would be apparent to those skilled in the software arts. Aspects and implementations described above employing software and / or software modules may also include appropriate hardware to support the implementation of the machine-executable instructions of the software and / or software modules.

[0073] Such software may be a computer program product employing a machine-readable storage medium. A machine-readable storage medium may be any medium capable of storing and / or encoding sequences of instructions for execution by a machine (e.g., a computing device) and causing the machine to perform any one of the methodologies and / or embodiments described herein. Examples of machine-readable storage media include, but are not limited to, magnetic disks, optical disks (e.g., CDs, CD-Rs, DVDs, DVD-Rs, etc.), magneto-optical disks, read-only memory "ROM" devices, random-access memory "RAM" devices, magnetic cards, optical cards, solid-state memory devices, EPROMs, EEPROMs, and any combination thereof. As used herein, machine-readable media is intended to include a single medium as well as a collection of physically separate media, such as, for example, a collection of compact discs or one or more hard disk drives in combination with computer memory. As used herein, machine-readable storage media does not include a transitory form of signal transmission.

[0074] Such software may also include information (e.g., data) carried in a data signal on a data carrier, such as a carrier wave. For example, machine-executable information may be included in a data carrier signal embodied in a data carrier, the signal encoding sequences of instructions, or portions thereof, for execution by a machine (e.g., a computing device), and any associated information (e.g., data structures and data) that cause the machine to perform any one of the methodologies and / or embodiments described herein.

[0075] Examples of computing devices include, but are not limited to, e-book reading devices, computer workstations, terminal computers, server computers, mobile devices (e.g., tablet computers, smartphones, etc.), web appliances, network routers, network switches, network bridges, any machine capable of executing a sequence of instructions that specify actions to be taken by that machine, and any combination thereof. In one embodiment, a computing device may include and / or be included in a kiosk.

[0076] 11 shows a diagrammatic representation of one embodiment of a computing device in the exemplary form of a computer system 1100 upon which a set of instructions for causing a control system to perform any one or more aspects and / or methodologies of the present disclosure may be executed. It is also contemplated that multiple computing devices may be utilized to execute a set of instructions specifically configured to cause one or more devices to perform any one or more aspects and / or methodologies of the present disclosure. Computer system 1100 includes a processor 1104 and memory 1108, which communicate with each other and with other components via a bus 1112. Bus 1112 may include any of several types of bus structures, including, but not limited to, a memory bus, a memory controller, a peripheral bus, a local bus, and any combination thereof, using any of a variety of bus architectures.

[0077] Processor 1104 may include any suitable processor, such as, but not limited to, a processor incorporating logic circuitry for performing arithmetic and logical operations, such as an arithmetic and logic unit (ALU), which may be regulated by a state machine and directed by operational inputs from memory and / or sensors. As one non-limiting example, processor 1104 may be organized according to a von Neumann and / or Harvard architecture. Processor 1104 may include, incorporate, and / or be incorporated into, but not limited to, a microcontroller, a microprocessor, a digital signal processor (DSP), a field programmable gate array (FPGA), a complex programmable logic device (CPLD), a graphics processing unit (GPU), a general-purpose GPU, a tensor processing unit (TPU), an analog or mixed-signal processor, a trusted platform module (TPM), a floating-point unit (FPU), and / or a system-on-a-chip (SoC).

[0078] Memory 1108 may include a variety of components (e.g., machine-readable media), including, but not limited to, random-access memory components, read-only components, and any combination thereof. In one example, a basic input / output system 1116 (BIOS), containing the basic routines that help to transfer information between elements within computer system 1100, such as during start-up, may be stored in memory 1108. Memory 1108 may also include (e.g., stored on one or more machine-readable media) instructions (e.g., software) 1120 that embody any one or more aspects and / or methodologies of the present disclosure. In another example, memory 1108 may further include any number of program modules, including, but not limited to, an operating system, one or more application programs, other program modules, program data, and any combination thereof.

[0079] Computer system 1100 may also include a storage device(s) 1124. Examples of a storage device (e.g., storage device(s) 1124) include, but are not limited to, a hard disk drive, a magnetic disk drive, an optical disk drive combined with optical media, a solid-state memory device, and any combination thereof. Storage device 1124 may be connected to bus 1112 by an appropriate interface (not shown). Exemplary interfaces include, but are not limited to, SCSI, Advanced Technology Attachment (ATA), Serial ATA, Universal Serial Bus (USB), IEEE 1394 (FIREWIRE), and any combination thereof. In one embodiment, storage device 1124 (or one or more components thereof) may be removably interfaced with computer system 1100 (e.g., via an external port connector (not shown)). In particular, storage device 1124 and associated machine-readable media 1128 may provide non-volatile and / or volatile storage of machine-readable instructions, data structures, program modules, and / or other data for computer system 1100. In one embodiment, software 1120 may reside, completely or partially, within machine-readable medium 1128. In another embodiment, software 1120 may reside, completely or partially, within processor 1104.

[0080] Computer system 1100 may also include input devices 1132. In one embodiment, a user of computer system 1100 may input commands and / or other information into computer system 1100 via input devices 1132. Examples of input devices 1132 include, but are not limited to, an alphanumeric input device (e.g., a keyboard), a pointing device, a joystick, a gamepad, an audio input device (e.g., a microphone, a voice response system, etc.), a cursor control device (e.g., a mouse), a touchpad, an optical scanner, a video capture device (e.g., a still camera, a video camera), a touch screen, and any combination thereof. Input devices 1132 may be interfaced to bus 1112 via any of a variety of interfaces (not shown), including, but not limited to, a serial interface, a parallel interface, a gameport, a USB interface, a FIREWIRE interface, a direct interface to bus 1112, and any combination thereof. Input devices 1132 may include a touchscreen interface, which may be part of or separate from display 1136, as discussed further below. The input device 1132 may be utilized as a user selection device for selecting one or more graphical representations in the graphical interface, as described above.

[0081] A user may also input commands and / or other information into computer system 1100 via storage device 1124 (e.g., a removable disk drive, flash drive, etc.) and / or network interface device 1140. A network interface device, such as network interface device 1140, may be utilized to connect computer system 1100 to one or more various networks, such as network 1144, and one or more remote devices 1148 connected thereto. Examples of network interface devices include, but are not limited to, a network interface card (e.g., a mobile network interface card, a LAN card), a modem, and any combination thereof. Examples of networks include, but are not limited to, a wide area network (e.g., the Internet, an enterprise network), a local area network (e.g., a network associated with an office, building, campus, or other relatively small geographic space), a telephone network, a data network associated with a telephone / voice provider (e.g., a mobile communications provider's data and / or voice network), a direct connection between two computing devices, and any combination thereof. A network, such as network 1144, may employ wired and / or wireless modes of communication. In general, any network topology may be used. Information (e.g., data, software 1120, etc.) may be communicated to and / or from computer system 1100 via network interface device(s) 1140.

[0082] Computer system 1100 may further include a video display adapter 1152 for communicating images displayable on a display device, such as display device 1136. Examples of display devices include, but are not limited to, a liquid crystal display (LCD), a cathode ray tube (CRT), a plasma display, a light emitting diode (LED) display, and any combination thereof. Display adapter 1152 and display device 1136 may be utilized in combination with processor 1104 to provide graphical representations of aspects of the present disclosure. In addition to a display device, computer system 1100 may include one or more other peripheral output devices, including, but not limited to, audio speakers, a printer, and any combination thereof. Such peripheral output devices may be connected to bus 1112 via peripheral interface 1156. Examples of peripheral interfaces include, but are not limited to, a serial port, a USB connection, a FIREWIRE connection, a parallel connection, and any combination thereof.

[0083] The foregoing is a detailed description of exemplary embodiments of the present invention. Various modifications and additions may be made without departing from the spirit and scope of the present invention. Features of each of the various embodiments described above may be combined with features of other described embodiments as needed to provide various combinations of features in related new embodiments. Moreover, the foregoing describes multiple separate embodiments, and what has been described herein is merely illustrative of the application of the principles of the present invention. Furthermore, certain methods herein may be described and / or illustrated as being performed in a particular order, which order is highly variable within the ordinary skill in the art for achieving methods, systems, and software according to the present disclosure. Accordingly, the present description is to be taken by way of example only and is not intended to otherwise limit the scope of the present invention.

[0084] Exemplary embodiments are disclosed above and illustrated in the accompanying drawings. It will be understood by those skilled in the art that various modifications, omissions, and additions may be made to what is specifically disclosed herein without departing from the spirit and scope of the present invention.

Claims

1. receiving a bitstream including a sequence parameter set and a first coded picture; Detecting that a coded sub-picture is present in the first coded picture and at a position of the coded sub-picture in the first coded picture in the sequence parameter set associated with the first coded picture; extracting the coded sub-picture from the first coded picture; decoding the coded sub-pictures extracted to form reference pictures; determining a predictor from the reference picture using a scaling constant, the scaling constant being determined from information in the bitstream; using the predictor to decode a subsequent picture in the bitstream; [0023] including circuitry configured to: decoder.

2. A method of image processing by an information processing device, comprising: receiving a bitstream including a sequence parameter set and a first coded picture; detecting, in the sequence parameter set associated with the first coded picture, that a coded sub-picture region is present in the first coded picture and a position of the coded sub-picture region in the first coded picture; extracting the coded sub-picture from the first coded picture; decoding the extracted coded sub-pictures to form decoded independent pictures to be used as reference pictures; determining a predictor from the reference picture using a scaling constant, the scaling constant being determined from information in the bitstream; using the predictor to decode a subsequent picture in the bitstream; A method comprising:

3. The decoder of claim 1 , wherein the information in the bitstream used to obtain the scaling constant includes an index.

4. The decoder of claim 1 , wherein the predictor is formed using an interpolation filter.

5. The decoder of claim 1 , wherein the subsequent picture is a subsequent independent sub-picture.

6. receiving a bitstream including a sequence parameter set and a first coded picture; Detecting a coded sub-picture in the sequence parameter set associated with the first coded picture and a position of the first coded sub-picture in the first coded picture; extracting the coded sub-picture from the first coded picture; decoding the extracted coded sub-pictures to form reference pictures; determining a predictor in the reference picture using a scaling constant obtained from information in the bitstream, the information including an index; using the predictor to decode a subsequent picture; [0023] including circuitry configured to: decoder.

Images