Method and system for adaptive cropping - Patents.com

Abstract

The decoder includes circuitry configured to receive a bitstream, identify a current frame and a cropped sub-frame of the current frame as a function of the bitstream, determine a scaling constant associated with the cropped sub-frame as a function of the bitstream, and reconstruct pixel data for the cropped sub-frame using the scaling constant.

Description

Related Applications

[0001] This application claims the benefit of priority to U.S. non-provisional patent application Ser. No. 17,091,052, entitled "METHODS AND SYSTEMS FOR ADAPTIVE CROPPING," filed on November 6, 2020, which is incorporated by reference in its entirety, and to U.S. provisional patent application Ser. No. 62 / 932,597, entitled "METHODS AND SYSTEMS FOR ADAPTIVE CROPPING," filed on November 8, 2019, which is incorporated by reference in its entirety. [Technical field]

[0002] The present invention relates generally to the field of video compression. In particular, the present invention is directed to a method and system for adaptive cropping. [Background technology]

[0003] A video codec may include electronic circuitry or software that compresses or decompresses digital video. It may convert uncompressed video to a compressed format or vice versa. In the context of video compression, a device that compresses video (and / or performs some of the functions) may typically be called an encoder, and a device that decompresses video (and / or performs some of the functions) may be called a decoder.

[0004] The format of the compressed data may conform to standard video compression specifications. The compression may be lossy, in that the compressed video lacks some information present in the original video. Consequences of this may include that the decompressed video may have lower quality than the original uncompressed video, since there is insufficient information to accurately reconstruct the original video.

[0005] There can be a complex relationship between video quality, the amount of data used to represent the video (e.g., as determined by bit rate), the complexity of the encoding and decoding algorithms, sensitivity to data loss and errors, ease of editing, random access, end-to-end delay (e.g., latency), etc.

[0006] Motion compensation may include an approach to predicting a video frame or a portion thereof given a reference frame, such as a previous and / or future frame, by considering the motion of the camera and / or objects in the video. It may be employed in encoding and decoding video data for video compression, such as encoding and decoding using the Motion Picture Experts Group (MPEG) advanced video coding (AVC) standard (also called H.264). Motion compensation may describe a picture in terms of a transformation from a reference picture to a current picture. The reference picture may be earlier in time compared to the current picture, or may be from the future compared to the current picture. Compression efficiency may be improved when an image can be accurately synthesized from previously transmitted and / or stored images. Summary of the Invention

[0007] In one aspect, the decoder includes circuitry configured to receive a bitstream, identify a current frame and a cropped sub-frame of the current frame as a function of the bitstream, determine a scaling constant associated with the cropped sub-frame as a function of the bitstream, and reconstruct pixel data for the cropped sub-frame using the scaling constant.

[0008] In another aspect, a method includes receiving, by a decoder, a bitstream; identifying, by the decoder and as a function of the bitstream, a current frame and a cropped sub-frame of the current frame; determining, by the decoder and as a function of the bitstream, a scaling constant associated with the cropped sub-frame; and reconstructing, by the decoder, pixel data of the cropped sub-frame using the scaling constant.

[0009] The details of one or more variations of the subject matter described herein are set forth in the accompanying drawings and the description below. Other features and advantages of the subject matter described herein will become apparent from the description and drawings, and from the claims. [Brief description of the drawings]

[0010] For the purpose of illustrating the invention, the drawings show aspects of one or more embodiments of the invention, it being understood, however, that the invention is not limited to the precise arrangements and instrumentalities shown in the drawings.

[0011] [Figure 1] FIG. 1 is a block diagram illustrating an example embodiment of an existing cropping method. [Diagram 2] FIG. 2 is a schematic diagram of an exemplary embodiment of reference frames and sub-frames. [Diagram 3] FIG. 3 is a schematic diagram of an exemplary embodiment of reference frames and sub-frames. [Figure 4] FIG. 4 is a schematic diagram of an exemplary reference frame and an exemplary predicted frame. [Diagram 5] FIG. 5 is a schematic diagram of an exemplary reference frame and an exemplary predicted frame. [Figure 6] FIG. 6 is a process flow diagram illustrating an example process according to some implementations of the present subject matter. [Figure 7]FIG. 7 is a system block diagram illustrating an example decoder capable of decoding bitstreams according to some implementations of the present subject matter. [Figure 8] FIG. 8 is a process flow diagram illustrating an example process for encoding video in accordance with some implementations of the present subject matter. [Figure 9] FIG. 9 is a system block diagram illustrating an example video encoder in accordance with some implementations of the present subject matter. [Figure 10] FIG. 10 is a block diagram of a computing system that may be used to implement any one or more of the methodologies disclosed herein, and any one or more portions thereof.

[0012] The drawings are not necessarily to scale and may be illustrated by phantom lines, diagrammatic representations, and partial views. In some cases, details that are not necessary for an understanding of the embodiments or that make other details difficult to grasp may be omitted. Like reference numerals in the various drawings indicate like elements. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] In conventional video coding schemes, a video sequence is divided into Group-of-Pictures (GOPs). Each GOP is self-contained in the sense of temporal and spatial prediction. Usually, the first picture in the group is used as a reference picture for subsequent pictures. The temporal and spatial relationships between pictures allow very efficient compression using predictive coding.

[0014] In some scenarios during the transmission of video, it may become necessary to crop frames to be decoded, for example to reduce the bit rate to facilitate continuous playback. In many current state-of-the-art encoders, cropping is managed by re-encoding and re-transmitting an entire portion of the video, known as a group of pictures (GOP), which contains intra-frames (I-frames) that are used as references and information that can be used to predict other frames from the references. This requires transmitting intra-frames (I-frames), which may incur additional costs, since these frames account for a large portion of the bits in the GOP. As an illustrative example, in scenarios where a video transmission system experiences a sudden drop in available bandwidth, or where a video consumer is interested in a particular region in a video picture, it may be necessary to transmit and present a given region of interest in the video picture. Existing video coding schemes achieve this by discarding portions of the video already transmitted and re-encoding and re-transmitting the new video. For example, as illustrated in FIG. 1, both the first reference picture 104 and the second reference picture 108 may be encoded and transmitted as 112. This is not only inefficient but also induces delays at the decoder side that degrade the user experience.

[0015] The subject matter relates to adaptive cropping systems and processes, techniques that enable more flexibility for video encoders / decoders that allow bitrate savings in various use cases. In general, adaptive cropping combines cropping as signaled in the bitstream with adaptive scaling as signaled in the bitstream. Adaptive cropping may include performing prediction using a reference region of a reference frame as a reference for a current frame, where the reference region "subframe" has an area, e.g., defined in pixels, smaller than the area of ​​the reference frame. In current coding standards, predicted frames are generated from the entire reference frame. In adaptive cropping, a reference region in a reference frame may be used as a reference picture to generate a predicted frame. This approach may be used to reduce video resolution to focus on areas of interest to a viewer and / or to focus on areas identified by automated or user-friendly detection as containing more relevant visual data for some purpose and / or task, thus reducing the bitrate.

[0016] In some embodiments, adaptive cropping may be used for any number of frames at any position within a GOP, thus eliminating the requirement to re-encode and / or re-transmit I-frames.

[0017] Referring now to FIG. 2, an example embodiment of a frame 204 is described, which may include a reference frame having a sub-frame that represents a cropped portion of a video view. The sub-frame may be identified within the reference frame, for example, by a decoder, as described in further detail below. The sub-frame may be used as a reference region in a manner similar to the use of a cropped reference frame, without requiring a separate transmission of the cropped reference frame to generate the decoded frame 208. Alternatively, the reference frame may be used without cropping to generate the current frame, and may similarly be cropped and / or resized, as described in further detail below, without retransmission of the cropped reference frame. Advantages afforded by this approach may include saving bandwidth used in video transmission, saving resources used in encoding the video, and / or saving time required to decode and play the video. Devices and / or networks implementing the disclosed embodiments may result in a better user experience as well as a more efficient use of resources.

[0018] Referring now to FIG. 3, as one non-limiting and illustrative example, a reference frame may have a first resolution defined by a first width D1 and a first height D2, where D1 and D2 may be a number of units of measurement, such as, but not limited to, pixels and / or fractions of pixels. The area of ​​the reference frame may be defined as an area of ​​a rectangular array of units of measurement D1 and D2, defined as a resolution D1×D2. A sub-frame 304 may have a width W and a height H, defining an area or resolution of W×H. A sub-frame 304 may define a sub-picture within a reference picture having the same or smaller dimension, where "smaller dimension" means that at least one of W and H is smaller than the corresponding dimension of the reference frame. In other words, either W is smaller than D1 or H is smaller than D2. As a result, the resolution or area W×H may be smaller than the resolution or area D1×D2. The sub-frame 304 may be defined by a 4-tuple (X,Y,W,H), where X,Y are the coordinates of the top left corner of the sub-frame 304 relative to the top left corner of the reference picture, and W,H are the width and height of the sub-frame 304 expressed in units of measurement. It should be noted that alternative 4-tuple may be selected to define the sub-frame 304, such as, but not limited to, alternative corner coordinates of the sub-frame 304, a set of two diagonally opposed vertices, and / or a vector to any defined point. The data defining the sub-frame 304 may be static across a GOP. For example, the 4-tuple (X,Y,W,H) or equivalent may be static across a GOP. Alternatively or additionally, the data defining the sub-frame 304 may be dynamic. For example, but not limited to, the sub-frame 304 changes between subsequent pictures of the GOP to follow the movement of an object and / or person of interest within the video picture. Data defining the sub-frame 304 may be provided for each picture of a group of pictures.This may be accomplished, for example and without limitation, by a set of data defining the sub-frames 304 in one picture, by further data describing the movement of the sub-frames 304 from one picture to a previous or subsequent picture, or the like, as described above, by a set of data defining the sub-frames 304 for each picture of a group of pictures. The data defining the sub-frames 304 may be specified and / or signaled in a sequence parameter set (SPS). Update data defining the sub-frames 304 may be provided in a picture parameter set (PPS) for one or more selected pictures and / or frames of the GOP.

[0019] 3, the cropped sub-frame 304 may be signaled using at least one vertical offset and at least one horizontal offset. For example, but not by way of limitation, a 4-tuple as described above may specify a vertical offset from a top edge of the frame, a vertical offset from a bottom edge of the frame, a horizontal offset from a left edge of the frame, and a horizontal offset from a right edge of the frame, where the offsets may be measured in pixels of the frame either before or after rescaling as described in further detail below. As one non-limiting example, the at least one vertical offset may include sps_conf_win_top_offset and sps_conf_win_bottom_offset, which may be signaled in the SPS and may identify a vertical offset from a top edge of the frame and a vertical offset from a bottom edge of the frame, respectively. As a further non-limiting example, the at least one horizontal offset may include sps_conf_win_left_offset and sps_conf_win_right_offset, which may be signaled in the SPS and identify a horizontal offset from the left edge of the frame and a horizontal offset from the right edge of the frame, respectively.

[0020] Continuing to refer to FIG. 3, alternatively or additionally, a sub-frame 304 that may be used as a cropped frame may be identified by designation of one or more tiles or slices to be included and / or excluded from the sub-frame 304. The number and location of tiles within the frame may be signaled in the picture header. In one embodiment, the signaling may be explicit. Alternatively or additionally, the PPS may signal any or all of the tile rows, columns, row heights, and / or column widths that may be combined and / or utilized by a decoder to determine the tile count and / or number. For example, without limitation, a PPS parameter denoted as pps_num_exp_tile_columns_minus1, plus 1, may explicitly specify the number of tile column widths provided. As a further non-limiting example, the parameter pps_tile_column_width_minus1[i] incremented by 1 may specify the ith tile column width, e.g., in units of coding tree blocks (CTBs), for i in the range from 0 to pps_num_exp_tile_columns_minus1. The parameter pps_tile_row_height_minus1[i] incremented by 1 may specify the ith tile row height, e.g., in units of CTBs for i. Alternatively or additionally, the signaled parameters may specify the number and / or dimensions of slices in one or more tiles. For example, the parameter denoted by pps_num_exp_slices_in_tile[i] may specify the number of explicitly provided slice heights for slices in the tile containing the ith slice. The parameter denoted by pps_slice_width_in_tiles_minus1[i] incremented by 1 may specify the width of the ith rectangular slice in units of tile columns.A parameter denoted pps_slice_height_in_tiles_minus1[i] incremented by 1 may, for example, specify the height of the i-th rectangular slice in units of tile rows when pps_num_exp_slices_in_tile[i] is equal to 0. Those skilled in the art will recognize, upon review of this disclosure in its entirety, various alternative or additional ways in which tile and / or slice parameters may be signaled and / or determined, whether implicitly or explicitly, in and / or from the bitstream and / or header parameters.

[0021] 3, a frame may be divided into two or more sub-frames 304. The sub-frames 304 may be identified and / or signaled with one or more tiles and / or slices as described above, including but not limited to, by designating the tiles and / or slices included in a given sub-frame 304. One or more such sub-frames 304 may be used as or excluded from a cropped sub-frame 304 as described in this disclosure.

[0022] Continuing with reference to FIG. 3, adaptive cropping may be implemented as a mode. In case of activation of adaptive cropping mode at some point during decoding, the decoder may already receive a reference frame at resolution D1×D2 and may select a sub-region using a 4-tuple as described above. In some implementations, the encoder may signal to the decoder using extra bits in the bitstream. The signaling bits may be a 4-tuple of the reference frame index and / or an index identifying the GOP in a buffer that identifies the picture index at the decoder, and the sub-frame 304. The decoder may then perform a cropping operation on the reference picture as identified by the index according to the 4-tuple and a scaling operation on the optionally cropped sub-frame. Subsequent frames may be predicted from the cropped and optionally scaled sub-frame 304, or may be predicted from the reference frame before cropping and optionally scaling if the sub-frame 304 is a frame decoded as a function of the reference frame. As described above, if the data defining the subframe 304 is dynamic, the later frame may be further predicted using such data and the reference frame. Advantageously, a single reference frame may be used for the subframe 304 that moves relative to the reference frame without requiring retransmission of the reference frame. Cropping and / or resizing may be performed on the reference frame itself before decoding the current frame using the subframe 304. Alternatively or additionally, the current frame may be decoded using the reference frame and then cropped according to the signaling, as described above. Similarly, rescaling may be performed on the reference frame and / or subframe before decoding the current frame, or on the current frame after decoding.

[0023] Now referring to FIG. 4, the decoder may implement adaptive cropping in one of several modes. In an exemplary embodiment, the decoder may implement adaptive cropping using Reduced Resolution Mode (RRM), as illustrated in FIG. 4. The RMM predicted picture may be of the same resolution as the subframe. For example, if the subframe is of a smaller resolution than the reference frame, the predicted picture may be provided at the same, smaller resolution. In other words, the cropped current frame may maintain the resolution of the subframe without rescaling. All operations required to obtain the predicted picture, such as prediction, may be performed directly on the region and / or on the reference frame before cropping. The predicted picture may then be rescaled to a smaller or larger picture.

[0024] With further reference to FIG. 4, the width and height of the smaller or larger frame may be obtained by multiplying the reference width and height by an arbitrary rescaling constant (Rc), also called a scaling factor and / or constant, and alternatively or additionally referred to as a variable name such as RefPicScale. For smaller frames, Rc may have a value between 0 and 1. For larger frames, Rc may have a value greater than 1. For example, Rc may have a value between 1 and 4, or other values. The rescaling constant may be different from one resolution dimension to another. For example, a rescaling constant Rch may be used to rescale the height, and another rescaling constant Rcw may be used to rescale the width.

[0025] Also, referring to Figure 4, rescaling may be implemented as a mode. In some implementations, the encoder may signal to the decoder the rescaling constant to be used as a function of picture parameters, such as, for example, the pps_pic_width_in_luma_samples, pps_scaling_win_right_offset, and / or pps_scaling_win_left_offset parameters. The signaling may be performed in the sequence parameter set (SPS) corresponding to the GOP that contains the current picture and / or in the picture parameter set (PPS) corresponding to the current picture. For example, without limitation, an encoder may signal the rescaled parameters using fields such as pps_pic_width_in_luma_samples, pps_pic_height_in_luma_samples, pps_scaling_win_left_offset, pps_scaling_win_right_offset, pps_scaling_win_top_offset, pps_scaling_win_bottom_offset, and / or sps_num_subpics_minus1. A parameter such as pps_scaling_window_explicit_signalling_flag equal to 1 may specify that a scaling window offset parameter is present in the PPS. A pps_scaling_window_explicit_signalling_flag equal to 0 may indicate that a scaling window offset parameter is not present in the PPS. When sps_ref_pic_resampling_enabled_flag is equal to 0, the value of pps_scaling_window_explicit_signalling_flag may be 0.pps_scaling_win_left_offset, pps_scaling_win_right_offset, pps_scaling_win_top_offset, and pps_scaling_win_bottom_offset may specify offsets applied to the picture size for scaling ratio calculation. When not present, the values ​​of pps_scaling_win_left_offset, pps_scaling_win_right_offset, pps_scaling_win_top_offset, and pps_scaling_win_bottom_offset may be inferred to be equal to pps_conf_win_left_offset, pps_conf_win_right_offset, pps_conf_win_top_offset, and pps_conf_win_bottom_offset, respectively.

[0026] 4, as described above, the W and H parameters may be represented using, but are not limited to, the variables CurrPicScalWinWidthL and CurrPicScalWinHeightL, respectively. These variables may be derived from the signaled parameters, as described above, using one or more mathematical relationships between the signaled parameters and the variables. For example, but not limited to, CurrPicScalWinWidthL may be derived according to the following formula: CurrPicScalWinWidthL = pps_pic_width_in_luma_samples - SubWidthC * (pps_scaling_win_right_offset + pps_scaling_win_left_offset)

[0027] As a further non-limiting example, CurrPicScalWinHeightL may be derived according to the following formula: CurrPicScalWinWidthL = pps_pic_width_in_luma_samples - SubWidthC * (pps_scaling_win_right_offset + pps_scaling_win_left_offset)

[0028] Those skilled in the art will be aware of various alternative calculations that may be used to derive the above-mentioned variables upon reviewing this disclosure in its entirety. Alternatively or additionally, the encoder may directly signal one or more such variables, Rc, Rch, and / or Rcw, for example, but not limited to, in the PPS and / or SPS.

[0029] Also referring to FIG. 4, alternatively or additionally, as described above, the rescaling constant and / or set of rescaling constants may be signaled in the bitstream using a reference to an index of one or more stored scaling constants and / or a reference to an index of a frame and / or block signaled using one or more previously signaled and / or utilized scaling constants. The reference to an index of the stored scaling constant may be explicitly signaled and / or determined from one or more additional parameters signaled in the bitstream. For example, but not by way of limitation, the decoder may identify a reference frame and / or group of pictures, including the current frame. If a rescaling constant has previously been signaled and / or used in such group of pictures, with a reference frame signaled as applicable to the current frame and / or current group of pictures, or the like, the decoder may identify that rescaling constant for use as a rescaling constant in the current frame.

[0030] Continuing with reference to FIG. 4, in some implementations, the rescaling operation may be performed at the block level of the coded frame and / or subframe. For example, the reference frame may be first rescaled and then prediction may be performed. The block prediction process may be performed on the scaled reference frame (having the scaled resolution) rather than the original reference frame. Rescaling the reference frame may include rescaling according to any parameters signaled by the encoder, as described above. For example, if a reference frame to be used with the current picture has been signaled, such as, but not limited to, via a reference to an index value associated with the reference frame, or the like, the signaled reference frame may be rescaled according to any of the methods of rescaling described above prior to prediction. The rescaled reference frame may be stored in a memory and / or buffer, which may include, but is not limited to, a buffer that identifies the frame contained therein by an index according to which a frame search is performed. The buffer may include a decoded picture buffer (DCB) and / or one or more additional buffers implemented by the decoder. The prediction process may include, for example, inter-picture prediction including motion compensation.

[0031] Also, referring to FIG. 4, some implementations of block-based rescaling may allow flexibility to apply optimal filters to each block instead of applying the same filter to the entire frame. In some implementations, the skip-rescaling mode may allow some blocks to be in skip-rescaling mode (such that rescaling does not change the bitrate) (e.g., based on pixel uniformity and bitrate cost). The skip-rescaling mode may be signaled in the bitstream. For example, but not limited to, the skip-rescaling mode may be signaled in the PPS parameter. Alternatively or additionally, the decoder may determine that the skip-rescaling mode is active based on one or more parameters set by the decoder and / or signaled in the bitstream.

[0032] Also, referring to FIG. 4, the spatial filter used in the rescaling may include, but is not limited to, a bicubic spatial filter applying bicubic interpolation, a bilinear spatial filter applying a bilinear interpretation, a sinc filter, a Lanczos filter using Lanczos filtering and / or Lanczos resampling using a combination of sinc function interpolation and / or signal reconstruction techniques, or the like. Those skilled in the art will recognize various filters that may be used for interpolation consistent with the present disclosure upon reviewing the entirety of the present disclosure. As one non-limiting example, the interpolation filter may include any of the filters described above, a low pass filter, which may be used as, but is not limited to, an upsampling process in which pixels between pixels of a block and / or frame before scaling may be initialized to zero, and then the output of the low pass filter may be input. Alternatively or additionally, any luma sample interpolation filtering process may be used. The luma sample interpretation may include the calculation of an interpolation value at a half-sample interpolation filter index that is located between two consecutive sample values ​​of the unscaled sample array. The calculation of the interpolated value may be performed, without limitation, by retrieving coefficients and / or weights from a lookup table. The selection of the lookup table may be performed as a function of the motion model of the coding unit and / or the amount of scaling ratio, for example, as determined using a scaling constant as described above. The calculation may include, without limitation, performing a weighted sum of adjacent pixel values, where the weights are retrieved from a lookup table. Alternatively or additionally, the calculated value may be shifted. For example, without limitation, the value may be shifted by Min(4,BitDepth-8), 6, Max(2,14-BitDepth), or the like. Those skilled in the art will recognize various alternative or additional implementations that may be used for the interpolation filter upon reviewing the entirety of this disclosure.

[0033] Referring now to FIG. 5, the decoder may implement adaptive cropping using Same Resolution Mode (SRM). In SRM, the sub-frames may be rescaled, e.g., using a rescaling constant as described above, to match the full resolution of the video. For example, W and H may each be multiplied by a selected Rc to scale W and H to the same size as D1 and D2 as described above, such as, but not limited to, Rc=D1 / W. Prediction and other operations may be performed to obtain a predicted picture using the rescaled sub-frames.

[0034] Referring now to FIG. 6, an example embodiment of a method 600 of adaptive cropping is disclosed. At step 605, a bitstream is received at a decoder. A current frame including a current block may be included in the bitstream that the decoder receives. The bitstream may include data found in a stream of bits that is input to the decoder when using data compression, for example. The bitstream may include information necessary to decode the video. Receiving may include extracting and / or parsing the block and associated signaling information from the bitstream. In some implementations, the current block may include a coding tree unit (CTU), a coding unit (CU), or a prediction unit (PU).

[0035] Continuing to refer to FIG. 6, at step 610, a first frame is identified using the bitstream. Cropped subframes of the first frame are identified as a function of the bitstream. Identifying the cropped subframe may include receiving, in the bitstream, a geometric characterization of the cropped subframe. This may be implemented in any manner described in this disclosure, such as, but not limited to, a 4-tuple such as (X,Y,W,H), as described above. The geometric characterization may include updates to data defining the cropped subframe, where the position of the cropped subframe relative to a reference frame is dynamic, as described above.

[0036] 6, in step 615, the decoder determines a scaling constant associated with the cropped sub-frame as a function of the bitstream, which may be implemented, but is not limited to, as described above.

[0037] 6, in step 620, the decoder reconstructs pixel data for the cropped sub-frame using the scaling constant, which may be implemented as, but is not limited to, described above.

[0038] Also, referring to FIG. 6, in some implementations, the sub-frames may be represented as pictures within a GOP, for example.

[0039] Continuing with reference to FIG. 6, in some implementations, the second frame or a portion thereof in an adaptive cropping mode using the subframe as a reference frame. The decoding may include, but is not limited to, decoding with RRM as described above, i.e., in other words, performing a decoding operation without rescaling the subframe. The decoding may include, but is not limited to, decoding with SRM as described above. The decoding may include generating a scaled reference frame using the scaled subframe and a scaling constant, and decoding the second frame or a portion thereof using the scaled reference frame.

[0040] 7 is a system block diagram illustrating an example decoder 700 capable of adaptive cropping. The decoder 700 may include an entropy decoder processor 704, an inverse quantization and inverse transform processor 708, a deblocking filter 712, a frame buffer 716, a motion compensation processor 720, and / or an intra prediction processor 724.

[0041] Also referring to FIG. 7, in operation, a bitstream 728 may be received by the decoder 700 and input to an entropy decoder processor 704, which may entropy decode a portion of the bitstream into quantized coefficients. The quantized coefficients may be provided to an inverse quantization and inverse transform processor 708, which may perform inverse quantization and inverse transform to generate a residual signal, which may be added to the output of a motion compensation processor 720 or an intra-prediction processor 724 according to a processing mode. The output of the motion compensation processor 720 and the intra-prediction processor 724 includes block prediction values ​​based on previously decoded blocks. The sum of the prediction value and the residual may be processed by a deblocking filter 712 and stored in a frame buffer 716.

[0042] Also, referring to FIG. 7, in one embodiment, the decoder 700 may include circuitry configured to implement any of the operations as described above in any of the embodiments as described above, in any order and with any degree of repetition. For example, the decoder 700 may be configured to repeatedly execute a single step or sequence until a desired or commanded result is achieved. The repetition of a step or sequence of steps may be performed iteratively and / or recursively, using the output of a previous iteration as an input to a subsequent iteration, aggregating the inputs and / or outputs of an iteration to generate an aggregate result, reducing or decrementing one or more variables such as global variables, and / or dividing a large processing task into a set of smaller processing tasks that are addressed iteratively. The decoder may perform any step or sequence of steps as described in this disclosure in parallel, using two or more parallel threads, processor cores, or the like to perform a step two or more times, simultaneously and / or substantially simultaneously. The division of tasks among parallel threads and / or processes may be performed according to any protocol suitable for dividing tasks among iterations. Those skilled in the art will recognize, upon reviewing this disclosure in its entirety, the various ways in which steps, sequences of steps, processing tasks, and / or data may be subdivided, shared, or otherwise handled using iterative, recursive, and / or parallel processing.

[0043] 8 is a process flow diagram illustrating one embodiment of a process 800 for encoding video with adaptive cropping, which may allow for more flexibility for video encoders / decoders enabling bitrate savings in various use cases. At step 805, a video frame may undergo initial block partitioning, for example, using a tree-structured macroblock partitioning scheme, which may include partitioning a picture frame into CTUs and CUs.

[0044] Also referring to FIG. 8, at step 810, a block-based adaptive cropping may be performed, including selection of a subframe of a frame or a portion thereof. The region may be selected by means of automatic or expert input. As one non-limiting example, automatic selection may be achieved by a computer vision algorithm that detects a particular object. Object detection may include further processing, such as object classification, or the like. Expert input selection may be achieved using manual human intervention, such as, but not limited to, selecting a close-up of a person and / or object of interest in a video, such as a person in a surveillance video. Another possible use case may be to select a region of maximum interest that contributes most to bitrate reduction. The adaptive cropping may further include selection of a geometric characterization of the subframe. For example, but not limited to, selection of a geometric characterization of the subframe may include selection of a 4-tuple as described above, such as, but not limited to (X, Y, W, H). The selection of the geometric characterization of the subframe may include update information and / or information indicating changes to the data defining the subframe from one frame to another, as described above with respect to the dynamic data defining the subframe.

[0045] Also referring to FIG. 8, at step 815, the block may be coded and included in a bitstream. The coding may include, for example, utilizing inter-prediction and intra-prediction modes. The coding may include adding bits to the bitstream characterization (X, Y, W, H), identifying an adaptive cropping mode, or the like, as described above. The coding may include encoding update information, as described above with respect to dynamic data defining a subframe, and / or information indicating changes to data defining a subframe from one frame to another. In an embodiment, the encoder may determine and transmit a rescaling constant, as described above.

[0046] FIG. 9 is a system block diagram illustrating an example video encoder 900 capable of adaptive cropping. The example video encoder 900 may receive an input video 904, which may be initially segmented or divided according to a processing scheme, such as a tree-structured macroblock partition scheme (e.g., quadtree plus binary tree). One example of a tree-structured macroblock partition scheme may include dividing a picture frame into large block elements called coding tree units (CTUs). In some implementations, each CTU may be further divided one or more times into multiple sub-blocks called coding units (CUs). The end result of this division may include a group of sub-blocks, which may be called prediction units (PUs). Transform units (TUs) may also be utilized.

[0047] 9, the example video encoder 900 may include an intra prediction processor 908, a motion estimation / compensation processor 912, which may be referred to as an inter prediction processor, capable of constructing a motion vector candidate list, including adding global motion vector candidates to the motion vector candidate list, a transform / quantization processor 916, an inverse quantization / inverse transform processor 920, an in-loop filter 924, a decoded picture buffer 928, and / or an entropy coding processor 932. Bitstream parameters may be input to the entropy coding processor 932 for inclusion in an output bitstream 936.

[0048] Continuing to refer to FIG. 9, an operation may determine, for each block of a frame of the input video 904, whether to process the block via intra-picture prediction or to process the block using motion estimation / compensation. The block may be provided to an intra-prediction processor 908 or a motion estimation / compensation processor 912. If the block is to be processed via intra-prediction, the intra-prediction processor 908 may perform processing to output a predictor. If the block is to be processed via motion estimation / compensation, the motion estimation / compensation processor 912 may perform processing including constructing a motion vector candidate list, including adding a global motion vector candidate to the motion vector candidate list, if applicable.

[0049] 9, the residual may be formed by subtracting a predictor from the input video. The residual may be received by a transform / quantization processor 916, which may perform a transform operation (e.g., a discrete cosine transform (DCT)) to generate coefficients, which may be quantized. The quantized coefficients and any associated signaling information may be provided to an entropy coding processor 932, where they may be entropy coded and included in the output bitstream 936. The entropy coding processor 932 may support coding of signaling information associated with coding the current block. Further, the quantized coefficients may be provided to an inverse quantization / inverse transform processor 920, which may regenerate pixels, which may be combined with a predictor and processed by an in-loop filter 924, the output of which may be stored in a decoded picture buffer 928 for use by the motion estimation / compensation processor 912, which may construct a motion vector candidate list, including adding global motion vector candidates to the motion vector candidate list.

[0050] 9, although several variations have been described in detail above, other modifications or additions are possible. For example, in some implementations, the current block may include any symmetric block (8x8, 16x16, 32x32, 64x64, 128x128, etc.) as well as any asymmetric block (8x4, 16x8, etc.).

[0051] Also, referring to FIG. 9, in some implementations, a quad-tree plus binary decision tree (QTBT) may be implemented. In QTBT, at the coding tree unit level, partition parameters of the QTBT may be dynamically derived to adapt to local characteristics without transmitting any overhead. Then, at the coding unit level, a joint classifier decision tree structure may eliminate unnecessary repetitions and control the risk of erroneous prediction. In some implementations, an LTR frame block update mode may be available as an additional selection available at each leaf node of the QTBT.

[0052] Also, referring to Figure 9, in some implementations, additional syntax elements may be signaled at different hierarchical levels of the bitstream. For example, a flag may be valid for an entire sequence by including an enable flag coded in the sequence parameter set (SPS). Additionally, a CTU flag may be coded at the coding tree unit (CTU) level.

[0053] Some embodiments may include a non-transitory computer program product (i.e., a tangibly embodied computer program product) storing instructions that, when executed by one or more data processors of one or more computing systems, cause at least one data processor to perform operations described herein.

[0054] Also, with reference to FIG. 9, the encoder 900 may include circuitry configured to implement any of the operations as described above with reference to FIG. 8 or FIG. 9 in any embodiment, in any order and with any degree of repetition. For example, the encoder 900 may be configured to repeatedly execute a single step or sequence until a desired or commanded result is achieved. The repetition of a step or sequence of steps may be performed iteratively and / or recursively, using the output of a previous iteration as an input to a subsequent iteration, aggregating the inputs and / or outputs of an iteration to generate an aggregate result, reducing or decrementing one or more variables such as global variables, and / or dividing a large processing task into a set of smaller processing tasks that are addressed iteratively. The encoder 900 may perform any step or sequence of steps as described in this disclosure in parallel, such as performing a step two or more times, simultaneously and / or substantially simultaneously, using two or more parallel threads, processor cores, or the like. The division of tasks between parallel threads and / or processes may be performed according to any protocol suitable for dividing tasks between iterations. Those skilled in the art will recognize, upon reviewing this disclosure in its entirety, the various ways in which steps, sequences of steps, processing tasks, and / or data may be subdivided, shared, or otherwise handled using iterative, recursive, and / or parallel processing.

[0055] Continuing with reference to FIG. 9, a non-transitory computer program product (i.e., a physically embodied computer program product) may store instructions that, when executed by one or more data processors of one or more computing systems, cause at least one data processor to perform operations and / or steps thereof described herein, including, but not limited to, may be configured to perform any of the operations described above with reference to FIG. 6 and FIG. 8, and / or any of the operations of the decoder 700 and / or the encoder 900. Similarly, a computer system is described that may also include one or more data processors and a memory coupled to the one or more data processors. The memory may temporarily or permanently store instructions that cause at least one processor to perform one or more operations described herein. Furthermore, the method may be implemented either by one or more data processors in a single computing system or by one or more data processors distributed among two or more computing systems. Such computing systems may be connected and exchange data and / or commands or other instructions, or the like, via one or more connections, including connections via a network (e.g., the Internet, a wireless wide area network, a local area network, a wide area network, a wired network, or the like), via direct connections between one or more of the multiple computing systems, or the like.

[0056] Embodiments disclosed herein include a circuit configured to receive a bitstream, identify a current frame and a cropped sub-frame of the current frame as a function of the bitstream, determine a scaling constant associated with the cropped sub-frame as a function of the bitstream, and reconstruct pixel data for the cropped sub-frame using the scaling constant.

[0057] In an embodiment, the cropped subframe may be signaled in the bitstream. The cropped subframe may be signaled in a sequence parameter set (SPS). The cropped subframe may be signaled using at least one vertical offset and at least one horizontal offset. The at least one vertical offset may include sps_conf_win_top_offset and sps_conf_win_bottom_offset, and the at least one horizontal offset may include sps_conf_win_left_offset and sps_conf_win_right_offset. The scaling constant may include a vertical scaling component and a horizontal scaling component. The scaling constant may be signaled in the bitstream. The scaling constant is signaled in a picture parameter set (PPS). The scaling constant may be signaled as a function of the pps_pic_width_in_luma_samples, pps_scaling_win_right_offset, and pps_scaling_win_left_offset parameters. The decoder may be configured to scale the cropped sub-frame by interpolating pixel values.

[0058] In one embodiment, a method includes receiving, by a decoder, a bitstream; identifying, by the decoder and as a function of the bitstream, a current frame and a cropped sub-frame of the current frame; determining, by the decoder and as a function of the bitstream, a scaling constant associated with the cropped sub-frame; and reconstructing, by the decoder, pixel data of the cropped sub-frame using the scaling constant.

[0059] In an embodiment, the cropped subframe may be signaled in the bitstream. The cropped subframe may be signaled in a sequence parameter set (SPS). The cropped subframe may be signaled using at least one vertical offset and at least one horizontal offset. The at least one vertical offset may include sps_conf_win_top_offset and sps_conf_win_bottom_offset, and the at least one horizontal offset may include sps_conf_win_left_offset and sps_conf_win_right_offset. The scaling constant may include a vertical scaling component and a horizontal scaling component. The scaling constant may be signaled in the bitstream. The scaling constant is signaled in a picture parameter set (PPS). The scaling constant may be signaled as a function of the pps_pic_width_in_luma_samples, pps_scaling_win_right_offset, and pps_scaling_win_left_offset parameters. The decoder may be configured to scale the cropped sub-frame by interpolating pixel values.

[0060] It should be noted that any one or more aspects and embodiments described herein may be advantageously implemented using digital electronic circuitry, integrated circuits, specially designed application specific integrated circuits (ASICs), field programmable gate array (FPGA) computer hardware, firmware, software, and / or combinations thereof, as embodied and / or implemented in one or more machines (e.g., one or more computing devices utilized as user computing devices for electronic documents, one or more server devices, such as document servers, etc.) programmed according to the teachings herein, as would be apparent to one of ordinary skill in the computer arts. These various aspects or features may include implementation in one or more computer programs and / or software executable and / or interpretable on a programmable system that may be specialized or general purpose, including at least one programmable processor, at least one input device, and at least one output device, coupled to receive data and instructions from and transmit data and instructions to a storage system. Appropriate software coding may be readily produced by skilled programmers based on the teachings of the present disclosure, as would be apparent to one of ordinary skill in the software arts. The aspects and implementations described above employing software and / or software modules may also include appropriate hardware to support the implementation of the machine-executable instructions of the software and / or software modules.

[0061] Such software may be a computer program product employing a machine-readable storage medium. A machine-readable storage medium may be any medium capable of storing and / or encoding a sequence of instructions for execution by a machine (e.g., a computing device) and causing the machine to perform any one of the methodologies and / or embodiments described herein. Examples of machine-readable storage media include, but are not limited to, magnetic disks, optical disks (e.g., CDs, CD-Rs, DVDs, DVD-Rs, etc.), magneto-optical disks, read-only memory "ROM" devices, random access memory "RAM" devices, magnetic cards, optical cards, solid-state memory devices, EPROMs, EEPROMs, programmable logic devices (PLDs), and / or any combination thereof. As used herein, a machine-readable medium is intended to include a single medium as well as a collection of physically separate media, such as, for example, a collection of compact discs or one or more hard disk drives in combination with a computer memory. As used herein, a machine-readable storage medium does not include a transitory form of signal transmission.

[0062] Such software may also include information (e.g., data) carried as a data signal on a data carrier, such as a carrier wave. For example, machine-executable information may be included as a data carrier signal embodied in a data carrier, which signal encodes sequences of instructions, or portions thereof, for execution by a machine (e.g., a computing device) and any associated information (e.g., data structures and data) that cause the machine to perform any one of the methodologies and / or embodiments described herein.

[0063] Examples of computing devices include, but are not limited to, e-book reading devices, computer workstations, terminal computers, server computers, mobile devices (e.g., tablet computers, smartphones, etc.), web appliances, network routers, network switches, network bridges, any machine capable of executing sequences of instructions that specify actions to be taken by the machine, and any combination thereof. In one example, a computing device may include and / or be included in a kiosk.

[0064] 8 illustrates a diagrammatic representation of one embodiment of a computing device in the exemplary form of a computer system 800 on which a set of instructions for causing a control system to perform any one or more aspects and / or methodologies of the present disclosure may be executed. It is also contemplated that multiple computing devices may be utilized to execute a set of instructions specifically configured to cause one or more devices to perform any one or more aspects and / or methodologies of the present disclosure. Computer system 800 includes a processor 804 and memory 808 that communicate with each other and with other components via a bus 812. Bus 812 may include any of several types of bus structures, including, but not limited to, a memory bus, a memory controller, a peripheral bus, a local bus, and any combination thereof, using any of a variety of bus architectures.

[0065] Memory 808 may include a variety of components (e.g., machine-readable media), including but not limited to random access memory components, read-only components, and any combination thereof. In one example, a basic input / output system 816 (BIOS), including the basic routines that help to transfer information between elements within computer system 800, such as during start-up, may be stored in memory 808. Memory 808 may also include (e.g., stored on one or more machine-readable media) instructions (e.g., software) 820 that embody any one or more aspects and / or methodologies of the present disclosure. In another example, memory 808 may further include any number of program modules, including but not limited to an operating system, one or more application programs, other program modules, program data, and any combination thereof.

[0066] Computer system 800 may also include a storage device 824. Examples of storage devices (e.g., storage device 824) include, but are not limited to, hard disk drives, magnetic disk drives, optical disk drives in combination with optical media, solid-state memory devices, and any combination thereof. Storage device 824 may be connected to bus 812 by an appropriate interface (not shown). Exemplary interfaces include, but are not limited to, SCSI, Advanced Technology Attachment (ATA), Serial ATA, Universal Serial Bus (USB), IEEE 1394 (FIREWIRE), and any combination thereof. In one example, storage device 824 (or one or more components thereof) may be removably interfaced with computer system 800 (e.g., via an external port connector (not shown)). In particular, storage device 824 and associated machine-readable media 828 may provide non-volatile and / or volatile storage of machine-readable instructions, data structures, program modules, and / or other data for computer system 800. In one example, the software 820 may reside, completely or partially, within the machine-readable medium 828. In another example, the software 820 may reside, completely or partially, within the processor 804.

[0067] Computer system 800 may also include input devices 832. In one embodiment, a user of computer system 800 may input commands and / or other information to computer system 800 via input devices 832. Examples of input devices 832 include, but are not limited to, alphanumeric input devices (e.g., keyboards), pointing devices, joysticks, gamepads, audio input devices (e.g., microphones, voice response systems, etc.), cursor control devices (e.g., mice), touchpads, optical scanners, video capture devices (e.g., still cameras, video cameras), touch screens, and any combination thereof. Input devices 832 may be interfaced to bus 812 via a variety of interfaces (not shown), including, but not limited to, a serial interface, a parallel interface, a game port, a USB interface, a FIREWIRE interface, a direct interface to bus 812, and any combination thereof. Input devices 832 may include a touch screen interface, which may be part of display 836 or may be separate, as discussed further below. The input device 832 may be utilized as a user selection device for selecting one or more graphical representations in the graphical interface, as described above.

[0068] A user may also input commands and / or other information to computer system 800 via storage device 824 (e.g., a removable disk drive, flash drive, etc.) and / or network interface device 840. A network interface device, such as network interface device 840, may be utilized to connect computer system 800 to one or more various networks, such as network 844, and one or more remote devices 848 connected thereto. Examples of network interface devices include, but are not limited to, a network interface card (e.g., a mobile network interface card, a LAN card), a modem, and any combination thereof. Examples of networks include, but are not limited to, a wide area network (e.g., the Internet, an enterprise network), a local area network (e.g., a network associated with an office, building, campus, or other relatively small geographic space), a telephone network, a data network associated with a telephone / voice provider (e.g., a mobile communications provider's data and / or voice network), a direct connection between two computing devices, and any combination thereof. A network, such as network 844, may employ wired and / or wireless modes of communication. In general, any network topology may be used. Information (eg, data, software 820 , etc.) may be communicated to and / or from computer system 800 via network interface device 840 .

[0069] The computer system 800 may further include a video display adapter 852 for communicating images displayable on a display device, such as the display device 836. Examples of display devices include, but are not limited to, a liquid crystal display (LCD), a cathode ray tube (CRT), a plasma display, a light emitting diode (LED) display, and any combination thereof. The display adapter 852 and the display device 836 may be utilized in combination with the processor 804 to provide graphical representations of aspects of the present disclosure. In addition to a display device, the computer system 800 may include one or more other peripheral output devices, including, but not limited to, audio speakers, a printer, and any combination thereof. Such peripheral output devices may be connected to the bus 812 via a peripheral interface 856. Examples of peripheral interfaces include, but are not limited to, a serial port, a USB connection, a FIREWIRE connection, a parallel connection, and any combination thereof.

[0070] The foregoing is a detailed description of exemplary embodiments of the present invention. Various modifications and additions may be made without departing from the spirit and scope of the present invention. Each feature of the various embodiments described above may be combined with features of other described embodiments as necessary to provide various combinations of features in related new embodiments. Moreover, the foregoing describes a number of separate embodiments, and what has been described herein is merely illustrative of the application of the principles of the present invention. Furthermore, certain methods herein may be described and / or described as being performed in a particular order, which order is highly variable within ordinary skill in the art to achieve the embodiments as disclosed herein. Thus, the present description is to be taken as an example only, and is not intended to otherwise limit the scope of the present invention.

[0071] In the above and in the claims, phrases such as "at least one of" or "one or more of" may appear following a connected list of elements or features. The term "and / or" may also appear in a list of two or more elements or features. Unless otherwise implicitly or explicitly contradicted by the context in which it is used, such phrases are intended to mean any listed element or feature individually or in combination with any other listed element or feature. For example, the phrases "at least one of A and B," "one or more of A and B," and "A and / or B" are intended to mean "A alone, B alone, or A and B together," respectively. A similar interpretation is also intended for lists containing more than two items. For example, the phrases "at least one of A, B, and C," "one or more of A, B, and C," and "A, B, and / or C" are intended to mean "A alone, B alone, C alone, A and B together, A and C together, B and C together, or A, B and C together." Additionally, use of the term "based on" above and in the claims is intended to mean "based at least in part on," allowing for unrecited features or elements.

[0072] The subject matter described herein may be embodied in a system, an apparatus, a method, and / or an article, depending on the desired configuration. The embodiments described in the preceding description do not represent all implementations consistent with the subject matter described herein. Instead, they are merely some examples consistent with aspects associated with the subject matter described. Although some variations have been described in detail above, other modifications or additions are possible. In particular, further features and / or variations may be provided in addition to those described herein. For example, the implementations described above may be directed to various combinations and subcombinations of the disclosed features, and / or combinations and subcombinations of some further features disclosed above. Furthermore, the logic flows depicted in the accompanying figures and / or described herein do not necessarily require the particular order shown, or sequence, to achieve the desired results. Other implementations may be within the scope of the following claims.

Claims

1. receiving a bitstream; identifying a current frame and a sub-frame of the current frame as a function of the bitstream; determining a scaling constant associated with the subframe as a function of the bitstream, the scaling constant being signaled within the bitstream and the scaling constant being signaled as a function of a picture width in luma samples, a scaling window right offset, and a scaling window left offset parameter; reconstructing pixel data for the sub-frame using the scaling constant; and A decoder comprising:

2. The decoder of claim 1 , wherein the subframes are signaled in a sequence parameter set (SPS).

3. The decoder of claim 1 , wherein the scaling constant comprises a vertical scaling component and a horizontal scaling component.

4. The decoder of claim 1 , wherein the scaling constant is signaled in a Picture Parameter Set (PPS).

5. The decoder of claim 1 , wherein the decoder is configured to scale the sub-frames by interpolating pixel values.

Images