Video downscaling for machine consumption

Abstract

NOT VISIBLE DUE TO STATUS OF PATENT

Description

VIDEO DOWNSCALING FOR MACHINE CONSUMPTIONTECHNICAL FIELD

[0001] Disclosed are embodiments related to video coding.BACKGROUNDVersatile Video Coding (WC)

[0002] Versatile Video Coding (VVC) is a block-based video codec standardized byInternational Telecommunication Union-Telecommunication (ITU-T) and Motion Picture Experts Group (MPEG) that utilizes both temporal and spatial prediction. Spatial prediction is achieved using intra (I) prediction from within a current picture. Temporal prediction is achieved using uni-directional (P) or bi-directional inter (B) prediction on a block level from previously decoded reference pictures. In the encoder, the difference between the original sample data and the predicted sample data, referred to as the residual, is transformed into the frequency domain, quantized, and then entropy coded before being transmitted together with necessary prediction parameters such as prediction mode and motion vectors, which are also entropy coded. The decoder performs entropy decoding, inverse quantization, and inverse transformation to obtain the residual and then adds the residual to an intra or inter prediction to reconstruct a picture.Reference Picture Resampling (RPR)

[0003] Reference picture resampling (RPR) is a VVC tool that can be used to enable switching between different resolutions in a video bitstream without encoding a startup of a new sequence with an intra picture. This gives more flexibility to adapt resolution to control bitrate which can be of use in for example video conferencing or adaptive streaming. RPR can make use of previously encoded pictures of lower or higher resolution than the current picture to be encoded by re-scaling them to the resolution of the current picture as part of inter prediction of the current picture.

[0004] The WC codec, besides providing around 50% higher compression efficiency compared to its predecessor, caters to various use cases such as screen content, high dynamic range coding, and scalable coding, among others.

[0005] One area that is currently being investigated by the Joint Video Experts Team(JVET), i.e., the team that developed WC, is the applicability of WC to machine vision applications. The group is currently studying algorithms that can provide compression benefits but do not change the decoding process, i.e., are conforming to the WC standard.Video Coding for Machines (VCM)

[0006] As noted above, one area that has gotten more and more attention in recent years is video coding for machines (VCM). Here videos are compressed in a manner that is not optimized for human consumption, but rather optimized for analysis by machine vision algorithms, for example neural networks.

[0007] One aspect that should be considered when compressing a video is the intended audience. While most codecs (and encoders) are optimized towards the human visual system, it is not clear that the same optimizations are beneficial for machine vision applications. Thus, specific algorithms are needed to optimize encoders for VCM use cases.

[0008] Machine vision algorithms that are often utilized are for example object detection or tracking. An object detection algorithm identifies objects that it can find in the current picture and describes them, often as a bounding box with the position, the size, and a label indicating the type of object. An object tracking task usually works similar to a detection task, but also assigns an identifier to each object, with the same object in different pictures being assigned the same identifier.Comparison of Bounding Boxes

[0009] In many cases, such as comparing different detection algorithms, it is necessary to compare bounding boxes to determine whether they represent the same object. The first step is to compare the labels associated with the bounding boxes because, if the labels are different, then the bounding boxes do not describe the same object. The second step is to compare the size and position of the bounding boxes, and this is often done by determining the “intersection over union (IoU)”. The loU is determined by dividing the overlap of the two bounding boxes, i.e., the intersection, by the union, i.e., the total area that is covered by the two bounding boxes. The resulting value can be compared to a threshold which is previously defined to mark the minimal value when two bounding boxes can be considered to describe the same object.Downscaling

[0010] Downscaling a picture refers to reducing the size of the picture. A downscaling factor is a multiple by which the size of the picture is reduced in the vertical direction, in the horizontal direction, or in both directions. For example, a downscaling factor of 4 applied to both the vertical and horizontal directions, also known as a 4x downsampling, reduces a 160x160 pixel picture to a 160 / 4x160 / 4 = 40x40 pixel picture. Hence, a lx downsampling does not reduce the size of the picture. As another example, a downscaling factor of 4 applied only to the vertical direction, also known as a 4x vertical downsampling, reduces a 160x160 pixel picture to a 160x160 / 4 = 160x40 pixel picture. Likewise a downscaling factor of 4 applied only to the horizontal direction, also known as a 4x horizontal downsampling, reduces a 160x160 pixel picture to a 160 / 4x160 = 40x160 pixel picture. As yet another example, a downscaling factor of 4 applied to the horizontal direction and a downscaling factor of 2 applied to the vertical direction reduces a 160x160 pixel picture to a 160 / 4x160 / 2 = 40x80 pixel pictureKeyframe analysis

[0011] Keyframe analysis generally comprises analyzing a picture, which is also known as the “keyframe”, to make a decision for other pictures, such as, for example, a set of pictures, which may be referred to as a segment, following the keyframe in encoding order. What is being decided depends on the implementation and the algorithm.

[0012] One example algorithm that analyzes a keyframe to make a decision for future pictures is the group of pictures (GOP)-based RPR encoder control that exist in the VVC reference software VTM-22.0. It can be enabled by setting the configuration parameter “GOPBasedRPR” equal to 1. The GOP-based RPR encoder controls analysis a first picture in a group of pictures (GOP). In random-access configuration the GOP refers to a picture coding hierarchy of length 32. The encoder control determines the encoding resolution based on resampling performance of the first picture in a GOP and the quantization parameter (QP) and then uses the determined resolution for the encoding all of the pictures in the GOP.

[0013] The resampling performance is measured in Peak Signal to Noise Ratio (PSNR) by downscaling to half resolution in both dimensions and then upscaling back to source resolution. If the PSNR is greater than a first threshold, then the pictures of the GOP are encoded in 1 / 2 resolution in both dimensions, else if the PSNR is greater than a second threshold, then thepictures are encoded in 2 / 3 resolution, else if the PSNR is greater than a third threshold, then the pictures are encoded in 4 / 5 resolution, otherwise the pictures are encoded in the source resolution. The thresholds are reduced with increasing QP so that they are easier to pass when QP is higher. The determined resolution can be one of source resolution or reduced resolution to 1 / 2, 2 / 3 or 4 / 5 in both dimensions. When encoding in reduced resolution the QP is reduced by -6, -4, and -2, for 1 / 2, 2 / 3, and 4 / 5 respectively to achieve similar bitrate as when encoding in source resolution.Evaluation of VCM technologies

[0014] The evaluation of VCM technologies is usually done utilizing the Bjontegaard- delta rate (BD-rate), which is a measure of difference between two rate-distortion curves and indicates a bitrate difference between the two curves at the same distortion level. In the VCM case, the distortion is measured in the task performance, i.e., how well the machine vision task performs on the decompressed video. The bitrate is taken from the size of the compressed bitstream. Generally, a negative value indicates a reduction in bitrate at the same task performance as the reference encoding. The performance can be evaluated with an object detection dataset, such as, for example, an SFU-HW dataset, or an object tracking dataset, such as, for example, the Tencent Video Dataset (TVD).SUMMARY

[0015] Certain challenges presently exist. For instance, deploying a machine vision task using decoded video that has the same resolution as the original source video resolution may result in redundant data that must be processed to complete the task, but the processing of this redundant data does not aid in the completion of the task and may slow down the completion of the task and / or may cause resources to be consumed unnecessarily.

[0016] Accordingly, in one aspect there is provided a method for encoding a video comprising a sequence of pictures, the sequence of pictures comprising a first keyframe and a first set of pictures. The method includes obtaining a first resampling metric based on content of the first keyframe. The method also includes determining, based on the first resampling metric, whether a first downscaling condition is satisfied. The method also includes, as a result of determining that the first downscaling condition is satisfied, downscaling the pictures in the first set of pictures using at least a first downscaling factor, wherein the first downscaling factor isgreater than 1, thereby producing a first set of downscaled pictures. The method also includes encoding each picture in the first set of downscaled pictures to produce a first set of encoded downscaled pictures.

[0017] In another aspect there is provided an apparatus for encoding a video comprising a sequence of pictures, the sequence of pictures comprising a first keyframe and a first set of pictures. The apparatus is configured to perform a method that includes obtaining a first resampling metric based on content of the first keyframe. The method also includes determining, based on the first resampling metric, whether a first downscaling condition is satisfied. The method also includes, as a result of determining that the first downscaling condition is satisfied, downscaling the pictures in the first set of pictures using at least a first downscaling factor, wherein the first downscaling factor is greater than 1 , thereby producing a first set of downscaled pictures. The method also includes encoding each picture in the first set of downscaled pictures to produce a first set of encoded downscaled pictures.

[0018] In another embodiment, the method for encoding the video includes obtaining a vertical resampling metric based on content of the first keyframe. The method also includes obtaining a horizontal resampling metric based on the content of the first keyframe. The method also includes using the vertical resampling metric and the horizontal resampling metric to determine whether a first downscaling condition is satisfied. The method also includes, as a result of determining that the first downscaling condition is satisfied, downscaling the pictures in the first set of pictures using at least one of a vertical downscaling factor or a horizontal downscaling factor, thereby producing a first set of downscaled pictures. The method also includes encoding each picture in the first set of downscaled pictures to produce a first set of encoded downscaled pictures. Using the vertical resampling metric and the horizontal resampling metric to determine whether the first downscaling condition is satisfied comprises: determining, based on the vertical resampling metric, whether a vertical downscaling condition is satisfied; and determining, based on the horizontal resampling metric, whether a horizontal downscaling condition is satisfied, and the first downscaling condition is satisfied if either the vertical downscaling condition is satisfied or the horizontal downscaling condition is satisfied.

[0019] In another aspect the apparatus is configured to perform a method that includes obtaining a vertical resampling metric based on content of the first keyframe. The method alsoincludes obtaining a horizontal resampling metric based on the content of the first keyframe. The method also includes using the vertical resampling metric and the horizontal resampling metric to determine whether a first downscaling condition is satisfied. The method also includes, as a result of determining that the first downscaling condition is satisfied, downscaling the pictures in the first set of pictures using at least one of a vertical downscaling factor or a horizontal downscaling factor, thereby producing a first set of downscaled pictures. The method also includes encoding each picture in the first set of downscaled pictures to produce a first set of encoded downscaled pictures. Using the vertical resampling metric and the horizontal resampling metric to determine whether the first downscaling condition is satisfied comprises: determining, based on the vertical resampling metric, whether a vertical downscaling condition is satisfied; and determining, based on the horizontal resampling metric, whether a horizontal downscaling condition is satisfied, and the first downscaling condition is satisfied if either the vertical downscaling condition is satisfied or the horizontal downscaling condition is satisfied.

[0020] In another aspect there is provided a computer program comprising instructions which when executed by processing circuitry of an apparatus causes the apparatus to perform any of the methods disclosed herein. In one embodiment, there is provided a carrier containing the computer program wherein the carrier is one of an electronic signal, an optical signal, a radio signal, and a computer readable storage medium. In one embodiment, there is provided an apparatus configured to perform any of the methods disclosed herein, wherein the apparatus is comprised in a video encoder.

[0021] An advantage of the embodiments disclosed herein is that they enable the encoder to apply downscaling when the machine vision task performance is not negatively affected (or only to a reasonable degree). This speeds up the encoder as the downscaled segment of the video contains less data (fewer pixels) that needs to be compressed. This will also benefit the decoder as there is potentially less data to be decoded and potentially less data to be analyzed by machine vision task, thereby not only reducing latency, but also reducing energy consumption.BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The accompanying drawings, which are incorporated herein and form part of the specification, illustrate various embodiments.

[0023] FIG. 1 illustrates a system according to an embodiment.

[0024] FIG. 2 is a block diagram of an encoder according to an embodiment.

[0025] FIG. 3 is a block diagram of a decoder according to an embodiment.

[0026] FIG. 4 is a flowchart illustrating a process according to an embodiment.

[0027] FIG. 5 is a flowchart illustrating a process according to an embodiment.

[0028] FIG. 6 is a block diagram of an apparatus according to an embodiment.DETAILED DESCRIPTION

[0029] FIG. 1 illustrates a system 100 according to an embodiment. System 100 includes an encoder 102 and a decoder 104. In some embodiments, encoder 102 is in communication with decoder 104 via a network 110 (e.g., the Internet or other network). Encoder 102 encodes a source video 101, produced by a camera 199, into a bitstream comprising an encoded video and may transmit the bitstream to decoder 104 via network 110. In some embodiments, encoder 102 is not in communication with decoder 104, and, in such an embodiment, rather than transmitting bitstream to decoder 104, the bitstream is stored in a data storage unit 190 and decoder 104 retrieves the bitstream from data storage unit 190.

[0030] Decoder 104 decodes the pictures included in the encoded video to produce video data for display and / or further image processing (e.g. a machine vision task, such as generating a 3D of an object-of-interest (OOI) in the pictures of the video). Accordingly, decoder 104 may be part of a device 103 having an image processor 105 and / or a display 106. The image processor 105 may perform machine vision tasks on the decoded pictures. One such machine vision task may be locating an object in the picture and creating a 3D model of the object. The device 103 may be a mobile device, a set-top device, a head-mounted display, or any other device.

[0031] Additionally, as shown in FIG. 1, an analyzer 190 is in communication with the encoder 102. Analyzer 190 functions to analyze the pictures of the video and, based on the analysis, provide control data to encoder 102 that will influence how encoder 102 will process the pictures. While in the embodiment shown in FIG. 1, analyzer is shown as being separate from encoder 102, in some embodiments, analyzer 190 is a component of encoder 102.

[0032] As described above, a challenge presently exists because deploying a machine vision task using decoded video that has the same resolution as the original source video resolution may result in redundant data that must be processed to complete the task, but theprocessing of this redundant data does not aid in the completion of the task and may slow down the completion of the task and / or may cause resources to be consumed unnecessarily.

[0033] Accordingly, this disclosure describes a system in which certain pictures of a video are analyzed, these pictures are referred to as “keyframes”, and, based on information produced from the analysis of a keyframe, a decision is made as to whether or not to code a group of pictures (a.k.a., “segment”) following or preceding the keyframe in encoding order in a lower resolution than the source video resolution. For instance, this disclosure describes an algorithm that is used to decide a downscaling factor to apply to a segment of the video based on an analysis of a keyframe. Accordingly, in one embodiment, analyzer 190 implements an object detection algorithm that, for each keyframe, detects objects (or rather, detects at least one object) in the keyframe, produces information regarding the objects, such as, for example, information indicting the number of objects detected, information indicating the size of the objects, such as, for example, information indicating the area of the smallest bounding box that can surround the object and / or information indicating one or more dimensions (height, width) of the smallest bounding box that can surround the object, and / or a list of the detected objects. The information produced by analyzer 190 is then used to select a resolution at which a segment will be encoded (e.g., a segment following the keyframe).

[0034] FIG. 2 illustrates functional components of encoder 102 according to some embodiments. It should be noted that encoders may be implemented differently so implementation other than this specific example can be used.

[0035] Encoder 102 includes an RPR function (RPRF) 201 for performing resampling (e.g., downscaling). RPR function 201 receives the source video and control data from analyzer 190. In one embodiment, the control data includes a resampling metric produced as a result of analyzing a keyframe, and RPRF 201 uses the resampling metric to select a resolution. In another embodiment, analyzer 190 selects the resolution based on the resampling metric and the control data provided to RPRF 201 includes information specifying the selected resolution. In either case, if the selected resolution is lower than the resolution of the source video, RPRF 201 re-scales a segment of the video, such as, for example, a segment following the keyframe in an encoding order, to the selected resolution.

[0036] As further shown in FIG. 2, encoder 102 employs a subtractor 241 to produce a residual block which is the difference in sample values between an input block provided by RPRF 201 and a prediction block (i.e., the output of a selector 251, which is either an inter prediction block output by an inter predictor 250 (a.k.a., motion compensator) or an intra prediction block output by an intra predictor 249). Then a forward transform 242 is performed on the residual block to produce a transformed block comprising transform coefficients. A quantization unit 243 quantizes the transform coefficients based on a quantization parameter (QP) value (e.g., a QP value obtained based on a picture QP value for the picture in which the input block is a part and a block specific QP offset value for the input block), thereby producing quantized transform coefficients which are then encoded into the bitstream by encoder 244 (e.g., an entropy encoder) and the bitstream with the encoded transform coefficients is output from encoder 102. Next, encoder 102 uses the quantized transform coefficients to produce a reconstructed block. This is done by first applying inverse quantization 245 and inverse transform 246 to the transform coefficients to produce a reconstructed residual block and using an adder 247 to add the prediction block to the reconstructed residual block, thereby producing the reconstructed block, which is stored in the reconstruction picture buffer (RPB) 266. Loop filtering by a loop filter (LF) stage 267 is applied and the final decoded picture is stored in a decoded picture buffer (DPB) 268, where it can then be used by the inter predictor 250 to produce an inter prediction block for the next picture to be processed. LF stage 267 may include three sub-stages: i) a deblocking filter, ii) a sample adaptive offset (SAO) filter, and iii) an Adaptive Loop Filter (ALF).

[0037] FIG. 3 illustrates functional components of decoder 104 according to some embodiments. It should be noted that decoder 104 may be implemented differently so implementations other than this specific example can be used. Decoder 104 includes a decoder module 361 (e.g., an entropy decoder) that decodes from the bitstream quantized transform coefficient values of a block. Decoder 104 also includes a reconstruction stage 398 in which the quantized transform coefficient values are subject to an inverse quantization process 362 and inverse transform process 363 to produce a residual block. This residual block is input to adder 364 that adds the residual block and a prediction block output from selector 390 to form a reconstructed block. Selector 390 either selects to output an inter prediction block or an intra prediction block. The reconstructed block is stored in a RPB 365. The inter prediction block isgenerated by the inter prediction module 350 and the intra prediction block is generated by the intra prediction module 369. Following the reconstruction stage 398, a loop filter stage 367 applies loop filtering to the reconstructed picture or block and the final decoded, reconstructed picture may be stored in a decoded picture buffer (DPB) 368 and output to image processor (IP) 105 (also may be output to display 106). Pictures are stored in the DPB for two primary reasons: 1) to wait for picture output and 2) to be used for reference when decoding future pictures.Additional Examples

[0038] All the following embodiments are based on a keyframe being analyzed. A keyframe can be set at regular intervals, e.g., every 16thor 32ndframe, or a separate algorithm could determine the keyframe placement. In the results included below, unless stated otherwise, the keyframe interval is set to 32 frames.

[0039] The outcome of the keyframe analysis is a decision whether to encode a segment of the video (e.g., a segment immediately following the keyframe in encoding order) at the original resolution or at a lower resolution. In some cases, there are several lower resolutions available and different thresholds indicate different downscaling factors.

[0040] As mentioned above, RPRF 201 can be used to resample, such as, for example, downscale, segments of the video. This has the advantage that decoder 104 knows exactly which filters are used for the resampling and it can produce the exact same reconstruction as encoder 102.

[0041] In one embodiment, analyzer 190 is configured to detect objects (or rather, detect at least one object) in each keyframe and use information regarding the detected object to produce a resampling metric, but it is also possible to use any other machine vision task to produce the resampling metric, or to train a machine learning (ML) model, such as, for example, a neural network, specifically for the purpose of determining whether to downscale a segment of the video.Example 1: Number of objects

[0042] In one embodiment, analyzer 190 is used to analyze a full resolution keyframe, i.e., a keyframe that has the original resolution of the source video, to detect objects (or rather,detect at least one object) in the keyframe. In this case, the resampling metric is the number of detected objects. If the resampling metric exceeds a threshold, then no downscaling is applied, i.e., the selected resolution is the full resolution.

[0043] In one embodiment, there are several thresholds and different downscaling factors, i.e., different resolutions, are selected based on the resampling metric, which in this case is the number of detected objects, meaning the number of detected at least one object.

[0044] In one example, two thresholds are employed - - a first threshold is set to 20 and the second threshold is set to 10 - and 25 objects are detected in the keyframe. Because the number of detected objects exceeds the first threshold, which exceeds the second threshold, no downscaling is applied, e.g., the downscaling factor is set to 1. If however, 15 objects were detected, then the second threshold is exceeded, i.e., the lower of the two thresholds, but not the first threshold and, as a result, the downscaling factor is set to 1.5. And if only 5 objects are detected, not exceeding any of the thresholds, then a downscaling with a downscaling factor of 2.0 is applied.

[0045] In one embodiment, the threshold(s) are set based on the size of the keyframe, e.g., larger keyframe sizes allow for more objects in a frame and still downscaling it.

[0046] Setting this type of threshold can provide a BD-rate benefit of up to -11.2% for SFU-HW and -8.0% for TVD when using a single threshold and setting it to 20, i.e., segments of the video sequences are downsampled by a factor of 1.25 if there are less than 20 objects in the keyframe.Example 2: Size of objects

[0047] Like in example 1, in example 2 analyzer 190 is used to analyze a full resolution keyframe to detect objects in the keyframe, meaning to detect at least one object in the keyframe. In one embodiment, the analyzer then calculates a resampling metric based on the sizes of the detected objects. In one embodiment the size of an object is the area of a bounding box that surrounds the object. For example, in one embodiment, the resampling metric is the mean size of the detected objects. In another embodiment, the resampling metric is the median size of the detected objects. In another embodiment, the resampling metric is a size where 25% of the detected objects have a smaller size than the resampling metric, i.e., the resampling metric is thefirst quartile object size (the 25th percentile). In yet another embodiment, the resampling metric is a size where 75% of the detected objects have a smaller size than the resampling metric. In each of the above cases, the resampling metric is compared to a threshold. If the resampling metric exceeds the threshold, downsampling is applied, otherwise not.

[0048] In one embodiment, there are several thresholds and different downscaling factors are applied based on the resampling metric.

[0049] In one example, two thresholds are employed - - a first threshold is set to 100,000 pixels and the second threshold is set to 50,000 pixels — and if the resampling metric (e.g., mean object size) is 125,000 pixels it is bigger than both thresholds, and the maximum downscaling factor, such as, for example, factor 2.0, is used. In another keyframe, the resampling metric is 75,000 pixels; this is smaller than the first threshold but larger than the second threshold, and hence a middle downscaling factor (e.g., factor 1.5) is selected. Finally, in yet another keyframe, the resampling metric is 25,000; because this is smaller than both thresholds, this means that no downscaling takes place.

[0050] When setting this type of threshold, BD-rate benefits are observed. For SFU-HW, setting a single threshold to 15,000 pixels and evaluating the mean object size provides a BD-rate result of -9.3%. For TVD, setting the threshold for the mean object size to 35,000 pixels provides a BD-rate result of -5.5%.

[0051] Alternatively, using the median object size for SFU-HW with a threshold of 25,000 pixels results in a BD-rate result of -9.9% and using the quartile object size for TVD with a threshold of 10,000 pixels gives -7.3% BD-rate.

[0052] In a variant of any of the previous embodiments the threshold(s) are relative to the size of the analyzed picture. A larger picture allows for larger objects to be recognized after downscaling.

[0053] In another embodiment, the analyzer calculates two resampling metrics - a vertical resampling metric and a horizontal resampling metric. The vertical resampling metric is based on the vertical sizes (heights) of the detected objects (or rather, the detected at least one object) and the horizontal resampling metric is based on the horizontal sizes (widths) of the detected objects, meaning the detected at least one object. For example, in one embodiment, thevertical and horizontal resampling metrics are the mean or median height of the detected objects and the mean or median width of the detected object, respectively. In another embodiment, the vertical resampling metric is a size where 25% of the detected objects have a smaller height than the vertical resampling metric, and the horizontal resampling metric is a size where 25% of the detected objects have a smaller width than the horizontal resampling metric. In yet another embodiment, the vertical resampling metric is a size where 75% of the detected objects have a smaller height than the vertical resampling metric, and the horizontal resampling metric is a size where 75% of the detected objects have a smaller width than the horizontal resampling metric. In each of the above cases, the vertical resampling metric is compared to a vertical threshold and the horizontal resampling metric is compared to a horizontal threshold, which may be equal to or different than the vertical threshold. If the vertical resampling metric exceeds the vertical threshold, then vertical downsampling is applied, otherwise not. Likewise, if the horizontal resampling metric exceeds the horizontal threshold, then horizontal downsampling is applied. As with the other embodiment discussed above, in some embodiments, there are several vertical thresholds and different vertical downscaling factors are applied based on the vertical resampling metric, and there are several horizontal thresholds and different horizontal downscaling factors are applied based on the horizontal resampling metric.Example 3: Compress + analyze

[0054] Like in examples 1 and 2, in example 3 analyzer 190 is used to analyze a full resolution keyframe to detect objects in the keyframe, meaning detect at least one object in the keyframe. The keyframe is then compressed using a lossy compression algorithm and then decompressed to produce a reconstructed keyframe, and the analyzer then analyzes the reconstructed keyframe to detect objects in the reconstructed keyframe. The compression factor used for compressing the keyframe might be the same quantization parameter (QP) as intended for use in the encoding of the full resolution input video or an increased QP to allow for faster processing and to simulate additional artifacts that might be introduced by the rescaling.

[0055] In one embodiment, for each object detected in the original keyframe, the analyzer determines whether the same object was also detected in the reconstructed keyframe. In this embodiment, the resampling metric is the total number of objects detected in the original keyframe that are also detected (or “found”) in the reconstructed keyframe. If the resamplingmetric exceeds a “found” threshold, then downscaling is applied. That is, if a sufficient number of the objects detected in the original keyframe have been “found” in the reconstructed keyframe, then downscaling is applied. As with examples 1 and 2, and as described below, several found thresholds may be employed, with each indicating a different downscaling factor.

[0056] In one embodiment, an loU threshold is used to determine whether an object detected in the original keyframe has been found in the reconstructed keyframe. For example, the loU of the object in the original keyframe and an object detected in the reconstructed keyframe is determined and compared to the loU threshold. If the loU exceeds the loU threshold, the object detected in the original keyframe is determined to have been found in the reconstructed keyframe.Example 4: Downscale + analyze

[0057] In example 4, analyzer 190 is used to analyze a full resolution keyframe to detect objects in the keyframe, meaning detect at least one object in the keyframe. The keyframe is then downscaled using a downscaling factor to generate a downscaled keyframe. The analyzer then analyzes the downscaled keyframe to detect objects in the downscaled keyframe.

[0058] In one embodiment, for each object detected in the original keyframe, the analyzer determines whether the same object was also detected in the downscaled keyframe. In this embodiment, the resampling metric is the total number of objects detected in the original keyframe that are also found in the downscaled keyframe. If the resampling metric exceeds the found threshold, then downscaling is applied. That is, if a sufficient number of the objects detected in the original keyframe have been “found” in the downscaled keyframe, then downscaling is applied.

[0059] Like in example 3, in one embodiment, the loU threshold is used to determine whether an object detected in the original keyframe has been found in the reconstructed keyframe.

[0060] For SFU-HW, setting the found threshold to 0.5 and the loU threshold to 0.75 and utilizing a keyframe interval of 16 pictures gives a BD-rate result of -9.0%. For TVD, setting both the found threshold and the loU threshold to 0.75 gives a BD-rate result of -7.7%.Example 5: Downscale + compress + analyze

[0061] In example 5, analyzer 190 is used to analyze a full resolution keyframe to detect objects in the keyframe, meaning detect at least one object in the keyframe. The keyframe is then downscaled using a downscaling factor to generate a downscaled keyframe. The downscaled keyframe is then compressed using a lossy compression algorithm and then decompressed to produce a reconstructed downscaled keyframe, and the analyzer then analyzes the reconstructed downscaled keyframe to detect objects in the reconstructed downscaled keyframe. The compression factor used for compressing the downscaled keyframe might be the same QP as intended for use in the encoding of the full resolution input video or an increased QP to allow for faster processing and to simulate additional artifacts that might be introduced by the rescaling.

[0062] In one embodiment, for each object detected in the original keyframe, the analyzer determines whether the same object was also detected in the reconstructed downscaled keyframe. In this embodiment, the resampling metric is the total number of objects detected in the original keyframe that are also found in the reconstructed downscaled keyframe. If the resampling metric exceeds the found threshold, then downscaling is applied. That is, if a sufficient number of the objects detected in the original keyframe have been “found” in the reconstructed downscaled keyframe, then downscaling is applied.

[0063] Like in example 3, in one embodiment, the loU threshold is used to determine whether an object detected in the original keyframe has been found in the reconstructed keyframe.

[0064] For SFU-HW, setting the found threshold to 0.75 and the loU threshold to 0.5 gives a BD-rate result of -4.8%. For TVD, setting both found threshold and loU threshold to 0.75 gives a BD-rate result of -2.8%.

[0065] With respect to examples 3, 4, and 5, in one embodiment two found thresholds are employed, such as, for example, a first found threshold set equal to 0.75 and a second found threshold equal to 0.5.

[0066] Assume that analysis of the original keyframe results in 10 objects being detected, but when analyzing the processed keyframe (for instance, compressed and decompressed) only 8 of the 10 objects were found in the processed keyframe. Then because 8 / 10 = 0.8 is larger than both thresholds, a first downscaling factor of 2.0 is selected. If instead the analysis only finds 6 objects in the processed keyframe, then since 6 / 10 = 0.6 is smaller than one threshold but largerthan the other threshold, a middle downscaling factor of 1.5 is selected. Finally, if only 4 objects are found in the processed keyframe, no downscaling is performed because 4 / 10 = 0.4 is smaller than both thresholds.Example 6: Adaptive Thresholding

[0067] In one embodiment, the above described thresholds are adaptive. For example, the above described thresholds are set based on a sequence or part of a sequence. In one example, a first sequence, sequence 1, uses a first threshold T1 for making the decision for using any resampling and a second sequence, sequence2, uses a second threshold T2 for making the decision for using any resampling, where T1 and T2 are not the same.

[0068] In another example a first part of a sequencel uses a first threshold T1 for making the decision for using any resampling and a second part of the same sequencel uses a second threshold T2 for making the decision for using any resampling, where T1 and T2 are not the same.

[0069] In these embodiments, the threshold may be determined by analyzing the sequence or a part of the sequence, for instance based on a bitrate vs. detection error trade-off, based on the number of objects in the keyframe, absolute or relative sizes of the objects in the keyframe, size ratio of two objects, e.g., the smallest object and the largest object, etc.

[0070] In one example, for the trade-off, the system prefers lower bandwidth (for example due to bandwidth limitations). Then the encoder adapts the threshold based on the detection error budget.

[0071] As an example for determining the threshold based on the number of objects in a keyframe, one can have two threshold schemes in the downscale+compress+analyze category, one where the found threshold is set to 0.75 and a second scheme where the found threshold is set to 0.5. Based on the number of objects in the original keyframe the analyzer decides which of the schemes to apply, e.g., if the keyframe contains less than 15 objects the first scheme is applied and if there are at least 15 objects in the keyframe the second scheme is applied.

[0072] In a variant of this embodiment there is a decision tree for determining the threshold value(s) based on the properties of the sequence, such as the picture resolution, number of the objects in the keyframe, etc.

[0073] In another variant of this embodiment there is at least a first and a second threshold and the first threshold is fixed (not changing for different sequences) and the second threshold is determined adaptively to the sequence or part of the sequence.Example 7 - Upscaling at the Receiver

[0074] In some embodiments, information indicating the vertical and / or horizontal downscaling factors are provided to device 103 so that decoder 104 or image processor 105 can upscale the downscaled pictures and then perform the object detection task using the upscaled pictures.Summary of Various EmbodimentsAl. A method 400 (see FIG. 4) for encoding a video comprising a sequence of pictures, the sequence of pictures comprising a first keyframe and a first set of pictures, the method comprising: obtaining (s402) a first resampling metric based on content of the first keyframe; determining (s404), based on the first resampling metric, whether a first downscaling condition is satisfied; as a result of determining that the first downscaling condition is satisfied, downscaling (s406) the pictures in the first set of pictures using at least a first downscaling factor, wherein the first downscaling factor is greater than 1, thereby producing a first set of downscaled pictures; and encoding (s408) each picture in the first set of downscaled pictures to produce a first set of encoded downscaled pictures.A2. The method of embodiment Al, wherein obtaining the first resampling metric comprises inputting the first keyframe and / or information derived from the first keyframe into a trained machine learning, ML, model, wherein the ML model outputs the first resampling metric.A3. The method of embodiment Al, wherein obtaining the first resampling metric comprises detecting objects in the first keyframe.A4. The method of embodiment A3, wherein obtaining the first resampling metric further comprises, for each objected detected in the first keyframe, determining a size of the object, and the first resampling metric is a value that is a function of the determined sizes.A5. The method of embodiment A4, wherein determining the size of the object comprises determining: a vertical size of the object (e.g., the height of a bounding box that surrounds the object), a horizontal size of the object (e.g., the width of the bounding box that surrounds the object), and / or an area of the object (e.g., an area of the bounding box that surrounds the object).A6. The method of embodiment A4 or A5, wherein the first resampling metric is: the mean of the determined sizes (e.g., mean of the heights, mean of the widths, or mean of the areas), the median of the determined sizes, the first quartile of the determined sizes, or the third quartile of the determined sizes.A7. The method of embodiment A4 or A5 or A6, wherein the first downscaling condition is satisfied if the first resampling metric exceeds a first threshold.A8. The method of embodiment A7, wherein the downscaling condition is also satisfied if the first resampling metric exceeds a second threshold that is lower than the first threshold, the first resampling metric exceeds the second threshold, downscaling the pictures in the first set of pictures using a downscaling factor comprises selecting a first downscaling factor if the resampling metric exceeds the first threshold, otherwise selecting a second downscaling factor, andthe second downscaling factor is lower than the first downscaling factor.A9. The method of any one of embodiments Al -A3, wherein obtaining the first resampling metric further comprises determining the total number of objects that have been detected in the first keyframe, and the first resampling metric is a value equal to or a function of the total number of objects that have been detected in the first keyframe.A10. The method of embodiment A9, wherein obtaining the resampling metric further comprises: processing the first keyframe to produce a first processed keyframe; detecting objects in the first processed keyframe; and for each objected detected in the first keyframe, determining whether the same object has been detected in the first processed keyframe; and determining the total number of objects detected in the first keyframe that were also detected in the first processed keyframe, wherein the resampling metric is a function of i) the total number of objects detected in the first keyframe and ii) the total number of objects detected in the first keyframe that were also detected in the first processed keyframeAl 1. The method of embodiment A10, wherein processing the first keyframe comprises: downscaling the first keyframe to produce a first downscaled keyframe, wherein the first processed keyframe is the first downscaled keyframe; or compressing the first keyframe to produce a first compressed keyframe and decompressing the first compressed keyframe to produce a first reconstructed keyframe, wherein the first processed keyframe is the first reconstructed keyframe; or downscaling the first keyframe to produce a first downscaled keyframe, compressing the first downscaled keyframe to produce a first compressed downscaled keyframe, and decompressing the first compressed downscaled keyframe to produce a first reconstructed downscaled keyframe, wherein the first processed keyframe is the first reconstructed downscaled keyframe.Al 2. The method of any one of embodiments A9-A11, wherein the downscaling condition is satisfied if the first resampling metric is lower than a first threshold (e.g., 10).Al 3. The method of embodiment Al 2, wherein the downscaling condition is also satisfied if the first resampling metric is lower than a second threshold (e.g., 20), wherein the second threshold is greater than the first threshold, the first resampling metric is lower than the second threshold, downscaling the pictures in the first set of pictures using a downscaling factor comprises selecting a first downscaling factor (e.g., 2x) if the resampling metric is lower than the first threshold (e.g., 10), otherwise selecting a second downscaling factor (e.g., 1.5x), and the second downscaling factor is lower than the first downscaling factor.Al 4. The method of embodiment A7, A8, Al 2, or Al 3, wherein the first threshold is determined based on any combination of: an analysis of the first set of pictures or the first keyframe, a size ratio of two objects in the first keyframe, the number of objects in the first keyframe, a picture resolution, or an object detection error margin.Al 5. The method of any one of claims Al -Al 4, wherein the first resampling metric is a vertical resampling metric and the first downscaling factor is a vertical downscaling factor, or the first resampling metric is a horizontal resampling metric and the first downscaling factor is a horizontal downscaling factor.Bl. A method 500 (see FIG. 5) for encoding a video comprising a sequence of pictures, the sequence of pictures comprising a first keyframe and a first set of pictures, the method comprising:obtaining (s502) a vertical resampling metric based on content of the first keyframe; obtaining (s504) a horizontal resampling metric based on the content of the first keyframe; using (s506) the vertical resampling metric and the horizontal resampling metric to determine whether a downscaling condition is satisfied; as a result of determining that the downscaling condition is satisfied, downscaling (s508) the pictures in the first set of pictures using at least a vertical downscaling factor or a horizontal downscaling factor, thereby producing a first set of downscaled pictures; and encoding (s510) each picture in the first set of downscaled pictures to produce a first set of encoded downscaled pictures, wherein using the vertical resampling metric and the horizontal resampling metric to determine whether a downscaling condition is satisfied comprises: determining, based on the vertical resampling metric, whether a vertical downscaling condition is satisfied; and determining, based on the horizontal resampling metric, whether a horizontal downscaling condition is satisfied, and the downscaling condition is satisfied if either the vertical downscaling condition is satisfied or the horizontal downscaling condition is satisfied.B2. The method of embodiment Bl, wherein downscaling the pictures in the first set of pictures using at least a vertical downscaling factor or a horizontal downscaling factor comprises: downscaling the pictures using the vertical downscaling factor if the vertical downscaling condition is satisfied; and downscaling the pictures using the horizontal downscaling factor if the horizontal downscaling condition is satisfied.Cl. A computer program (643) comprising instructions (644) which when executed by processing circuitry (602) of an apparatus causes the apparatus to perform the method of any one of claims Al -Al 3.C2. A carrier containing the computer program of embodiment Cl, wherein the carrier is one of an electronic signal, an optical signal, a radio signal, and a computer readable storage medium (642).DI. An apparatus 600 (see FIG. 6) for encoding a video comprising a sequence of pictures, the sequence of pictures comprising a first keyframe and a first set of pictures, wherein the apparatus is configured to perform a method comprising: obtaining (s402) a first resampling metric based on content of the first keyframe; determining (s404), based on the first resampling metric, whether a first downscaling condition is satisfied; as a result of determining that the first downscaling condition is satisfied, downscaling (s406) the pictures in the first set of pictures using at least a first downscaling factor, wherein the first downscaling factor is greater than 1, thereby producing a first set of downscaled pictures; and encoding (s408) each picture in the first set of downscaled pictures to produce a first set of encoded downscaled pictures.D2. The apparatus of embodiment DI, wherein obtaining the first resampling metric comprises inputting the first keyframe and / or information derived from the first keyframe into a trained machine learning, ML, model, wherein the ML model outputs the first resampling metric.D3. The apparatus of embodiment DI, wherein obtaining the first resampling metric comprises detecting objects in the first keyframe.D4. The apparatus of embodiment D3, wherein obtaining the first resampling metric further comprises, for each objected detected in the first keyframe, determining a size of the object, and the first resampling metric is a value that is a function of the determined sizes.D5. The apparatus of embodiment D4, wherein determining the size of the object comprises determining: a vertical size of the object (e.g., the height of a bounding box that surrounds the object), a horizontal size of the object (e.g., the width of the bounding box that surrounds the object), and / or an area of the object (e.g., an area of the bounding box that surrounds the object).D6. The apparatus of embodiment D4 or D5, wherein the first resampling metric is: the mean of the determined sizes (e.g., mean of the heights, mean of the widths, or mean of the areas), the median of the determined sizes, the first quartile of the determined sizes, or the third quartile of the determined sizes.D7. The apparatus of embodiment D4 or D5 or D6, wherein the first downscaling condition is satisfied if the first resampling metric exceeds a first threshold.D8. The apparatus of embodiment D7, wherein the downscaling condition is also satisfied if the first resampling metric exceeds a second threshold that is lower than the first threshold, the first resampling metric exceeds the second threshold, downscaling the pictures in the first set of pictures using a downscaling factor comprises selecting a first downscaling factor if the resampling metric exceeds the first threshold, otherwise selecting a second downscaling factor, and the second downscaling factor is lower than the first downscaling factor.D9. The apparatus of any one of embodiments D1-D3, wherein obtaining the first resampling metric further comprises determining the total number of objects that have been detected in the first keyframe, andthe first resampling metric is a value equal to or a function of the total number of objects that have been detected in the first keyframe.DIO. The apparatus of embodiment D9, wherein obtaining the resampling metric further comprises: processing the first keyframe to produce a first processed keyframe; detecting objects in the first processed keyframe; and for each objected detected in the first keyframe, determining whether the same object has been detected in the first processed keyframe; and determining the total number of objects detected in the first keyframe that were also detected in the first processed keyframe, wherein the resampling metric is a function of i) the total number of objects detected in the first keyframe and ii) the total number of objects detected in the first keyframe that were also detected in the first processed keyframeDl l. The apparatus of embodiment DIO, wherein processing the first keyframe comprises: downscaling the first keyframe to produce a first downscaled keyframe, wherein the first processed keyframe is the first downscaled keyframe; or compressing the first keyframe to produce a first compressed keyframe and decompressing the first compressed keyframe to produce a first reconstructed keyframe, wherein the first processed keyframe is the first reconstructed keyframe; or downscaling the first keyframe to produce a first downscaled keyframe, compressing the first downscaled keyframe to produce a first compressed downscaled keyframe, and decompressing the first compressed downscaled keyframe to produce a first reconstructed downscaled keyframe, wherein the first processed keyframe is the first reconstructed downscaled keyframe.DI 2. The apparatus of any one of embodiments D9-D11, wherein the downscaling condition is satisfied if the first resampling metric is lower than a first threshold (e.g., 10).DI 3. The apparatus of embodiment DI 2, wherein the downscaling condition is also satisfied if the first resampling metric is lower than a second threshold (e.g., 20), wherein the second threshold is greater than the first threshold, the first resampling metric is lower than the second threshold, downscaling the pictures in the first set of pictures using a downscaling factor comprises selecting a first downscaling factor (e.g., 2x) if the resampling metric is lower than the first threshold (e.g., 10), otherwise selecting a second downscaling factor (e.g., 1.5x), and the second downscaling factor is lower than the first downscaling factor.D14. The apparatus of embodiment D7, D8, D12, or D13, wherein the first threshold is determined based on any combination of: an analysis of the first set of pictures or the first keyframe, a size ratio of two objects in the first keyframe, the number of objects in the first keyframe, a picture resolution, or an object detection error margin.DI 5. The apparatus of any one of claims DI -DI 4, wherein the first resampling metric is a vertical resampling metric and the first downscaling factor is a vertical downscaling factor, or the first resampling metric is a horizontal resampling metric and the first downscaling factor is a horizontal downscaling factor.El. An apparatus 600 (see FIG. 6) for encoding a video comprising a sequence of pictures, the sequence of pictures comprising a first keyframe and a first set of pictures, wherein the apparatus is configured to perform a method comprising: obtaining (s502) a vertical resampling metric based on content of the first keyframe; obtaining (s504) a horizontal resampling metric based on the content of the first keyframe;using (s506) the vertical resampling metric and the horizontal resampling metric to determine whether a downscaling condition is satisfied; as a result of determining that the downscaling condition is satisfied, downscaling (s508) the pictures in the first set of pictures using at least a vertical downscaling factor or a horizontal downscaling factor, thereby producing a first set of downscaled pictures; and encoding (s510) each picture in the first set of downscaled pictures to produce a first set of encoded downscaled pictures, wherein using the vertical resampling metric and the horizontal resampling metric to determine whether a downscaling condition is satisfied comprises: determining, based on the vertical resampling metric, whether a vertical downscaling condition is satisfied; and determining, based on the horizontal resampling metric, whether a horizontal downscaling condition is satisfied, and the downscaling condition is satisfied if either the vertical downscaling condition is satisfied or the horizontal downscaling condition is satisfied.E2. The apparatus of embodiment El, wherein downscaling the pictures in the first set of pictures using at least a vertical downscaling factor or a horizontal downscaling factor comprises: downscaling the pictures using the vertical downscaling factor if the vertical downscaling condition is satisfied; and downscaling the pictures using the horizontal downscaling factor if the horizontal downscaling condition is satisfied.

[0075] FIG. 4 illustrates a method 400 for encoding a video comprising a sequence of pictures, the sequence of pictures comprising a first keyframe and a first set of pictures.Furthermore, the method 400 is comprising obtaining (s402) a first resampling metric based on content of the first keyframe, and determining (s404), based on the first resampling metric, whether a first downscaling condition is satisfied. As a result of determining that the first downscaling condition is satisfied, the pictures in the first set of pictures are downscaled (s406) using at least a first downscaling factor, wherein the first downscaling factor is greater than 1,thereby producing a first set of downscaled pictures, and encoding (s408) each picture in the first set of downscaled pictures to produce a first set of encoded downscaled pictures.

[0076] FIG. 5 illustrates a method 500 for encoding a video comprising a sequence of pictures, the sequence of pictures comprising a first keyframe and a first set of pictures. Furthermore, the method 500 is comprising obtaining (s502) a vertical resampling metric based on content of the first keyframe, obtaining (s504) a horizontal resampling metric based on the content of the first keyframe, and using (s506) the vertical resampling metric and the horizontal resampling metric to determine whether a downscaling condition is satisfied. As a result of determining that the downscaling condition is satisfied, the pictures in the first set of pictures are downscaled (s508) using at least a vertical downscaling factor or a horizontal downscaling factor, thereby producing a first set of downscaled pictures. Then, each picture in the first set of downscaled pictures is encoded (s510) to produce a first set of encoded downscaled pictures, wherein using the vertical resampling metric and the horizontal resampling metric to determine whether a downscaling condition is satisfied comprises determining, based on the vertical resampling metric, whether a vertical downscaling condition is satisfied, and determining, based on the horizontal resampling metric, whether a horizontal downscaling condition is satisfied. The downscaling condition is satisfied if either the vertical downscaling condition is satisfied or the horizontal downscaling condition is satisfied.

[0077] FIG. 6 is a block diagram of an apparatus 600 for implementing encoder 102 or decoder 104 according to some embodiments. When apparatus 600 implements encoder 102, apparatus 600 may be referred to as an encoder apparatus, when apparatus 600 implements decoder 104, apparatus 600 may be referred to as a decoder apparatus. As shown in FIG. 6, apparatus 600 may comprise: processing circuitry (PC) 602, which may include one or more processors (P) 655 (e.g., one or more general purpose microprocessors and / or one or more other processors, such as an application specific integrated circuit (ASIC), field-programmable gate arrays (FPGAs), and the like), which processors may be co-located in a single housing or in a single data center or may be geographically distributed (i.e., encoder apparatus 600 may be a distributed computing apparatus); at least one network interface 648 (e.g., a physical interface or air interface) comprising a transmitter (Tx) 645 and a receiver (Rx) 647 for enabling apparatus 600 to transmit data to and receive data from other nodes connected to a network 110 (e.g., an Internet Protocol (IP) network) to which network interface 648 is connected (physically orwirelessly) (e.g., network interface 648 may be coupled to an antenna arrangement comprising one or more antennas for enabling encoder apparatus 600 to wirelessly transmit / receive data); and a storage unit (a.k.a., “data storage system”) 608, which may include one or more nonvolatile storage devices and / or one or more volatile storage devices. In embodiments where PC 602 includes a programmable processor, a computer readable storage medium (CRSM) 642 may be provided. CRSM 642 may store a computer program (CP) 643 comprising computer readable instructions (CRI) 644. CRSM 642 may be a non-transitory computer readable medium, such as, magnetic media (e.g., a hard disk), optical media, memory devices (e.g., random access memory, flash memory), and the like. In some embodiments, the CRI 644 of computer program 643 is configured such that when executed by PC 602, the CRI causes encoder apparatus 600 to perform steps described herein (e.g., steps described herein with reference to the flow charts). In other embodiments, encoder apparatus 600 may be configured to perform steps described herein without the need for code. That is, for example, PC 602 may consist merely of one or more ASICs. Hence, the features of the embodiments described herein may be implemented in hardware and / or software.

[0078] While various embodiments are described herein, it should be understood that they have been presented by way of example only, and not limitation. Thus, the breadth and scope of this disclosure should not be limited by any of the above-described exemplary embodiments. Moreover, any combination of the above-described elements in all possible variations thereof is encompassed by the disclosure unless otherwise indicated herein or otherwise clearly contradicted by context.

[0079] As used herein transmitting a message “to” or “toward” an intended recipient encompasses transmitting the message directly to the intended recipient or transmitting the message indirectly to the intended recipient (i.e., one or more other nodes are used to relay the message from the source node to the intended recipient). Likewise, as used herein receiving a message “from” a sender encompasses receiving the message directly from the sender or indirectly from the sender (i.e., one or more nodes are used to relay the message from the sender to the receiving node). Further, as used herein “a” means “at least one” or “one or more.”

[0080] Additionally, while the processes described above and illustrated in the drawings are shown as a sequence of steps, this was done solely for the sake of illustration. Accordingly, itis contemplated that some steps may be added, some steps may be omitted, the order of the steps may be re-arranged, and some steps may be performed in parallel.

Claims

CLAIMS1. A method (400) for encoding a video comprising a sequence of pictures, the sequence of pictures comprising a first keyframe and a first set of pictures, the method comprising: obtaining (s402) a first resampling metric based on content of the first keyframe; determining (s404), based on the first resampling metric, whether a first downscaling condition is satisfied; as a result of determining that the first downscaling condition is satisfied, downscaling (s406) the pictures in the first set of pictures using at least a first downscaling factor, wherein the first downscaling factor is greater than 1, thereby producing a first set of downscaled pictures; and encoding (s408) each picture in the first set of downscaled pictures to produce a first set of encoded downscaled pictures.

2. The method according to claim 1, wherein obtaining the first resampling metric comprises inputting the first keyframe and / or information derived from the first keyframe into a trained machine learning, ML, model, wherein the ML model outputs the first resampling metric.

3. The method according to claim 1 or 2, wherein obtaining the first resampling metric comprises detecting at least one object in the first keyframe.

4. The method according to claim 3, wherein obtaining the first resampling metric further comprises, for each of the at least one object detected in the first keyframe, determining a size of the at least one object, and the first resampling metric is a value that is a function of the determined size of the at least one object.

5. The method according to claim 4, wherein determining the size of the at least one object comprises determining:a vertical size of the at least one object, a horizontal size of the at least one object, an area of the at least one object, and / or an area of the smallest bounding box that can surround the at least one object.

6. The method according to claim 5, wherein the vertical size of the at least one object is the height of a bounding box that surrounds the at least one object.

7. The method according to claim 5 or 6, wherein the horizontal size of the at least one object is the width of the bounding box that surrounds the at least one object.

8. The method according to any one of claims 5 to 7, wherein the area of the at least one object is an area of the bounding box that surrounds the at least one object.

9. The method according to any one of claims 4 to 8, wherein the first resampling metric is specified as: the mean of the determined sizes, the median of the determined sizes, the first quartile of the determined sizes, or the third quartile of the determined sizes.

10. The method according to claim 9, wherein the mean of the determined sizes is specified as at least one of: the mean of the heights, the mean of the widths, or the mean of the areas.

11. The method according to any one of claims 1 to 10, wherein the first downscaling condition is satisfied if the first resampling metric exceeds a first threshold.

12. The method according to claim 11, wherein the first downscaling condition is also satisfied if the first resampling metric exceeds a second threshold that is lower than the first threshold, downscaling the pictures in the first set of pictures using a downscaling factor comprises selecting a first downscaling factor if the first resampling metric exceeds the first threshold, otherwise selecting a second downscaling factor, and the second downscaling factor is lower than the first downscaling factor.

13. The method according to any one of claims 1-3, wherein obtaining the first resampling metric further comprises determining the number of objects that have been detected in the first keyframe, and the first resampling metric is a value equal to or a function of the number of objects that have been detected in the first keyframe.

14. The method according to claim 13, wherein obtaining the first resampling metric further comprises: processing the first keyframe to produce a first processed keyframe; detecting the at least one object in the first processed keyframe; and for the at least one object detected in the first keyframe, determining whether the same at least one object has been detected in the first processed keyframe; and determining the number of objects detected in the first keyframe that were also detected in the first processed keyframe, wherein the first resampling metric is a function of i) the number of objects detected in the first keyframe and ii) the number of objects detected in the first keyframe that were also detected in the first processed keyframe.

15. The method according to claim 14, wherein processing the first keyframe comprises: downscaling the first keyframe to produce a first downscaled keyframe, wherein the first processed keyframe is the first downscaled keyframe; orcompressing the first keyframe to produce a first compressed keyframe and decompressing the first compressed keyframe to produce a first reconstructed keyframe, wherein the first processed keyframe is the first reconstructed keyframe; or downscaling the first keyframe to produce a first downscaled keyframe, compressing the first downscaled keyframe to produce a first compressed downscaled keyframe, and decompressing the first compressed downscaled keyframe to produce a first reconstructed downscaled keyframe, wherein the first processed keyframe is the first reconstructed downscaled keyframe.

16. The method according to any one of claims 13-15, wherein the first downscaling condition is satisfied if the first resampling metric is lower than a first threshold.

17. The method according to claim 16, wherein the first downscaling condition is also satisfied if the first resampling metric is lower than a second threshold, wherein the second threshold is greater than the first threshold, downscaling the pictures in the first set of pictures using a downscaling factor comprises selecting a first downscaling factor if the first resampling metric is lower than the first threshold, otherwise selecting a second downscaling factor, and the second downscaling factor is lower than the first downscaling factor.

18. The method according to any one of claims 11, 12, 16 and 17, wherein the first threshold is determined based on any combination of: an analysis of the first set of pictures or the first keyframe, a size ratio of two objects in the first keyframe, the number of objects in the first keyframe, a picture resolution, or an object detection error margin.

19. The method according to any one of claims 1-18, whereinthe first resampling metric is a vertical resampling metric and the first downscaling factor is a vertical downscaling factor, or the first resampling metric is a horizontal resampling metric and the first downscaling factor is a horizontal downscaling factor.

20. A method (500) for encoding a video comprising a sequence of pictures, the sequence of pictures comprising a first keyframe and a first set of pictures, the method comprising: obtaining (s502) a vertical resampling metric based on content of the first keyframe; obtaining (s504) a horizontal resampling metric based on the content of the first keyframe; using (s506) the vertical resampling metric and the horizontal resampling metric to determine whether a first downscaling condition is satisfied; as a result of determining that the first downscaling condition is satisfied, downscaling (s508) the pictures in the first set of pictures using at least one of a vertical downscaling factor or a horizontal downscaling factor, thereby producing a first set of downscaled pictures; and encoding (s510) each picture in the first set of downscaled pictures to produce a first set of encoded downscaled pictures, wherein using the vertical resampling metric and the horizontal resampling metric to determine whether the first downscaling condition is satisfied comprises: determining, based on the vertical resampling metric, whether a vertical downscaling condition is satisfied; determining, based on the horizontal resampling metric, whether a horizontal downscaling condition is satisfied; and the first downscaling condition is satisfied if either the vertical downscaling condition is satisfied or the horizontal downscaling condition is satisfied.

21. The method according to claim 20, wherein downscaling the pictures in the first set of pictures using at least one of a vertical downscaling factor or a horizontal downscaling factor comprises at least one of: downscaling the pictures using the vertical downscaling factor if the vertical downscaling condition is satisfied, anddownscaling the pictures using the horizontal downscaling factor if the horizontal downscaling condition is satisfied.

22. A computer program (643) comprising instructions (644) which when executed by processing circuitry (602) of an apparatus causes the apparatus to perform the method of at least one of claims 1-21.

23. A carrier containing the computer program according to claim 22, wherein the carrier is one of an electronic signal, an optical signal, a radio signal, and a computer readable storage medium (642).

24. An apparatus (600) for encoding a video comprising a sequence of pictures, the sequence of pictures comprising a first keyframe and a first set of pictures, wherein the apparatus is configured to perform a method comprising: obtaining (s402) a first resampling metric based on content of the first keyframe; determining (s404), based on the first resampling metric, whether a first downscaling condition is satisfied; as a result of determining that the first downscaling condition is satisfied, downscaling (s406) the pictures in the first set of pictures using at least a first downscaling factor, wherein the first downscaling factor is greater than 1, thereby producing a first set of downscaled pictures; and encoding (s408) each picture in the first set of downscaled pictures to produce a first set of encoded downscaled pictures.

25. The apparatus according to claim 24, wherein obtaining the first resampling metric comprises inputting the first keyframe and / or information derived from the first keyframe into a trained machine learning, ML, model, wherein the ML model outputs the first resampling metric.

26. The apparatus according to claim 24 or 25, whereinobtaining the first resampling metric comprises detecting at least one object in the first keyframe.

27. The apparatus according to claim 26, wherein obtaining the first resampling metric further comprises, for at least one object detected in the first keyframe, determining a size of the at least one object, and the first resampling metric is a value that is a function of the determined size of the at least one object.

28. The apparatus according to claim 27, wherein determining the size of the at least one object comprises determining: a vertical size of the at least one object, a horizontal size of the at least one object, an area of the at least one object, and / or an area of the smallest bounding box that can surround the at least one object.

29. The apparatus according to claim 28, wherein the vertical size of the at least one object is the height of a bounding box that surrounds the at least one object.

30. The apparatus according to claim 28 or 29, wherein the horizontal size of the at least one object is the width of the bounding box that surrounds the at least one object.

31. The apparatus according to any one of claims 28 to 30, wherein the area of the at least one object is an area of the bounding box that surrounds the at least one object.

32. The apparatus according to any one of claims 27 to 31, wherein the first resampling metric is specified as: the mean of the determined sizes, the median of the determined sizes, the first quartile of the determined sizes, or the third quartile of the determined sizes.

33. The apparatus according to claim 32, wherein the mean of the determined sizes is specified as at least one of: the mean of the heights, the mean of the widths, or the mean of the areas.

34. The apparatus according to any one of claims 28 to 33, wherein the first downscaling condition is satisfied if the first resampling metric exceeds a first threshold.

35. The apparatus according to claim 34, wherein the first downscaling condition is also satisfied if the first resampling metric exceeds a second threshold that is lower than the first threshold, downscaling the pictures in the first set of pictures using a downscaling factor comprises selecting a first downscaling factor if the first resampling metric exceeds the first threshold, otherwise selecting a second downscaling factor, and the second downscaling factor is lower than the first downscaling factor.

36. The apparatus according to any one of claims 28 to 33, wherein obtaining the first resampling metric further comprises determining the number of objects that have been detected in the first keyframe, and the first resampling metric is a value equal to or a function of the number of objects that have been detected in the first keyframe.

37. The apparatus according to claim 36, wherein obtaining the first resampling metric further comprises: processing the first keyframe to produce a first processed keyframe; detecting the at least one object in the first processed keyframe; and for at least one object detected in the first keyframe, determining whether the same object has been detected in the first processed keyframe; anddetermining the number of objects detected in the first keyframe that were also detected in the first processed keyframe, wherein the first resampling metric is a function of i) the number of objects detected in the first keyframe and ii) the number of objects detected in the first keyframe that were also detected in the first processed keyframe.

38. The apparatus according to claim 37, wherein processing the first keyframe comprises: downscaling the first keyframe to produce a first downscaled keyframe, wherein the first processed keyframe is the first downscaled keyframe; or compressing the first keyframe to produce a first compressed keyframe and decompressing the first compressed keyframe to produce a first reconstructed keyframe, wherein the first processed keyframe is the first reconstructed keyframe; or downscaling the first keyframe to produce a first downscaled keyframe, compressing the first downscaled keyframe to produce a first compressed downscaled keyframe, and decompressing the first compressed downscaled keyframe to produce a first reconstructed downscaled keyframe, wherein the first processed keyframe is the first reconstructed downscaled keyframe.

39. The apparatus according to any one of claims 36 to 38, wherein the first downscaling condition is satisfied if the first resampling metric is lower than a first threshold.

40. The apparatus according to claim 39, wherein the first downscaling condition is also satisfied if the first resampling metric is lower than a second threshold, wherein the second threshold is greater than the first threshold, downscaling the pictures in the first set of pictures using a downscaling factor comprises selecting a first downscaling factor if the first resampling metric is lower than the first threshold, otherwise selecting a second downscaling factor, and the second downscaling factor is lower than the first downscaling factor.

41. The apparatus according to any one of claims 34, 35, 39 and 40, wherein the first threshold is determined based on any combination of: an analysis of the first set of pictures or the first keyframe, a size ratio of two objects in the first keyframe, the number of objects in the first keyframe, a picture resolution, or an object detection error margin.

42. The apparatus according to any one of claims 24 to 41, wherein the first resampling metric is a vertical resampling metric and the first downscaling factor is a vertical downscaling factor, or the first resampling metric is a horizontal resampling metric and the first downscaling factor is a horizontal downscaling factor.

43. An apparatus (600) for encoding a video comprising a sequence of pictures, the sequence of pictures comprising a first keyframe and a first set of pictures, wherein the apparatus is configured to perform a method comprising: obtaining (s502) a vertical resampling metric based on content of the first keyframe; obtaining (s504) a horizontal resampling metric based on the content of the first keyframe; using (s506) the vertical resampling metric and the horizontal resampling metric to determine whether a first downscaling condition is satisfied; as a result of determining that the first downscaling condition is satisfied, downscaling (s508) the pictures in the first set of pictures using at least one of a vertical downscaling factor or a horizontal downscaling factor, thereby producing a first set of downscaled pictures; and encoding (s510) each picture in the first set of downscaled pictures to produce a first set of encoded downscaled pictures, wherein using the vertical resampling metric and the horizontal resampling metric to determine whether the first downscaling condition is satisfied comprises: determining, based on the vertical resampling metric, whether a vertical downscaling condition is satisfied;determining, based on the horizontal resampling metric, whether a horizontal downscaling condition is satisfied; and the first downscaling condition is satisfied if either the vertical downscaling condition is satisfied or the horizontal downscaling condition is satisfied.

44. The apparatus according to claim 43, wherein downscaling the pictures in the first set of pictures using at least one of a vertical downscaling factor or a horizontal downscaling factor comprises at least one of: downscaling the pictures using the vertical downscaling factor if the vertical downscaling condition is satisfied, and downscaling the pictures using the horizontal downscaling factor if the horizontal downscaling condition is satisfied.

45. The apparatus according to claim 24, wherein the apparatus is further configured to perform the method of any one of claims 2-21.

46. The apparatus according to claim 24 or 43, wherein the apparatus is comprised in a video encoder.