Adaptive video thinning based on post-analysis and reconstruction requirements

Abstract

A method (400) for thinning a video including a sequence of pictures. The method includes determining whether to perform a video thinning process on a picture of the video. The method also includes performing a video thinning process on the picture of the video as a result of the decision to perform the video thinning process. The method also includes determining whether to perform the video thinning process on another picture of the video. The method also includes, after determining not to perform the video thinning process on the other picture, encoding the other picture to produce an encoded picture. The method further includes adding the encoded picture to a bitstream.

Description

[Technical Field]

[0001] This disclosure relates to video thinning. [Background technology]

[0002] 1. Video Compression A video consists of a series of pictures (also known as images or frames). Therefore, a video is often referred to as a video sequence. Each picture in a video sequence consists of one or more components. Each component can be described as a two-dimensional rectangular array of sample values ​​(also known as pixel values ​​or pixels for short). A picture typically consists of three components: one luma component (Y), where the pixel values ​​are luma values, and two chroma components (Cb and Cr), where the pixel values ​​are chroma values. The components are sometimes called "color components."

[0003] Video is the dominant form of data traffic in today's networks, and its share is expected to continue to grow. One way to reduce the data traffic per video is through compression. Here, video is encoded into a bitstream that constitutes the encoded video, which can then be stored and transmitted to the end user. Using a decoder, the end user can extract the video data and display it on a screen. However, because an encoder cannot know what type of device the encoded bitstream will be sent to, encoders typically compress the video according to a standardized compression scheme and format. This allows the video to be decoded by all devices that support the selected standard.

[0004] Compression can be lossless, meaning the decoded video is identical to the source given to the encoder, or lossy, where some degradation of the content is acceptable. This has a big impact on the bitrate, or how much compression can be achieved, as factors such as noise can make lossless compression quite expensive.

[0005] 2. Commonly Used Video Coding Standards Video standards are usually established by international organizations. The most widely used video compression standard today is H.264 / AVC, jointly developed by ITU-T and ISO. The first version of H.264 / AVC was completed in 2003 and has been updated several times over the years. The successor to H.264 / AVC, also developed by ITU-T and ISO, known as H.265 / HEVC (High Efficiency Video Coding), was completed in 2013.

[0006] High Efficiency Video Coding (HEVC) is a block-based video codec that utilizes both temporal and spatial prediction. Spatial prediction is achieved using intra (I) prediction from within the current picture. Temporal prediction is achieved using unidirectional (P) or bidirectional inter (B) prediction at the block level from a previously decoded reference picture. At the encoder, the difference between the original and predicted pixel data (called the residual) is transformed to the frequency domain, quantized, entropy coded, and transmitted along with necessary prediction parameters, such as the prediction mode and motion vectors, which are also entropy coded. The decoder performs entropy decoding, inverse quantization, and an inverse transform to obtain the residual, which is then added to the intra or inter prediction to reconstruct the picture.

[0007] MPEG and ITU-T have completed the successor to HEVC within the Joint Video Exploratory Team (JVET). This video codec is called Versatile Video Coding (VVC), and version 1 of the VVC specification was published in 2020 as Rec. ITU-T H.266|ISO / IEC23090-3 "Versatile Video Coding."

[0008] 3. Picture Order Count (POC) Pictures in HEVC and VVC are identified by a Picture Order Count (POC) value. Both the encoder and decoder keep track of the POC and assign a POC value to each picture that is coded / decoded.

[0009] There are three commonly used picture types: I-frames, P-frames, and B-frames. I-frames are coded independently of all other frames and can be decoded without reference pictures. A video usually begins with an I-frame. P-frames and B-frames use inter-prediction from other frames. P-frames can be predicted from one other frame, while B-frames can be predicted from at least one other frame. It doesn't matter what type of frame is referenced; for example, I-frames are usually coded at high quality, so it's quite common to predict from I-frames.

[0010] 4. Compressed Video Structure A coded video sequence begins with an independently coded picture (e.g., an I-frame). Then there are usually several frames that predict from at least one other frame, called B-frames. Coding is usually done hierarchically: frame 0 is coded first, then frame 16, which predicts from 0, then frame 8, which predicts from 0 and 16, etc. This is known as a group of pictures (GOP) structure.

[0011] The GOP structure defines how pictures can reference each other and defines specific settings for each picture. A GOP can be divided into temporal sublayers, as shown in Figure 7. In this example, there are five temporal sublayers with IDs (TId) 0 to 4. Pictures in one temporal sublayer may not reference pictures in higher temporal sublayers. The figure shows a typical GOP structure for random access in HEVC. Each GOP has 16 pictures, starting with the picture with TId 0. Arrows indicate references; for example, POC 8 is used as a reference for POCs 4, 6, 7, 9, 10, and 12. The decoding order, i.e., the order in which pictures are processed by the decoder, is typically optimized to minimize latency and memory usage by ensuring that referenced pictures are as close to the picture as possible. Summary of the Invention

[0012] Currently, several challenges exist. For example, many videos, even after compression, generally consist of a large amount of data, and transmitting and / or storing such a large amount of data can be costly. Moreover, some parts of the video data may be less important (i.e., lower priority) than other parts of the video data. In use cases where the video is intended to be used primarily for machine vision tasks, some pictures or image details that are expensive to transmit and / or store do not always contribute to the quality or accuracy of the machine vision task performed at the decoder side. These pictures or image details may not even be needed for human consumption of the decoded video.

[0013]

[0009] Accordingly, in one aspect, a video encoding method for thinning a video including a sequence of pictures is provided. The video encoding method includes determining whether to perform a video thinning process on a picture of the video. The method also includes performing the video thinning process on the picture of the video as a result of determining to perform the video thinning process. The method also includes determining whether to perform the video thinning process on another picture of the video. The method also includes, after determining not to perform the video thinning process on the other picture, encoding the other picture to produce a coded picture. The method further includes adding the coded picture to a bitstream.

[0014] In another aspect, a computer program is provided that includes instructions that, when executed by a processing circuit of a video encoding device, cause the video encoding device to perform the video encoding method disclosed herein. In another aspect, a carrier is provided that includes the computer program, the carrier being one of an electronic signal, an optical signal, a radio signal, and a computer-readable storage medium.

[0015] In another aspect, a video encoding apparatus is provided, wherein the video encoding apparatus is adapted to perform the video encoding method disclosed herein. In some embodiments, the video encoding apparatus includes a processing circuit and a memory including instructions executable by the processing circuit that cause the video encoding apparatus to operate to perform the video encoding method disclosed herein.

[0016] In another aspect, a video decoding method is provided for decoding an encoded video, wherein at least one picture of the video has been subjected to a video thinning process, the picture including machine vision features. The method includes obtaining a bitstream including the encoded video. The method also includes identifying rules for reconstructing the machine vision features. The method further includes using the rules and information obtained from the bitstream to reconstruct the machine vision features.

[0017] In another aspect, a computer program is provided that includes instructions that, when executed by a processing circuit of a video decoding device, cause the video decoding device to perform the video decoding method disclosed herein. In another aspect, a carrier is provided that includes the computer program, the carrier being one of an electronic signal, an optical signal, a radio signal, and a computer-readable storage medium.

[0018] In another aspect, a video decoding apparatus is provided, wherein the video decoding apparatus is adapted to perform any of the video decoding methods disclosed herein. In some embodiments, the video decoding apparatus includes a processing circuit and a memory including instructions executable by the processing circuit that cause the video decoding apparatus to operate to perform the video decoding method disclosed herein.

[0019] An advantage of embodiments is better utilization of bandwidth for transmission and storage of video content. This can be in the form of lower total bandwidth required or, in some embodiments, improved accuracy of the machine vision task at the decoder. This is achieved through a better trade-off in the form of devoting more bandwidth to video details that are important to the machine vision task and less bandwidth to video details that are less important. In addition, the thinned video bitstream may be decoded faster compared to the original bitstream due to a potentially lower number of pictures and / or a higher quantization parameter. Also, the thinned video bitstream may be decoded with a lower amount of energy and / or processing power compared to the original bitstream due to a potentially lower number of pictures. This may be important in use cases where there are hard constraints on decoding resources.

[0020] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate various embodiments. [Brief explanation of the drawings]

[0021] [Figure 1] 1 illustrates a system according to one embodiment. [Figure 2] 1 is a schematic block diagram of a video encoder according to one embodiment; [Figure 3] 1 is a schematic block diagram of a video decoder according to one embodiment; [Figure 4] 1 is a flowchart illustrating a process according to one embodiment. [Figure 5] 1 is a flowchart illustrating a process according to one embodiment. [Figure 6] FIG. 1 is a block diagram of an apparatus according to one embodiment. [Figure 7] A hierarchical GOP structure is shown. DETAILED DESCRIPTION OF THE INVENTION

[0022] As mentioned above, video may consist of a large amount of data, and some of this data (e.g., certain pictures of the video) may be of little value, especially in the context of machine vision applications. Accordingly, the present disclosure provides a video encoder operable to "thin" video. In this context, thinning video means "removing" data (especially less valuable data) from the video. Removing such less valuable data allows for better utilization of transmission bandwidth and storage space, and can be performed at the decoder side without significantly degrading the machine vision task. The thinning process may include (1) encoding only a subset of pictures (i.e., removing pictures), (2) using a relatively high quantization parameter (QP) for encoding and decoding less important frames, and / or (3) encoding high-priority pictures at a lower resolution.

[0023] Machine Vision Machine vision is a technology that is increasingly used in both industrial and consumer applications. Generally speaking, machine vision applications take input from a sensor (usually a camera), perform some processing, and provide an output. The range of applications is very broad and includes barcode scanners, product inspection on assembly lines, augmented reality applications for phones, and decision-making in self-driving cars.

[0024] Processing in machine vision applications can be performed by different algorithms running on different hardware. For some applications, a simple digital signal processor may be sufficient, while in other cases one or more graphics processing units (GPUs) are required. In recent years, the use of neural networks to process inputs has become very popular due to their versatility and the fact that they often outperform other machine vision methods.

[0025] The results generated by processing algorithms can also vary widely: a barcode scanner in a store might reveal a product number, a product inspection system might tell you if a product is defective, an augmented reality application on your phone might provide additional information to a filtered picture, or an algorithm in a self-driving car might tell you if the vehicle needs to slow down.

[0026] That said, there are a variety of tasks that machine vision algorithms can perform, including, for example:

[0027] (1) Object detection - Localizing objects in an input image or video according to their position and size. It is also possible to extract information about the nature of the detected objects, which can be used, for example, in the automatic tagging of image databases.

[0028] (2) Object Tracking - Based on the object detection task, it tracks objects through different frames of the input video. An example application is a surveillance system in a store that tracks customer movements.

[0029] (3) Object segmentation - Dividing an image or video into distinct regions that are easier to analyze or process. For example, applications such as replacing the background in a video stream use segmentation.

[0030] (4) Event detection - Based on input, the algorithm determines whether some kind of event is occurring. For example, a car's system may detect whether another car is changing lanes.

[0031] Related Applications The video encoder can analyze one or more frames, make decisions based on the detected content, and adjust encoding parameters. An example of an implementation of such a system is described in Axis Communications, "Axis Zipstream technology," Whitepaper (January 2018) (available at www(dot)axis(dot)com;path= / files / whitepaper / ;filename=wp_zipstream_71496_en_1801_lo.pdf). Three different aspects of the encoding can be adjusted:

[0032] (1) Region of Interest (ROI) -- A portion of a video is coded at a higher quality than the rest of the video.

[0033] (2) Group of Pictures (GOP) -- Based on the content, if little motion is detected, I-frames (which are independent of other frames) can be omitted to reduce the bitrate.

[0034] (3) Frames per second (fps) -- If few changes in content are detected, the number of frames encoded per second can be reduced. In some cases, this can be solved by sending null frames (frames consisting only of skipped blocks) instead of actual video frames to maintain the appearance of a constant frame rate.

[0035] 1 illustrates a system 100 according to one embodiment. The system 100 includes a video encoder 102 and a video decoder 104, with the encoder 102 in communication with the decoder 104 over a network 110 (e.g., the Internet or other network). That is, the encoder 102 encodes a video sequence 101 into a bitstream comprising the encoded video sequence and transmits the bitstream to the decoder 104 over a network 108. In some embodiments, rather than transmitting the bitstream to the decoder 104, the bitstream is stored in a data storage device.

[0036] The decoder decodes pictures included in the encoded video sequence to produce video data for display. Thus, the decoder 104 may be part of a device 103 having a display device 105. The device 103 may be a mobile device, a set-top device, a head-mounted display, etc.

[0037] 2 illustrates a video encoder 102 according to one embodiment. In this embodiment, the video encoder 102 includes a video thinning function (VTF) 201. The video thinning function 201 receives pictures of a source video and analyzes the pictures to determine whether a video thinning process should be performed on the pictures. That is, for example, the VTF 201 determines whether a picture is a "low priority" picture. In one embodiment, pictures of the video are either "high priority" pictures (i.e., pictures to which video thinning is not applied) or "low priority" pictures (i.e., pictures to which the video thinning process is applied). In one particular embodiment, the video thinning process consists of dropping pictures. That is, each low priority picture is dropped, while each high priority picture is output for further processing (however, in some embodiments, even low priority pictures are output for further processing).

[0038] In the illustrated embodiment, each picture output from VTF 201 is passed to motion estimation / compensation block 250 and intra predictor 249. The outputs from motion estimation / compensation block 250 and intra predictor 249 are input to selector 251, which selects either intra or inter prediction for the current block of pixels. The output from selector 251 is input to an error calculator in the form of adder 241, which also receives pixel values ​​for the current block of pixels. Adder 241 calculates and outputs a residual error as the difference in pixel values ​​between the block of pixels and a prediction of the block of pixels. The error is transformed in transformer 242, such as by a discrete cosine transform, quantized by quantizer 243, and then coded in encoder 244, such as by an entropy encoder. In inter-coding, estimated motion vectors are also provided to encoder 244 to generate a coded representation of the current block of pixels. The transformed and quantized residual error for the current block of pixels is also provided to an inverse quantizer 245 and an inverse transformer 246 to recover the original residual error. This error is added by an adder 247 to the block prediction output from the motion compensator 250 or the intra predictor 249 to create a reference block of pixels that can be used in predicting and coding the next block of pixels. This new reference block is first processed by the deblocking filter 200, which filters out blocking artifacts. The processed new reference block is then temporarily stored in a frame buffer 248, where it is made available to the intra predictor 249 and the motion estimator / compensator 250.

[0039] 3 is a schematic block diagram of decoder 104 according to some embodiments. Decoder 104 includes a decoder 361, such as an entropy decoder, for decoding the coded representation of a block to obtain a set of quantized and transformed residual errors. These residual errors are inverse quantized in an inverse quantizer 362 and inverse transformed by an inverse transformer 363 to obtain a set of residual errors. These residual errors are added to pixel values ​​of a reference block in an adder 364. The reference block is determined by a motion estimator / compensator 367 or an intra predictor 366 depending on whether inter- or intra-prediction is performed. A selector 368 is thereby interconnected to adder 364, motion estimator / compensator 367, and intra predictor 366. As a result, the decoded block output from adder 364 is input to a deblocking filter unit 300 (which may perform the same function as filter unit 200 of encoder 102). The filtered block is output from decoder 104 and is preferably further provided temporarily to frame buffer 365 so that it can be used as a reference block of pixels for subsequent blocks to be decoded. Frame buffer 365 is connected to motion estimator / compensator 367 to thereby make the stored block of pixels available to motion estimator / compensator 367. The output from adder 364 is also preferably input to intra predictor 366 to be used as an unfiltered reference block.

[0040] Video Thinning Function (VTF) 201 As mentioned above, in one embodiment, encoder 102 includes VTF 201, and in one particular embodiment, only a subset of video pictures are encoded and decoded according to normal procedures (e.g., only non-low priority pictures), while pictures determined to be low priority are subject to a video thinning process. For example, low priority pictures are dropped or encoded in a particular manner that results in video thinning. Thus, in an embodiment, VTF 201 determines, for each input picture, the priority level to assign to that picture (e.g., VTF 201 determines whether the picture is low priority). VTF 201's decision to determine a picture to be low priority can be based on, but is not limited to, one or more of the following:

[0041] In one embodiment, for each input picture, VTF 201 obtains (e.g., calculates) a similarity measure that indicates the degree to which the picture is similar to one or more other pictures (either previous or future pictures). If the similarity measure is greater than a threshold, the picture is determined to be a low-priority picture, i.e., eligible for video thinning. Many methods for determining a similarity measure are known. For example, VTF 201 in one embodiment calculates MSE=(1 / n)SUM[(Ai - Bi) for i=1 to n]. 2 ], where Ai is the ith pixel of picture A and Bi is the ith pixel of picture B. The MSE provides a similarity measure that indicates the similarity between picture A and picture B. In another embodiment, the VTF 201 uses the MSE to calculate the peak signal-to-noise ratio (PSNR). That is, the VTF 201 calculates PSNR=20log 10 (Max / (MSE) 1 / 2 ), where Max is a predetermined maximum signal value. PSNR also provides a similarity measure that indicates the similarity between picture A and picture B. In other embodiments, the similarity measure is a structural similarity (SSIM) metric or a multi-scale SSIM (MS-SSIM) metric, as known in the art of image processing.

[0042] In another embodiment, for each input picture, VTF 201 obtains (e.g., calculates) a similarity measure indicating the degree to which the content of the picture is similar to the content of one or more other pictures (previous or future pictures). The content of a picture can be detected, for example, by a machine vision algorithm. If the content is sufficiently similar to the content of one or more of the other pictures (e.g., if the similarity measure is greater than a threshold), VTF 201 can determine that the picture is a low-priority picture. As an example, a machine vision task might detect an object in picture 0 and also detect the same object in picture 1. If the object has not moved more than a certain distance, e.g., a few pixels, between picture 0 and picture 1, VTF 201 can determine that picture 1 is unnecessary because reusing picture 0 in the decoder will produce sufficiently good and very similar results, and as a result, VTF 201 determines picture 1 as the low-priority picture.

[0043] In another embodiment, VTF 201 analyzes a portion or the entire video sequence and determines every nth picture to be a low priority picture based on the rate of events in that portion of the video sequence.

[0044] In another embodiment, for each input picture, the VTF 201 defaults to determining that the picture is low priority unless certain conditions are met.

[0045] For example, in one embodiment, a picture is determined to be low priority unless: i) VTF 201 detects a new object in the picture (e.g., the picture contains an object, such as a red balloon, and none of the previous M pictures contained that object, where M is an integer greater than 0), ii) VTF 201 detects a new overlap area between two objects, iii) VTF 201 detects a previously defined event, such as object A colliding with object B, iv) VTF 201 detects a previously defined event, such as object A moving out of a defined area in the video picture, and / or v) VTF 201 detects a change in the predicted trajectory of an object.

[0046] As another example, for each input picture, VTF 201 performs a machine vision task on that picture (and possibly some other pictures), and based on the output of the machine vision task, VTF 201 determines that the picture is not low priority.

[0047] As another example, the condition is met when the number of low priority pictures reaches a preset upper limit. In one example, the maximum number of consecutive pictures that can be considered low priority is set to N, and if VTF 201 determines that N consecutive pictures are low priority, VTF 201 determines that picture N+1 is not low priority.

[0048] As another example, all pictures are low priority except for so-called "key pictures." In one embodiment, a key picture is a picture that is a particular type of picture or contains a particular slice type in a predefined GOP structure. In one example, all I-frames (or pictures containing one or more I-slices) in a GOP structure are designated as key pictures. In another example, a picture is a key picture unless it is a B-frame (i.e., I-frames and P-frames are key pictures).

[0049] In one embodiment, the video is encoded using a two-pass scheme. In the first pass, each picture in the video is analyzed. The determination of whether a picture is a key picture is based on, but not limited to, one or more of: i) a new event or object is detected in the picture; ii) a similarity measure indicating significant differences between the picture and one or more previous pictures; and / or iii) a neural network for key picture determination indicating that the current picture should be used as a key picture. In the second pass, the key pictures are encoded into the bitstream.

[0050] In one embodiment, when a low-priority picture is dropped, the encoder may temporarily reduce the frame rate. As an example, an encoder may output a video stream at 60 frames per second (fps), but, when certain conditions are met, drop every other picture to as low as 30 fps. In an alternative embodiment, rather than dropping low-priority pictures, the encoder may "skip" the frames. That is, it may encode the low-priority picture as cheaply as possible. A typical way to do this is to code all blocks of the low-priority picture with a motion vector of 0. This results in a picture with exactly the same content as the previous picture. This allows the encoder to maintain a constant frame rate of 60 fps, which may be necessary to accommodate certain decoders that cannot handle frame rate changes. While this method of signaling pictures using skips is not completely free, it is typically much cheaper per bit than encoding the picture as a regular picture.

[0051] In one embodiment, low-priority pictures are not dropped or skipped, but are instead coded using a higher QP value, which indicates lower visual quality. As with the case where low-priority pictures are coded skipped, this has the side effect of reducing the bit rate without changing the frame rate. However, in this embodiment, low-priority pictures are often not identical to the previous picture, which can be useful because it can roughly preserve motion within the sequence. This can be particularly useful when the video is to be viewed by humans in addition to being processed by machine vision algorithms.

[0052] Decoder Functionality In one embodiment, the decoder 104 is configured to use rules to reconstruct machine vision features in the low priority pictures. The rules may be, but are not limited to, interpolation rules, extrapolation rules, or prescribed trajectories.

[0053] In an embodiment, rules for reconstructing features in low priority pictures are specified at the encoder side and sent to the decoder in the thinned bitstream (in-band) or via a separate channel (out-of-band). The decoder 104 decodes the bitstream to produce decoded pictures, and using the decoded pictures and the received rules, the decoder 104 reconstructs machine vision features in the low priority pictures.

[0054] Interpolation rule examples In one example, from a sequence of pictures A, B, and C at the encoder side, picture B is determined to be a low priority picture, and as a result, encoder 102 drops picture B. Pictures A and C are coded, and the coded pictures A and C are sent to decoder 104 along with an interpolation rule. Decoder 104 decodes coded pictures A and C, and using the interpolation rule, features extracted from the decoded pictures A and C are interpolated to reconstruct the features of picture B.

[0055] Extrapolation Rule Example In one example, from a sequence of input pictures A and B at the encoder side, the encoder decides to drop picture B. Picture A is encoded and sent along with an extrapolation rule indicating a specific change in the location of a feature relative to the location of feature X in picture A. Decoder 104 decodes picture A and calculates the location of the feature in dropped picture B using the location of feature X extracted from picture A and the decoded extrapolation rule. The location of feature X' in picture B is determined by applying the extrapolation rule to the location of feature X in picture A. For example, picture A may contain an object (e.g., a kicked football), and picture B may also contain the object. The encoder may include delta y and delta x values ​​in the bitstream, and the decoder may determine the location of the football in picture B by calculating x + delta x and y + delta y, where x and y are the position of the football in picture A.

[0056] In one embodiment, the rules are not signaled to the decoder 104, but are assumed (e.g., the decoder 104 is pre-configured with the rules). In one example, the position of a machine vision feature in a dropped picture is always assumed to be the average of the positions of the feature in the pictures immediately before and after the dropped picture.

[0057] In another example, from a sequence of input pictures A, B, C, and D at an encoder, the encoder decides to drop pictures B and C. Feature X in picture A (e.g., an object in picture A, such as a football) is moving to a new position in picture D. Trajectory information indicating the path the object takes from its location in picture A to its location in picture D is encoded into the bitstream. Using the trajectory information included in the bitstream, a decoder can determine the position of the football in dropped picture B and the position of the football in dropped picture C. For example, in an embodiment, if the trajectory of the object is assumed to be a circular path with all points equally spaced along the circumference of a circle, the trajectory information need only include the center point of the circle because the decoder can determine the position of the feature in pictures A and D on its own, and once the decoder knows the center of the circle and the position of the feature in pictures A and D, it can use basic geometry to calculate the position of the feature in pictures B and C. In another embodiment, where the feature is a projectile, the trajectory information may be a first polynomial (e.g., y=-4.9t) to determine the y coordinate of the projectile. 2 +5t+7) and a second polynomial (e.g., x=.8t) to determine the x coordinate of the projectile. In this projectile embodiment, pictures A and D can also be dropped.

[0058] Adaptation of existing codecs In one embodiment, the skipping of pictures is signaled in the bitstream. In one embodiment, rules for reconstructing features in dropped pictures are signaled in the bitstream, such as, for example, in a Supplemental Enhancement Information (SEI) message. In one embodiment, the location of dropped pictures and rules for reconstructing features in dropped pictures are signaled in the bitstream, such as, for example, in an SEI message.

[0059] Changing GOP size or structure In one embodiment, the encoder 102 modifies the GOP structure of the thinned video sequence. As an example, the encoder 102 may have a default hierarchical GOP structure as shown in Figure 7. However, if the encoder determines that the video can be thinned, this GOP structure may be replaced, for example, as follows:

[0060] (1) A simple IPPPPP block structure where all P frames predict from the previous picture (this can be good if the encoder decides that all pictures except the first one in a GOP should be dropped),

[0061] (2) GOP structures with the same structure but different QP values,

[0062] (3) GOP structure with a small number of pictures,

[0063] (4) GOP structures with different numbers of temporal sublayers; and

[0064] (5) Asymmetric GOP structure, e.g., using multiple temporal sublayers in some parts of the GOP, but using only a single temporal sublayer in other parts of the GOP.

[0065] 4 is a flow chart illustrating a video encoding process 400 performed by the encoder 102. The process 400 may begin at step s402.

[0066] Step s402 involves determining whether to perform a video thinning process on the pictures of the video.

[0067] Step s404 includes performing a video thinning process on pictures of the video as a result of determining to perform a video thinning process.

[0068] Step s406 involves determining whether to perform the video thinning process on another picture of the video.

[0069] Step s408 includes, after determining not to perform the video thinning process on the other picture, encoding the other picture to produce a coded picture.

[0070] Step s410 involves adding the encoded picture to the bitstream.

[0071] In some embodiments, performing a video thinning process on a picture includes dropping the picture, skipping the picture, encoding the picture using a quantization parameter (QP) value associated with a lower priority picture, or encoding the picture to produce a lower resolution encoded picture.

[0072] In some embodiments, the picture includes a set of luma values ​​and a set of chroma values, and performing a video thinning process on the picture includes setting at least a subset of the luma values ​​to predetermined luma values ​​and setting at least a subset of the chroma values ​​to predetermined chroma values.

[0073] In some embodiments, determining whether to perform a video thinning process on a picture includes determining a picture order count (POC) of the picture and using the POC to determine whether to perform a video thinning process on the picture. In some embodiments, using the POC to determine whether to perform a video thinning process on the picture includes determining whether the POC is a multiple of N, where N is a predefined integer greater than or equal to 2. In some embodiments, the video encoder performs the video thinning process on every nth picture.

[0074] In some embodiments, the video encoding process also includes obtaining machine vision task information indicating a machine vision application to process the encoded pictures, and determining whether to perform a video thinning process on the pictures of the video includes using the machine vision task information in determining whether to perform the video thinning process on the pictures. In some embodiments, the machine vision task information identifies a machine vision task, and using the machine vision task information in determining whether to perform the video thinning process on the pictures includes using a threshold for the identified machine vision task in determining whether to perform the video thinning process on the pictures. In some embodiments, the machine vision task is one of an object detection task, an object tracking task, an object segmentation task, or an event detection task.

[0075] In some embodiments, the machine vision task is an event detection task, which includes one or more of detecting a new object, detecting a new overlap area between two objects, detecting a previously defined event such as object A colliding with object B, detecting a previously defined event such as object A moving out of a defined area in a video frame, or detecting a change in the predicted trajectory of an object.

[0076] In some embodiments, determining whether to perform a video thinning process on a picture includes obtaining a similarity measure indicative of similarity between the picture and one or more other pictures of the video.

[0077] In some embodiments, determining whether to perform a video thinning process on a picture includes obtaining a similarity measure indicating the similarity between the content of the picture and the content of one or more other pictures in the video.

[0078] In some embodiments, determining whether to perform a video thinning process on the picture includes using a neural network to determine the applicability of the video thinning process to the picture based on a machine vision task.

[0079] In some embodiments, the video encoding process also includes encoding one or more syntax elements into the bitstream, the one or more syntax elements specifying rules for reconstructing at least one machine vision feature of the picture. In some embodiments, the rules are one or more of an interpolation rule, an extrapolation rule, or a prescribed trajectory.

[0080] In some embodiments, one or more syntax elements specifying the rules are signaled in a supplemental enhancement information (SEI) message in the bitstream.

[0081] In some embodiments, the one or more syntax elements that specify the rule further specify the location of the picture (eg, the POC of the picture).

[0082] In some embodiments, the video encoding process also includes using a modified group of pictures (GOP) size or structure as a result of performing a video thinning process.

[0083] In some embodiments, performing a video thinning process on a picture includes skipping the picture, and skipping the picture includes encoding a frame skip syntax element into the bitstream.

[0084] In some embodiments, pictures of a video belong to a group of pictures (GOP). An example of a GOP is shown in Figure 7. As shown in Figure 7, each picture in the group is associated with a temporal sub-layer identifier. In some embodiments, the video encoding process, as a result of determining to perform the video thinning process on a picture, further includes performing a video thinning process on each picture in the group associated with a temporal sub-layer identifier greater than the temporal sub-layer identifier of the picture. In some embodiments, the process, as a result of determining to perform the video thinning process on a picture, further includes performing a video thinning process on each picture in the group associated with a temporal sub-layer identifier equal to the temporal sub-layer identifier of the picture.

[0085] In some embodiments, the pictures of the video belong to a group of pictures, one or more pictures in the group are dependent on the picture, and the video encoding process further includes, as a result of determining to perform the video thinning process on the picture, performing a video thinning process on each picture included in the group that is dependent on the picture.

[0086] 5 is a flow chart illustrating a process 500 performed by the decoder 104. The process 500 may begin at step s502.

[0087] Step s502 involves obtaining a bitstream containing the encoded video.

[0088] Step s504 includes identifying rules for reconstructing machine vision features.

[0089] Step s506 involves using the rules and information obtained from the bitstream to reconstruct machine vision features.

[0090] In some embodiments, identifying the rule includes decoding one or more syntax elements from the bitstream, the one or more syntax elements specifying the rule. In some embodiments, the one or more syntax elements are included in a Supplemental Enhancement Information (SEI) message.

[0091] In some embodiments, the rule is one or more of an interpolation rule, an extrapolation rule, or a prescribed trajectory.

[0092] In some embodiments, the rule is an interpolation rule, the information obtained from the bitstream includes an encoded version of a second picture of the video and an encoded version of a third picture of the video, and using the rule and information obtained from the bitstream to reconstruct the machine vision feature includes decoding the second picture and extracting a first feature from the decoded second picture, decoding the third picture and extracting a second feature from the decoded third picture, and interpolating the extracted features to reconstruct the machine vision feature.

[0093] In some embodiments, the rule is an extrapolation rule, the information obtained from the bitstream includes an encoded version of a second picture of the video, and using the rule and the information obtained from the bitstream to reconstruct the machine vision feature includes decoding the second picture to extract a first feature from the decoded second picture, determining a location of the first feature extracted from the second picture, and calculating i) the location of the first feature extracted from the second picture, and ii) the location of the machine vision feature using the extrapolation rule.

[0094] In some embodiments, the rule is a prescribed trajectory, and the information obtained from the bitstream includes an encoded version of a second picture of the video and an encoded version of a third picture of the video, and using the rule and information obtained from the bitstream to reconstruct the machine vision feature includes decoding the second picture and extracting a first feature from the decoded second picture, decoding the third picture and extracting a second feature from the decoded third picture, and applying the prescribed trajectory to reconstruct the machine vision feature.

[0095] 6 is a block diagram of an apparatus 600 for implementing the encoder 102 and / or the decoder 104 according to some embodiments. If the apparatus 600 implements a decoder, the apparatus 600 may be referred to as a "decoding apparatus 600," and if the apparatus 600 implements an encoder, the apparatus 600 may be referred to as an "encoding apparatus 600."

[0096] As shown in FIG. 6, device 600 may include processing circuitry (PC) 602, which may include one or more processors (P) 655 (e.g., a general-purpose microprocessor and / or one or more other processors, e.g., an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), etc.), which may be co-sited within a single housing or a single data center, or may be geographically distributed. That is, device 600 may be a distributed computing device. For example, some processors of the device may be used to implement VTF 201, while other processors of the device may be used to implement conventional video encoding functions, which may or may not be co-sited.

[0097] The device 600 may further comprise at least one network interface 648 including a transmitter (Tx) 645 and a receiver (Rx) 647 (e.g., the network interface 648 may be wirelessly connected to the network 110, in which case the network interface 648 is connected to an antenna configuration) to enable the device 600 to transmit data to and receive data from other nodes connected to the network 110 (e.g., an Internet Protocol (IP) network) to which the network interface 648 is connected (directly or indirectly), and a storage device (also known as a “data storage system”) 608, which may include one or more non-volatile storage devices and / or one or more volatile storage devices. In embodiments in which the PC 602 includes a programmable processor, a computer program product (CPP) 641 may be provided. The CPP 641 includes a computer-readable medium (CRM) 642 that stores a computer program (CP) 643, which includes computer-readable instructions (CRI) 644. CRM 642 may be a non-transitory computer-readable medium, such as a magnetic medium (e.g., a hard disk), an optical medium, a memory device (e.g., a random access memory, a flash memory), or the like. In some embodiments, CRI 644 of computer program 643, when executed by PC 602, configures CRI to cause device 600 to perform the steps described herein (e.g., steps described herein with reference to flowcharts). In other embodiments, device 600 may be configured to perform the steps described herein without the need for code. That is, for example, PC 602 may consist only of one or more ASICs. Thus, features of the embodiments described herein may be implemented in hardware and / or software.

[0098] Overview of Various Embodiments A1. A method for thinning a video comprising a sequence of pictures, the method comprising: determining whether to perform a video thinning process on a picture of the video; as a result of determining to perform the video thinning process, performing the video thinning process on the picture of the video; determining whether to perform the video thinning process on another picture of the video; and, after determining not to perform the video thinning process on the other picture, encoding the other picture to produce an encoded picture; and adding the encoded picture to a bitstream.

[0099] A2. The method of embodiment A1, in which performing a video thinning process on the picture includes dropping the picture.

[0100] A3. The method of embodiment A1, in which performing a video thinning process on a picture includes skipping the picture.

[0101] A4. The method of embodiment A1, wherein performing a video thinning process on a picture includes encoding the picture using a quantization parameter (QP) value associated with a lower priority picture.

[0102] A5a. The method of embodiment A1, in which performing a video thinning process on a picture includes encoding the picture to produce a lower-resolution encoded picture.

[0103] A5b. The method of embodiment A1, wherein performing a video thinning process on a picture includes encoding the picture to produce an encoded picture having a lower resolution than an encoded picture produced by encoding another picture.

[0104] A6. The method of embodiment A1, wherein the picture includes a set of luma values ​​and a set of chroma values, and performing a video thinning process on the picture includes setting at least a subset of the luma values ​​to predetermined luma values ​​and setting at least a subset of the chroma values ​​to predetermined chroma values.

[0105] A7. A method according to any one of embodiments A1 to A6, wherein determining whether to perform a video thinning process on a picture includes determining a picture order count (POC) of the picture and using the POC to determine whether to perform a video thinning process on the picture.

[0106] A8. The method described in embodiment A7, wherein using the POC to determine whether to perform a video thinning process on the picture includes determining whether the POC is a multiple of N, where N is a predefined integer greater than or equal to 2.

[0107] A9. The method of any one of embodiments A1 to A8, wherein the video thinning process is performed every nth picture.

[0108] A10. A method according to any one of embodiments A1 to A9, further comprising obtaining machine vision task information indicating that a machine vision application will process the encoded pictures, and determining whether to perform a video thinning process on the pictures of the video comprises using the machine vision task information in determining whether to perform the video thinning process on the pictures.

[0109] A11. The method of embodiment A10, wherein the machine vision task information identifies a machine vision task, and wherein using the machine vision task information in determining whether to perform a video thinning process on the picture includes using a threshold value for the identified machine vision task in determining whether to perform a video thinning process on the picture.

[0110] A12. The method of embodiment A11, wherein the machine vision task is at least one of an object detection task, an object tracking task, an object segmentation task, or an event detection task.

[0111] A13. The method of embodiment A11, wherein the machine vision task is an event detection task, and the event detection task includes one or more of detecting a new object, detecting a new overlap area between two objects, detecting a previously defined event such as object A colliding with object B, detecting a previously defined event such as object A moving out of a defined area in a video frame, or detecting a change in the predicted trajectory of an object.

[0112] A14. A method according to any one of embodiments A1 to A13, wherein determining whether to perform a video thinning process on a picture includes obtaining a similarity measure indicating similarity between the picture and one or more other pictures of the video.

[0113] A15. A method according to any one of embodiments A1 to A13, wherein determining whether to perform a video thinning process on a picture includes obtaining a similarity measure indicating similarity between the content of the picture and the content of one or more other pictures of the video.

[0114] A16. A method according to any one of embodiments A1 to A15, wherein determining whether to perform a video thinning process on the picture includes using a neural network to determine the applicability of the video thinning process on the picture based on a machine vision task.

[0115] A17. The method of any one of embodiments A1 to A16, further comprising encoding one or more syntax elements into the bitstream, the one or more syntax elements specifying rules for reconstructing at least one machine vision feature of the picture.

[0116] A18. The method of embodiment A17, wherein the rules are one or more of interpolation rules, extrapolation rules, or prescribed trajectories.

[0117] A19. The method of embodiment A17 or A18, wherein one or more syntax elements specifying the rules are signaled in a Supplemental Enhancement Information (SEI) message in the bitstream.

[0118] A20. The method of any one of embodiments A17 to A19, wherein the one or more syntax elements specifying the rule further specify a location of the picture (eg, a POC of the picture).

[0119] A21. The method of any one of embodiments A1 to A20, further comprising using a modified group of pictures (GOP) size or structure as a result of performing the video thinning process.

[0120] A22. A method according to any one of embodiments A1 to A21, wherein performing a video thinning process on a picture includes skipping the picture, and skipping the picture includes encoding a frame skip syntax element into the bitstream.

[0121] A23. A method according to any one of embodiments A1 to A22, wherein the pictures of the video belong to a group of pictures, each picture in the group being associated with a temporal sub-layer identifier, and wherein the method further includes, as a result of determining to perform a video thinning process on a picture, performing a video thinning process on one or more pictures in the group associated with a temporal sub-layer identifier that is greater than the temporal sub-layer identifier of the picture.

[0122] A24. The method of embodiment A23, wherein the method further includes, as a result of determining to perform a video thinning process on a picture, performing a video thinning process on each picture in the group associated with a temporal sub-layer identifier equal to the temporal sub-layer identifier of the picture.

[0123] A25. A method according to any one of embodiments A1 to A22, wherein a picture of the video belongs to a group of pictures, one or more pictures in the group are dependent on the picture, and the method further comprises, as a result of determining to perform a video thinning process on a picture, performing a video thinning process on each picture included in the group that is dependent on the picture.

[0124] B1. A computer program comprising instructions that, when executed by a processing circuit of a video encoding device, cause the video encoding device to perform a method according to any one of embodiments A1 to A25.

[0125] B2. A carrier containing the computer program of embodiment B1, wherein the carrier is one of an electronic signal, an optical signal, a radio signal, and a computer-readable storage medium.

[0126] C1. A video encoding device adapted to: determine whether to perform a video thinning process on a picture of a video; as a result of determining to perform the video thinning process, perform the video thinning process on the picture of the video; determine whether to perform the video thinning process on another picture of the video; and, after determining not to perform the video thinning process on the other picture, encode the other picture to produce an encoded picture; and add the encoded picture to a bitstream.

[0127] C2. The video encoding apparatus of embodiment C1, wherein the video encoding apparatus is further adapted to perform a method according to any one of embodiments A2 to A25.

[0128] D1. A video encoding apparatus comprising: a processing circuit; and a memory containing instructions executable by the processing circuit that cause the video encoding apparatus to perform the method of any of embodiments A1 to A25.

[0129] F1. A video decoding method performed by a video decoder for decoding encoded video, wherein at least one picture of the video has been subjected to a video thinning process, the picture including machine vision features, the method including: obtaining a bitstream including the encoded video; identifying rules for reconstructing the machine vision features; and using the rules and information obtained from the bitstream to reconstruct the machine vision features.

[0130] F2. The method of embodiment F1, in which identifying the rule includes decoding one or more syntax elements from the bitstream, the one or more syntax elements specifying the rule.

[0131] F3. The method of embodiment F2, wherein the one or more syntax elements are included in a Supplemental Enhancement Information (SEI) message.

[0132] F4. The method of embodiment F1, F2, or F3, wherein the rule is one or more of an interpolation rule, an extrapolation rule, or a prescribed trajectory.

[0133] F5. The method of any of embodiments F1 to F3, wherein the rules are interpolation rules, the information obtained from the bitstream includes an encoded version of a second picture of the video and an encoded version of a third picture of the video, and using the rules and information obtained from the bitstream to reconstruct machine vision features includes decoding the second picture and extracting a first feature from the decoded second picture, decoding the third picture and extracting a second feature from the decoded third picture, and interpolating the extracted features to reconstruct the machine vision features.

[0134] F6. The method of any of embodiments F1 to F3, wherein the rule is an extrapolation rule, the information obtained from the bitstream includes an encoded version of a second picture of the video, and using the rule and information obtained from the bitstream to reconstruct the machine vision feature includes decoding the second picture and extracting a first feature from the decoded second picture, determining a location of the first feature extracted from the second picture, and calculating i) the location of the first feature extracted from the second picture, and ii) the location of the machine vision feature using the extrapolation rule.

[0135] F7. The method of any of embodiments F1 to F3, wherein the rule is a prescribed trajectory, and the information obtained from the bitstream includes an encoded version of a second picture of the video and an encoded version of a third picture of the video, and wherein using the rule and information obtained from the bitstream to reconstruct machine vision features includes decoding the second picture and extracting a first feature from the decoded second picture, decoding the third picture and extracting a second feature from the decoded third picture, and applying the prescribed trajectory to reconstruct the machine vision features.

[0136] G1. A computer program comprising instructions that, when executed by a processing circuit of a video decoding device, cause the video decoding device to perform a method according to any one of embodiments F1 to F7.

[0137] G2. A carrier containing the computer program of embodiment G1, wherein the carrier is one of an electronic signal, an optical signal, a radio signal, and a computer-readable storage medium.

[0138] H1. A video decoding device adapted to: obtain a bitstream including encoded video; identify rules for reconstructing machine vision features; and use the rules and information obtained from the bitstream to reconstruct the machine vision features.

[0139] H2. The video decoding device of embodiment H1, wherein the video decoding device is further adapted to perform a method according to any one of embodiments F2 to F7.

[0140] I1. A video decoding device comprising: a processing circuit; and a memory containing instructions executable by the processing circuit that cause the video decoding device to perform the method of any of embodiments F1 to F7.

[0141] While various embodiments have been described herein, it should be understood that the embodiments have been presented by way of example only, and not limitation. Thus, the breadth and scope of the present disclosure should not be limited by any of the exemplary embodiments described above. Moreover, any combination of the above-described elements in all possible variations thereof is encompassed by the present disclosure unless otherwise indicated herein or otherwise clearly contradicted by context.

[0142] Additionally, while the processes described above and illustrated in the figures are depicted as a series of steps, this is done for illustrative purposes only, and it is therefore contemplated that some steps may be added, some steps may be omitted, the order of steps may be rearranged, and some steps may be performed in parallel.

Claims

1. 1. A method (400) for thinning a video comprising a sequence of pictures including at least a first picture and a second picture, the method comprising: determining whether to perform a video thinning process on a picture of the sequence of pictures by analyzing at least the first picture using a machine vision task (s402); performing a video thinning process on the first picture as a result of determining to perform a video thinning process (s404); and encoding the first picture based on the video thinning process to produce a first encoded picture; adding the first coded picture to a bitstream; determining (s406) whether to perform a video thinning process on the second picture of the sequence of pictures; After determining not to perform a video thinning process on the second picture, encoding the second picture to produce a second encoded picture (s408); adding (s410) the second coded picture to the bitstream; Including, The method further includes encoding one or more syntax elements into the bitstream; The method (400), wherein the one or more syntax elements specify rules for reconstructing objects detected by the machine vision task, the rules being one or more of an interpolation rule, an extrapolation rule, or a prescribed trajectory.

2. performing the video thinning process on the first picture; skipping the first picture; encoding the first picture using a quantization parameter (QP) value associated with a lower priority picture that is higher than a QP value associated with a higher priority picture; encoding the first picture to produce an encoded picture having a lower resolution than the encoded picture produced by encoding another picture; and If the first picture includes a set of luma values ​​and a set of chroma values, setting at least a subset of the luma values ​​to predetermined luma values ​​and setting at least a subset of the chroma values ​​to predetermined chroma values. The method of claim 1 , comprising one or more of:

3. the machine vision task: Object detection task, Object tracking task, Object segmentation tasks, or Event Detection Task The method of claim 1 , wherein the at least one of

4. The machine vision task is an event detection task, and the event detection task comprises: new object detection, Detecting new overlapping areas between two objects; Detection of a previously defined event, such as object A colliding with object B; Detection of a previously defined event, such as object A moving out of a defined area in a video frame, or Detecting changes in the predicted trajectory of an object The method of claim 1 , comprising one or more of:

5. 2. The method of claim 1, wherein the one or more syntax elements specifying the rules are signaled in a Supplemental Enhancement Information (SEI) message of the bitstream.

6. The method of claim 1 , further comprising using a modified group of pictures (GOP) size or structure as a result of said performing said video thinning process.

7. 2. The method of claim 1 , wherein performing the video thinning process on the first picture comprises skipping the first picture, and wherein skipping the first picture comprises encoding a frame skip syntax element into the bitstream.

8. the first picture and the second picture of the video belong to a group of pictures; each picture in the group is associated with a temporal sub-layer identifier; As a result of the method determining to perform the video thinning process on the first picture, a video thinning process for one or more pictures in the group associated with a temporal sub-layer identifier greater than the temporal sub-layer identifier of the first picture; and a video thinning process for each picture in the group associated with a temporal sub-layer identifier equal to the temporal sub-layer identifier of the first picture; The method of claim 1 , further comprising performing one of:

9. A non-transitory storage medium containing a computer program comprising instructions (644) that, when executed by a processing circuit (602) of a video encoding device (600), cause the video encoding device (600) to perform the method of any one of claims 1 to 8.

10. A video encoding device (600), comprising: A processing circuit (602); a memory (642) containing instructions (644) executable by the processing circuitry, the instructions causing the video encoding device (600) to: For a video including a sequence of pictures including at least a first picture and a second picture, determining (s402) whether to perform a video thinning process on pictures of the video by analyzing at least the first picture using a machine vision task; performing a video thinning process on the first picture as a result of determining to perform a video thinning process (s404); and encoding the first picture based on the video thinning process to produce a first encoded picture; adding the first coded picture to a bitstream; determining (s406) whether to perform a video thinning process on the second picture of the sequence of pictures; After determining not to perform a video thinning process on the second picture, encoding the second picture to produce a second encoded picture (s408); adding (s410) the second coded picture to the bitstream; a memory (642) operative to perform Equipped with The instructions (644) further include encoding one or more syntax elements into the bitstream; A video encoding device (600), wherein the one or more syntax elements specify rules for reconstructing objects detected by the machine vision task, the rules being one or more of an interpolation rule, an extrapolation rule, or a prescribed trajectory.

11. Video encoding apparatus (600) according to claim 10, further operative to perform the method according to any one of claims 2 to 8.

12. A video decoding method (500) performed by a video decoder (104) for decoding encoded video, wherein at least one picture of the video has been subjected to a video thinning process, the picture including at least one detected object, the method comprising: obtaining (s502) a bitstream containing the encoded video; identifying (s504) a rule for reconstructing the detected object by decoding one or more syntax elements from the bitstream that specify a rule, wherein the rule is one or more of an interpolation rule, an extrapolation rule, or a prescribed trajectory; using the rules and information obtained from the bitstream to reconstruct the detected objects (s506); and A video decoding method (500) comprising:

13. The method of claim 12 , wherein the one or more syntax elements are included in a Supplemental Enhancement Information (SEI) message.

14. The rule is: Interpolation rules, extrapolation rules, or prescribed trajectory 13. The method of claim 12, wherein the first and second electrodes are one or more of:

15. the rule is an interpolation rule, the detected object is a first detected object; the information obtained from the bitstream includes an encoded version of a second picture of the video and an encoded version of a third picture of the video; using the rules and the information obtained from the bitstream to reconstruct the detected objects; decoding the second picture and extracting a second detected object from the decoded second picture; decoding the third picture and extracting a third detected object from the decoded third picture; and and reconstructing the first detected object by interpolating the extracted second detected object and the extracted third detected object.

13. The method of claim 12, comprising:

16. the rule is an extrapolation rule, the detected object is a first detected object; the information obtained from the bitstream includes an encoded version of a second picture of the video; using the rules and the information obtained from the bitstream to reconstruct the detected objects; decoding the second picture and extracting a second detected object from the decoded second picture; determining a location of the second detected object extracted from the second picture; and i) the location of the second detected object extracted from the second picture, and ii) calculating the location of the first detected object using the extrapolation rule.

13. The method of claim 12, comprising:

17. The orbit is defined by the rules, the detected object is a first detected object; the information obtained from the bitstream includes an encoded version of a second picture of the video and an encoded version of a third picture of the video; using the rules and the information obtained from the bitstream to reconstruct the detected objects; decoding the second picture and extracting a second detected object from the decoded second picture; decoding the third picture and extracting a third detected object from the decoded third picture; and applying the defined trajectory to reconstruct the first detected object; 13. The method of claim 12, comprising:

18. A non-transitory storage medium containing a computer program comprising instructions that, when executed by a processing circuit (602) of a video decoding device (600), cause the video decoding device (600) to perform the method of any one of claims 12 to 17.

19. A video decoding device (600), comprising: A processing circuit (602); a memory (642), the memory including instructions (644) executable by the processing circuitry, the instructions operating the video decoding device (600) to decode encoded video, at least one picture of the video having been subjected to a video thinning process, the picture including at least one detected object, the video decoding device further comprising: Obtaining (s502) a bitstream containing encoded video; identifying (s504) a rule for reconstructing the detected object by decoding one or more syntax elements from the bitstream that specify a rule, wherein the rule is one or more of an interpolation rule, an extrapolation rule, or a prescribed trajectory; using the rules and information obtained from the bitstream to reconstruct the detected objects (s506); and a memory (642) operative to perform A video decoding device (600) comprising:

20. 20. Video decoding device (600) according to claim 19, further operative to perform the method according to any one of claims 13 to 17.

Images