Video processing systems and methods for machine vision

Abstract

A method for processing the encoded picture includes a decoder deriving a middle signal from a bitstream, deriving a final output signal using the middle signal, and outputting the middle signal and the final output signal. The method also includes performing a machine vision task using the middle signal output by the decoder.

Description

MEDIA PROCESSING SYSTEMS AND METHODSTECHNICAL FIELD

[0001] Disclosed are embodiments related to media (e.g., video and / or audio) processing systems and methods.BACKGROUND

[0002] Versatile Video Coding (VVC)

[0003] Versatile Video Coding (VVC) and its predecessor, High Efficiency VideoCoding (HEVC), are block-based video codecs standardized and developed jointly by ITU-T and MPEG. The codecs utilize both temporal and spatial prediction. VVC and HEVC are similar in many aspects. Spatial prediction is achieved using intra (I) prediction from within the current picture. Temporal prediction is achieved using uni-directional (P) or bi-directional inter (B) prediction on the block level from previously decoded reference pictures. In the encoder, the difference between the original pixel data and the predicted pixel data, referred to as the residual, is transformed into the frequency domain, quantized and then entropy coded before transmitted together with necessary prediction parameters such as prediction mode and motion vectors, also entropy coded. The decoder performs entropy decoding, inverse quantization, and inverse transformation to obtain the residual, and then adds the residual to an intra or inter prediction to produce a reconstructed a picture.

[0004] Parameter Sets

[0005] HEVC and VVC specifies three types of parameter sets, the picture parameter set (PPS), the sequence parameter set (SPS) and the video parameter set (VPS). The PPS contains data that is common for a whole picture, the SPS contains data that is common for a coded video sequence (CVS) and the VPS contains data that is common for multiple CVSs, e.g., data for multiple scalability layers in the bitstream. VVC also specifies one additional parameter set, the adaptation parameter set (APS). The APS carries parameters needed for an adaptive loop filter (ALF) tool, the LMCS tool and the scaling list tool. Both HEVC and VVC allow certain information (e.g., parameter sets) to be provided by external means. “By external means” should be interpreted as the information is not provided in the coded video bitstream but bysome other means not specified in the video codec specification, e.g., via metadata possibly provided in a different data channel, as a constant in the decoder, or provided through an API to the decoder.

[0006] Video Compression

[0007] Several video compression standards have been developed over the past decades, and each new generation of compression technology generally increases the compression efficiency, but at the cost of increasing the computational complexity of the decoder (e.g., the software and / or the hardware). In recent years filters that adjust the content of the video or picture after the decoding, so-called post-filters, have become more popular, as they can adjust the content to better suit the consumer of the decoded video, be it a machine or a human. In contrast to commonly used loop filters, the output of post-filters is not used to influence the prediction of samples during the decoding process.

[0008] Video Compression for Machines

[0009] One paradigm that has changed over the last years is that more video is being produced that is not consumed (e.g., viewed) by humans, but rather consumed (e.g., analyzed) by machines (e.g., machine vision applications performing machine visions tasks). Most encoders, however, are optimized towards the human visual system and the features of the video that are emphasized are not necessarily optimal for the machine vision tasks.

[0010] Machine Vision Tasks

[0011] Machine vision is a technology often used in industrial applications. In general, machine vision applications take input from a sensor, often a camera, perform some sort of processing and provide an output. The scope of applications is wide, including, for example, bar code scanners in grocery stores, product inspection devices in manufacturing plants, augmented reality application for phones, and self-driving cars.

[0012] The processing in machine vision applications can be done by different algorithms running on different hardware set-ups. In certain applications, a simple digital signal processor might suffice, whereas in other cases one or more graphics processing units (GPUs) are required. In recent years, processing the input with neural networks has gained strong attraction due to the versatility and performance of neural networks.

[0013] The result produced by the machine vision algorithm can also vary greatly. A barcode scanner in a grocery store can determine a product number, a product inspection device might tell whether a product is faulty, an augmented reality application on a phone could give you a filtered picture with additional information, and an algorithm in a self-driving car might provide an indication whether you need to reduce speed or not.

[0014] There are many different tasks that can be performed by machine vision algorithms, including:

[0015] (1) Object detection: objects in the input image or video are located corresponding to their position and size; it is also possible to extract information about the nature of the detected objects. This can for example be used in automated tagging of image databases;

[0016] (2) Object tracking: based on the object detection task, objects are traced through different frames of the input video; an example application is a surveillance system in a store that tracks the movement of customers;

[0017] (3) Object segmentation: an image or video is divided into different regions, with regions being easier to analyze or process (e.g., applications that replace the background in a video stream use segmentation); and

[0018] (4) Event detection: based on the input, the algorithm determines if there is a certain type of event happening, for example machines that detect whether a person or other object is in a danger zone and stop operations to avoid injuries.SUMMARY

[0019] Certain challenges presently exist. For instance, in conventional systems, a machine vision application receives only fully decoded pictures from the decoder; thus, the latency for the output of the machine vision application is the sum of latency for the full decoding process plus the processing time required by the machine task which leads to an undesirable amount of latency. For applications where human consumption is not the dominant mode of the consumption for the decoded picture or video, this introduces not only unnecessary latency, but also increased processing load and excessive power usage.

[0020] Accordingly, in one aspect there is provided a method for processing an encoded picture. In one embodiment, the method includes deriving a first set of post-reconstructionsamples from a bitstream. The method also includes filtering the first set of post-reconstruction samples using one or more loop-filters to produce a first set of last loop-filter output samples, wherein the first set of last loop-filter output samples is not identical to the first set of postreconstruction samples. The method also includes storing the first set of last loop-filter output samples in a decoded picture buffer (DPB). The method also includes outputting the first set of post-reconstruction samples and performing a machine vision task using the first set of postreconstruction samples. In another embodiment, the method for processing the encoded picture includes a decoder deriving a middle signal from a bitstream, deriving a final output signal using the middle signal, and outputting the middle signal and the final output signal. The method also includes performing a machine vision task using the middle signal output by the decoder.

[0021] In some aspects, there is provided a computer program comprising instructions which when executed by processing circuitry of an apparatus causes the apparatus to perform any of the methods disclosed herein. In one embodiment, there is provided a carrier containing the computer program wherein the carrier is one of an electronic signal, an optical signal, a radio signal, and a computer readable storage medium. In another aspect there is provided an apparatus that is configured to perform the methods disclosed herein. The apparatus may include memory and processing circuitry coupled to the memory.

[0022] The embodiments described in this disclosure provide numerous advantages.

[0023] A first advantage of embodiments disclosed herein is that the machine task can begin processing output from the decoder without having to wait for a fully decoded picture (or a fully decoded set of pictures). Because certain machine processing tasks can be done before a picture or video is fully decoded and output from the decoder, the results from the machine task are prepared faster compared to the solution where first the picture or video is output for human viewing and then the machine task is applied as a second step on the output picture or video. This results in a lower latency, which can be a significant advantage for latency critical applications.

[0024] A second advantage is that the embodiments provide the possibility for the machine task to be performed on an un-filtered picture or video. For some use cases, the performance of the machine task is improved if the input to the machine task process originates from the data point before some filtering steps in the decoding process such as loop filtering orpost-filtering steps. Especially in the case where the machine task is more NN-based and computationally heavy than the skipped steps (such as a NN-based loopfilter) it is easy to see that the machine task performance can improve by skipping the step. If the skipped processing is vital for good performance, the NN-based machine vision task could learn that processing as part of the machine vision task.

[0025] A third advantage of the embodiments is that they reduce the power consumption at the decoder side by using the knowledge about the target consumer of the decoded picture or video. The power saving at the decoder side is achieved when human consumption is not targeted and by skipping or shortening some of the decoding steps which are crucial for outputting picture or video for human consumption but not for the machine consumption. Power saving aspects can be more crucial for battery-based decoder scenarios. Because the decoding path for decoding a picture or video for machine consumption has lower complexity, this can have a larger impact on the power consumption in use cases where the machine consumption of the picture or video is the dominant mode and in comparison, human consumption of the decoded picture or video is happening less often.

[0026] Another advantage of the embodiments is that they allow using more complex networks for machine tasks at least for some of the pictures or frames in the video, compared to the pipelines that demand a full decode of the picture before starting to perform the machine task. In one example, the embodiments can use a machine task network for intra frames in a not all intra coded video sequence, where the machine task take up longer time for processing a picture or video frame than the time needed for full decode of a full frame, and use this solution with latency lower than the time required for full decoding of two consecutive pictures, compared to the traditional pipeline where the latency will be at least equal to the time required for full decoding of two consecutive pictures assuming a machine task with the same complexity.BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The accompanying drawings, which are incorporated herein and form part of the specification, illustrate various embodiments.

[0028] FIG. 1 illustrates a system according to an embodiment.

[0029] FIG. 2 is a functional block diagram of a decoder according to an embodiment.

[0030] FIG. 3 is a functional block diagram of a decoder according to an embodiment.

[0031] FIG. 4 illustrate an improvement in latency.

[0032] FIG. 5 illustrates use of a device input.

[0033] FIG. 6 illustrates using a more complex machine task at scene change or I-frames without increasing the machine task latency.

[0034] FIG. 7 illustrates a case for having interlaced high and low complexity machine tasks for every other frame.

[0035] FIG. 8 is a schematic block diagram of a decoder according to an embodiment.

[0036] FIG. 9 is a schematic block diagram of a decoder according to an embodiment.

[0037] FIG. 10 is a schematic block diagram of a decoder according to an embodiment.

[0038] FIG. 11 is a schematic block diagram of a decoder according to an embodiment.

[0039] FIG. 12 is a schematic block diagram of a decoder according to an embodiment.

[0040] FIG. 13 is a schematic block diagram of a decoder according to an embodiment.

[0041] FIG. 14 is a flowchart illustrating a process according to an embodiment.

[0042] FIG. 15 is a block diagram of an encoding apparatus according to an embodiment.

[0043] FIG. 16 is a flowchart illustrating a process according to an embodiment.DETAILED DESCRIPTION

[0044] FIG. 1 illustrates a media processing system (MPS) 100 according to an embodiment. MPS 100 includes an encoder 102 and a decoder 104, wherein encoder 102 is in communication with decoder 104 via a network 110 (e.g., the Internet or other network).Encoder 102 encodes a source media sequence 101 (which in the example that follow is a video sequence), which may be produced by a camera 199, into a bitstream 118 comprising an encoded video sequence and transmits the bitstream 118 to decoder 104 via network 108. In some embodiments, encoder 102 is not in communication with decoder 104, and, in such an embodiment, rather than transmitting bitstream 118 to decoder 104, the bitstream is stored in a data storage unit.

[0045] Decoder 104 decodes the pictures included in the encoded video sequence to produce video data for display and / or further processing. For example, the decoded video data output by decoder 104 may processed by a media processor (MP) 105. Accordingly, decoder 104 may be part of a device 103 having MP 105 and / or a display 106. MP 105 may perform machine vision tasks on the decoded pictures. One such machine vision task may be identifying objects in the picture. MP 105 may also perform post-filtering (e.g., neural network (NN) post-filtering) on the decoded pictures to produce post-filtered output samples (e.g., prior to or while performing the machine vision task(s)) and may out output the post-filtered output samples (e.g., a postfiltered picture) to display 106 for human consumption. The device 103 may be a mobile device, a set-top device, a head-mounted display, or any other device.

[0046] As an example, MPS 100 could be a surveillance system in which MP 105 detects a moving object and adjusts camera 199 to follow the moving object. As another example, MPS 100 can be used in a factory where work-pieces, such as steel sheets, are scanned by camera 199, the digital image of the sheet is analyzed by MP 105, and if any faults are detected by MP 105, the faulty pieces can be recycled or discarded. As the number of produced work-pieces is very high, it is necessary for MP 105 analyze them at very high speeds, i.e., with low latency, to avoid delays in the production.

[0047] As described above, a challenge presently exists because, in conventional media processing systems, the media processor has to wait for the decoder to complete its processing of a media item (e.g., picture) before the media processor can begin its processing.

[0048] Accordingly, this disclosure describes embodiments in which the decoder 102 outputs a middle signal 202 (see FIG. 2) which is not the final decoded picture or video and MP 105 uses middle signal 202 to perform one or more of its tasks (e.g., machine vision task or other processes to extract data or properties from one or more inputs). In one embodiment, the input(s) to MP 105 may be middle signal 202 and / or final output signal 204, and the task may be adaptable to the input signal if the input is a fully decoded picture or a middle signal from the decoder. In another example, only the middle signal is used as the input or the input can toggle between middle signal and the output signal or even both at the same time.

[0049] The media processor, therefore, may perform a task on a partial or mid-way decoded signal, e.g., picture / video, for use cases where low latency of the task results is critical.Some tasks can tolerate a level of coding artifacts and for particular use cases this tolerance can be exploited.

[0050] Referring back to FIG. 2, FIG. 2 illustrates one embodiment where an input to MP 105 is middle signal 202 (e.g., a partial or mid-way decoded picture / video) produced by a first decoding unit 211 of decoder 104. Some tasks can tolerate a level of coding artifacts and for particular use cases this tolerance can be exploited. For some other use cases, middle signal 202 is even a better input for the task than final output signal 204, which is produced by a second decoding unit 212 of decoder 104. In this case, using the middle signal can contribute to an improved performance of the task processing. In either case, latency of the output from MP 105 is improved (i.e., reduced) considering a shorter path that the signal takes from an encoded bitstream to the output of MP 105.

[0051] In other words, in some embodiments, the decoding process performed by decoder 104 is divided into a first part and a second part where the first part is performed faster compared to first and second parts together, and the machine task is performed on the output from the first part while the output of the second part is used for human consumption (the output from the second part may be post-filtered output samples). The rational is that if the machine task is performed on fully decoded frames for human consumption this could cause undesired delay. In one example, the heavy (e.g., neural network-based) post-filtering, which is not necessary for some machine tasks, is skipped. In another example, the first part of the decoding process with lower complexity and delay is the default for the output to be fed to MP 105, while the second part of the decoding process for human consumption purposes is performed only on demand.

[0052] Hence, as an example, first decoding unit 211 is a picture reconstruction unit and second decoding unit 212 is a filtering unit (e.g., a unit that includes a loop filter unit and may also include a post-filter). The filtering unit might include a computationally heavy filter such as a neural network (NN) filter which aims at removing compression artifacts and making the decoded signal visually pleasant for human consumption, which might not be necessary at all for machine consumption purposes.

[0053] In some cases, tasks performed on fully decoded frames aimed for human consumption causes unnecessary delay. For some applications, a low latency task is critical forinstance when the system is interactive and the reaction time is important. One example scenario is monitoring and controlling a production line in a factory, where simple capturing systems (cameras) are used at different critical points of the production line and the machine vision task is performed at the decoder side and the human consumption of the decoded picture / video is not the main mode. It is important that the machine vision task performs well, and the system reacts to observed failures with minimum latency.

[0054] In some use cases, the human vision consumption is not the main mode for the system. In this use cases, the human consumption mode could be served on demand while the lower complexity decoding for the purpose of machine task could be the default. Use of the lower complexity path as the more often used mode could reduce power consumption as well.

[0055] In summary, as described above, a process 1600 (see FIG. 16) for processing an encoded picture may be performed. Process 1600 starts with step sl602. Step sl602 comprises decoder 104 producing middle signal 202 using bitstream 118. Step si 604 comprises decoder 104 deriving final output signal 204 using middle signal 202. Step sl606 comprises decoder 104 outputting middle signal 202. Step sl608 comprises decoder 104 outputting final output signal 204. And step si 610 comprises MP 105 performing a machine vision task using the middle signal 202 output by decoder 104. In some embodiments, deriving the final output signal 204 using the middle signal comprises filtering the middle signal using a post-filter. In some embodiments, the method further comprises providing the final output signal to display 106 for displaying the final output signal. In some embodiments, the middle signal comprises a first set of loop-filter output samples, and the method further comprises storing the first set of loop-filter output samples in a decoded picture buffer. In some embodiments, the machine vision task is performed using the middle signal output by the decoder and the final output signal output by the decoder.

[0056] FIG. 3 illustrates decoding unit 211 and decoding unit 212 according to some embodiments. It should be noted that decoder 104 may be implemented differently so implementations other than this specific example can be used. In the embodiment shown in FIG. 3, decoding unit 211 is a picture reconstruction unit (or “reconstruction unit (RU)” for short) that includes a decoder module 361 (e.g., an entropy decoder) that decodes from the bitstream 118 quantized transform coefficient values of a block. The quantized transform coefficient valuesare subject to an inverse quantization process 362 and inverse transform process 363 to produce a residual block. This residual block is input to adder 364 that adds the residual block and a prediction block output from selector 390 to form a reconstructed block. Selector 390 either selects to output an inter prediction block or an intra prediction block. The reconstructed block is stored in a reconstructed picture buffer (RPB) 365. The inter prediction block is generated by the inter prediction module 350 and the intra prediction block is generated by the intra prediction module 369. Decoding unit 212 in this example includes a loop filter unit 367 (e.g., a unit that includes one or more loop filters) that applies loop filtering and the final decoded picture may be stored in a decoded picture buffer (DPB) 368. As illustrated in FIG. 3, media processor (MP) 105 may receive several inputs: 1) a reconstructed picture from RPB 365 and / or 2) loop filtered reconstructed picture from loop filter unit 367.

[0057] FIG. 4 is a diagram illustrating the latency improvement provided by embodiments of this disclosure.

[0058] General Embodiment

[0059] A general embodiment includes a method for decoding a bitstream where the bitstream is decoded in at least a first and a second step, wherein the output of the first step is a middle signal, and the output of the second step is a fully decoded item (e.g., picture or block), and the method comprises outputting the middle signal. One example is when the decoded signal is a picture or video. In one embodiment, the middle signal is output before outputting the fully decoded item. In one embodiment, the item is one coding unit of a picture. In another example, the item is a complete decoded picture. In some embodiments, the second step is a filtering step such as a loop filtering. In some embodiments, the second step uses a neural network for filtering purposes.

[0060] Machine vision task

[0061] In some embodiments, the middle signal is output from the decoder, and in addition the middle signal is further used as an input to MP 105. In one example, MP 105 is configured to perform a machine vision task. Examples of machine vision task are object recognition, object tracking, segmentation, event detection, etc. These machine vision tasks may be implemented using neural networks. In some embodiments, MP 105 is separate andapart from decoder 104, while in other embodiments MP 105 is an integral component of decoder 104.

[0062] In some embodiments, MP 105 may use the middle signal or the fully decoded signal (or both) as input.

[0063] FIG. 5 illustrates a variant of this embodiment in which a device input 502 specifies the input to MP 105, e.g., whether middle signal or the fully decoded signal or both is / are used as the input to MP 105. The device input may be a particular setting on the device such as a power setting, a mode setting or a preference setting. In one example, a first input signal from the device specifies the middle signal, a second input signal from the device specifies the fully decoded signal is used as the input to MP 105. In another example a third input signal from the device specifies that both the middle signal and the fully decoded signal are used as the input to MP 105.

[0064] Machine Vision Task More Complex than the Decoder

[0065] In some embodiments, MP 105 can be more complex than decoder 104 so that MP 105 may take longer to process a portion of a picture than the full decoding process for that portion of the picture (total time needed for the first and second decoding steps together). In some embodiments, MP 105 applies the machine vision task on particular picture types, e.g. I or P or B pictures.

[0066] FIG. 6 illustrates the embodiments providing the possibility for using a more complex machine task at scene change or I-frames without increasing the machine task latency.

[0067] FIG. 7 illustrates the case for having interlaced high and low complexity machine tasks for every other frame and that the latency in the traditional pipeline exceeds the decoding time of two pictures but the latency in the embodiments is kept within the decoding time of two pictures:

[0068] Both middle signal and the fully decoded first portion are used as the input to the MP

[0069] In some cases, the machine task can get more information if it obtains both middle signal 202 and signal 204. As an example, the picture before a loop filter (LF) (e.g., before a deblocking filter (DBF)) may be of value to processor 105 because the picture beforethe DBF has not been destroyed by the deblocking processing, which is mainly for human consumption purposes. Hence, in some cases, the middle signal is taken before the loop filter unit 367 (e.g., from RPB 365). However, after the DBF, some codecs have an adaptive loop filter (ALF). This filter takes in the samples output from the DBF as well as side information in the form of filter coefficients. Using this side information, the ALF filter can improve the quality compared to the original. If only using the middle signal, this improvement is lost to the machine task. Therefore, in one embodiment, both signals are sent to MP 105, and it thus receives the maximum amount of information, and can potentially make for a better decision. In some embodiments there is no gain in latency, since the machine task must wait for both the middle signal (which arrives first) as well as for the final signal (which arrives later) before being able to perform the machine task.

[0070] FIG. 8 further illustrates loop filter (LF) unit 367 according to an embodiment. In the embodiment shown, LF unit 367 includes three LF sub-stages: i) a first loop filter (LF1) 811, ii) a second loop filter (LF2) 812, and iii) a third loop filter (LF3) 813.

[0071] As described above, using the bitstream 118 and sample values (or simply “samples” for short) from DPB 368, the reconstruction unit 211 produces samples 870 that are then loop filtered with more loop filters 811, 812, and / or 813. For example, in the embodiment shown: filter 811 receives samples 870 and filters these samples to produce first filtered samples 871; filter 812 receives the first filtered samples 871 and filters these samples to produce second filtered samples 872; and filter 813 receives the second filtered samples 872 and filters these samples 872 to produce third filtered samples 873 (also denoted as “last loop filter output samples” or “last loop-filter output samples”), which are stored in DPB 368. In the embodiment shown, MP 105 obtain samples 893 from DPB 368 (these samples are denoted “MP input samples 893”). The samples may comprise samples for a full picture or for a partial picture.

[0072] As shown in FIG. 8, the first loop filter 811 may be a deblocking filter, the second loop filter 812 may be an SAO filter, and the third loop filter 813 may be an ALF, all of which are used in VVC. It should be understood, however, that these are just examples of loop filters and that there can be fewer or more than three. In the example shown, ALF is the last loop filter and hence the output samples of ALF are the last loop filter output samples 873. These samples are then fed to DPB 368.

[0073] In the example shown, MP 105 obtains the last loop filter output samples 873 from DPB 368 (MP 105 may obtain samples 873 from DPB 368 at essentially the same time the sample are stored in DPB 368 or after a delay). Hence, in the example shown, MP input samples 893 equals the last loop filter output samples 873 sent to the DPB 368, possibly at an earlier time (e.g., MP input samples 893 may be sent to MP 105 several video frames after the samples 873 are sent to DPB 368). The reason why there may be this delay is to allow the pictures sent to the MP 105 to be in display order while the pictures sent to DPB 368 are in decoding order.

[0074] One problem with using LF unit 367 is that the loop filtering performed by LF unit 367 can destroy information. Hence, having MP 105 process the samples only after ALF 813 has ‘destroyed them’ can therefore be undesirable. Another problem is that LF unit 367 creates latency.

[0075] Hence, this disclosure provides that the decoder may output samples (e.g., send out to MP 105 and / or make available to MP 105) that have not been loop-filtered with one or more (or any) of the loop filters of LF unit 367. This way MP 105 can be given or retrieve nonfiltered, non-destroyed samples from before a loop filter or loop filters.

[0076] In some embodiments, the decoder must still filter the samples that are put into the decoded picture buffer used for prediction with the specified loop filters, so as to avoid drift. In some embodiments, the MP 105, in addition to the non-loop filtered samples, also gets the loop filtered samples as input. In yet other embodiments, in addition to getting the non-loop filtered samples, MP 105 also gets the bitstream (the entire or parts thereof) as input. In yet other embodiments, in addition to getting the non-loop filtered samples, MP 105 also gets parameters that have been extracted from the bitstream. As an example, MP 105 may want to do its own filtering with the help of the ALF parameters that the encoder has sent. This filtering may be different from, and better than, the filtering that the ALF loop filter would do with the same ALF parameters. Because the order in which decoder 104 decodes pictures (often denoted the ‘decoding order’) may be different from the order in which the pictures are displayed or output (often denoted the ‘display order’ or ‘output order’), it may be necessary to store the samples in a buffer to change the order. This buffer may be separate from DPB 368 and is denoted MP picture buffer (MPB), and it may be placed before MP 105 or after MP 105, or it may be a part of MP 105. In some cases, such as when using low-delay configurations, aMPB may not be necessary. In some embodiments, the MPB is integrated into the DPB such that the DPB stores both the last loop filter output samples 873 and the post-reconstruction samples. In some implementations, the DPB stores both the last loop filter output samples 873 and post-filter output samples. Although the DPB and MPB are illustrated as different units in FIGs. 9-13, they should throughout this disclosure be seen as possibly merged into one buffer.

[0077] 1. Using post-reconstruction samples as input to MP 105

[0078] FIG. 9 illustrates an embodiment. In the embodiment shown, samples that come from a point in the decoding chain after the reconstruction unit 211 but before the last loop filter 813 are i) output (e.g., pushed to MP 105 or retrieved by MP 105) and ii) further loop filtered. These samples are denoted “post-reconstruction samples.” In FIG. 9, the postreconstruction samples are the samples 872 produced by filter 812. As shown, in this embodiment, these samples 872 are output to the MP 105 and also are provided to LF3 813 for further filtering. For instance, a copy of samples 872 are output to MP 105. Hence, the MP input samples 893 are identical to the samples 872, but different from the last loop filter output samples 873 (that is, the samples 872 provided to MP 105 are not filtered by LF3 813). As shown in FIG. 9, MP 105 is a component of a MP unit (MPU) 966 that also may include an MPB 968. Hence, samples 872 are output directly to MP 105 or output indirectly via MPB 968 if the MPB 968 is located between MP 105 and the input to MPU 966.

[0079] Like the embodiment shown in FIG. 8, in the embodiment of FIG. 9 the last loop filter output samples 873 are fed to DPB 368. Hence in this embodiment we have that DPB input samples equal last loop filter output samples 873, and MP input samples 893 equal samples 872. Hence, the last loop filter output samples 873 are not provided to MP 105 in this example. It should be noted that for the last loop filter output samples 873 to be different from the MP input samples 893, the loop filter 813 cannot always do the identity transform, because then these two signals (873 and 893) become equivalent.

[0080] FIG. 10 illustrates another embodiment. In this embodiment, the postreconstruction samples input to MPU 966 (and, therefore, input to MP 105 either directly or via MPB 968) are the samples 870 output from reconstruction unit 211.

[0081] In this embodiment, the DPB input samples are the samples 873 output from filter 813 and MP input samples 893 are the samples 870 output by the reconstruction unit 211.Hence, the MP input samples 893 are not the samples 873 output from filter 813.

[0082] In one embodiment, the decoder will output post-reconstructed samples (e.g., samples 870, 871, or 872), while the samples put into DPB 368 will be samples that have been filtered by all of the loop filters. In this embodiment, the decoder decodes the bitstream, including applying one or more loop filters, to generate the DPB input samples. These DPB input samples are stored in DPB 368 and may be used for prediction in the decoding process of future pictures. In the art, the decoder would output the DPB input samples. However, in this embodiment, the decoder instead outputs samples that come from a point in the decoding chain after the reconstruction but before the last loop filter. This means that the samples that are output (which are the post-reconstruction samples) are subject to one or more in-loop filters before the samples are stored in DPB 368. In this embodiment, there may be a MPB that enables reordering of samples from decoding order to display order.

[0083] Accordingly, in this embodiment, a method may include the following steps:

[0084] Step 1 : derive post-reconstruction samples from the bitstream. The postreconstruction samples come from a point in the decoding chain after the reconstruction but before the last loop filter 813 (e.g., the post-reconstruction samples may be samples 871 output from filter 811, samples 872 output from filter 812, or samples 870 output from reconstruction unit 211).

[0085] Step 2: filter the post-reconstruction samples using one or more loop filters to obtain last loop filter output samples (e.g., samples 873). The one or more loop filters may for instance be an ALF, an SAO filter, or an NN-based loop filter.

[0086] Step 3 : store the last loop filter output samples 873 in the DPB.

[0087] Step 4: output the post-reconstruction samples from the decoder. Hence, the post-reconstruction samples that are output are an example of the above described middle signal. In one embodiment, the output in this step is done by external means. For example, MP 105 can use a decoder API to request the post-reconstruction samples from the decoder. This may be implemented such that the API provides a pointer to the post-reconstruction samples.Optionally, using a buffer (e.g., MPB, DPB) the post-reconstruction samples may be reordered by decoding order before being output.

[0088] Optionally, the decoder may use the last loop filter output samples, that are stored in DPB 368, in the decoding process of a picture that follows the current picture in decoding order.

[0089] In some embodiments, MP 105 performs a machine vision task on the postreconstruction samples (and / or a post-filtered version of the post-reconstruction samples if MP 105 employs a post-filter). Hence, the following steps may be performed:

[0090] Step 1 : obtain a bitstream (the bitstream, which is typically a video bitstream, may for instance be obtained after transmitted over a network or may be obtained from a storage unit).

[0091] Step 2: derive post-reconstruction samples from the bitstream.

[0092] Step 3: filter the post-reconstruction samples using one or more loop filters to obtain last loop filter output samples.

[0093] Step 4: store the last loop filter output samples in DPB 368.

[0094] Step 5: MP 105 obtains the post-reconstruction samples as input and performs a machine vision task using the input (the input may also include the last loop filter outer samples and / or a post-filtered version of the last loop filter output samples).

[0095] FIG. 11 illustrates an embodiment in which both post-reconstruction samples (e.g., samples 872 in this example) and the last loop filter output samples 873 (directly from the last loop filter 813 (shown) or indirectly via the DPB 368 (not shown)) are used as input to the MP 105 (e.g., as input to MPB 968 from which MP 105 obtains the samples). In this alternative the decoder may additionally output the last loop filter output samples 873.

[0096] Using more than one post-reconstructed picture as input to MP 105

[0097] The post-reconstruction samples may comprise samples for a full picture or for a partial picture. In an alternative version, the post-reconstruction samples may comprise samples for more than one picture, where each comes from multiple points in the decoding chain after the reconstruction but before the last loop filter.

[0098] FIG. 12 illustrates an embodiment in which, multiple post-reconstructed samples (i.e.., samples 871 and 872 in the example shown) for more than one picture and the last loop filter output samples 873 (directly from the last loop filter 813 (shown) or indirectly via the DPB 368 (not shown)) are provided to MP 105 (directly or via MPB 968). As shown in FIG. 12, the post-reconstructed samples for a first picture coming from after LF1 but before LF2, and the post-reconstructed samples for a second picture coming from after the SAO loop filter but before the ALF loop filter, are used as input to MP 105 together with the last loop filter output samples.

[0099] FIG. 13 illustrates an embodiment where one or more of post-reconstruction samples 870, 871, 872 and the last loop filter output samples 873 (directly from the last loop filter 813 (shown) or indirectly via the DPB 368 (not shown)) are used as input to MP 105 (directly or via MPB 968). In a second embodiment, at least one of 870, 871, 872 is used as input to MP 105. FIG. 13 also illustrate that all or parts of the bitstream and / or parameter values derived from the bitstream may optionally also be input to the MP 105 in addition to the post-reconstructed samples.

[0100] Indicator for determining what samples to output

[0101] In one embodiment, an indicator value is used to determine what types of samples to output from the decoder. In one embodiment the indicator value is decoded from one or more syntax elements in the bitstream. The indicator may for instance be decoded from a sequence parameter set (SPS), picture parameter set (PPS), picture header (PH), an supplemental enhancement information (SEI) message or similar structure. The indicator is different from the picture output flag that is present in both HEVC and VVC. While the picture output flag specifies which pictures from DPB 368 to output from the decoder, the indicator of this embodiment indicates the point (or points) in the decoding chain the samples to be output from the decoder is taken from. Referring to FIG. 9, the indicator may specify whether samples870, 871 or 872 shall be output. In the case there are only two loop filters in the decoding chain, e.g., there is no LF2 812, the indicator may specify whether to output samples 870 or871. Note that for all options, the samples that are output differ from the samples that are stored in DPB 368. The indicator may be a binary flag or a parameter with more than two values.

[0102] In one embodiment, the indicator value specifies whether to output the postreconstructed samples or not from the decoder. In this version, the post-reconstructed samples may be outputted in addition to the last loop filter output samples (e.g., samples 873). This is exemplified in the syntax and semantics below with a flag, sps_output_post_reconstructed_samples in SPS. However, the location of the flag could as described above alternatively be present in e.g., the PPS, the PH, and SEI message or similar structure.TABLE 1

[0103] In one embodiment, when sps_output_post_reconstructed_samples is equal to 1 this specifies that post-reconstructed samples are output from the decoder; while sps_output_post_reconstructed_samples equal to 0 specifies that post-reconstructed samples are not output from the decoder.

[0104] In another embodiment, the indicator value specifies whether to output the postreconstructed samples or the last loop filter output samples from the decoder. This is exemplified with the same syntax as above but with a slightly different semantics for the flag as follows: when sps output post reconstructed samples is equal to 1 this specifies that postreconstructed samples are output from the decoder and that the last loop filter output samples are not output from the decoder; while sps_output_post_reconstructed_samples equal to 0 specifies that the last loop filter output samples are output from the decoder and that post-reconstructed samples are not output from the decoder.

[0105] In another embodiment, the indicator value specifies one of the following: i) the post-reconstructed samples are outputted from the decoder; ii) the last loop filter output samples are outputted from the decoder, or iii) both the post-reconstructed samples and the last loop filter output samples are outputted from the decoder.

[0106] In another embodiment, the point in the decoding chain from where to output samples from the decoder is specified by the indicator value. The possible options may comprise one or more of the following: i) Output post-reconstructed samples after reconstruction but before deblocking, ii) Output post-reconstructed samples after deblocking but before SAO, iii) Output post-reconstructed samples after SAO but before ALF, iv) or Output the last loop filter output samples.

[0107] In one embodiment the indicator is decoded from the same NAL unit as the information describing MP 105, e.g., in an SEI message.

[0108] In another embodiment the indicator value is not decoded from the bitstream but derived by external means. “By external means” should be interpreted as the information is not provided in the coded video bitstream but by some other means not specified in the video codec specification, e.g., via metadata possibly provided in a different data channel, as a constant in the decoder, or provided through an API to the decoder.

[0109] The Middle Signal is Provided in Parts

[0110] In some decoders, the loop filtering steps happen on a per-picture basis. As an example, the decoder may first reconstruct the entire picture, then output this fully reconstructed picture to the DBF, which then deblocks the entire picture and then feeds it to the ALF, which filters the entire picture to obtain the final signal. In this case one may reduce latency by extracting the middle signal from before the DBF. The resulting saving in latency will be the amount of time it takes for the DBF to filter the picture plus all the time spent performing the other filtering (e.g., SAO and ALF filtering).

[0111] Some decoders, however, do not operate on a picture level but rather on a smaller level, such as on the coding tree unit (CTU) level. In VVC, the picture is divided into CTUs of size 128x128 samples. Once the first (top left) CTU has been reconstructed, the reconstruction of the second CTU (to the right of the first one) can commence. In parallel to that, a clever decoder may start the deblocking process of the first CTU. Likewise, when reconstructing the third CTU, the decoder can in parallel deblock the second CTU and ALF-filter the first CTU. This means that using the samples before deblocking as the middle signal will only save two CTUs worth of processing (one for deblocking and one for ALF), rather than all of the processing in deblocking and ALF. Still, this reduction in latency may be advantageous for machine vision tasks.

[0112] In some cases MP 105 can start processing the first CTU as soon as it is finished. Therefore, in one embodiment of the present invention, the middle signal is sent to MP 105 CTU per CTU, as soon as the CTUs are finished.

[0113] As an example, after having reconstructed the first CTU, the decoder can simultaneously: a) start the reconstruction of the second CTU, b) send the first CTU to MP 105 so that MP 105 can perform its configured machine task, and c) perform deblocking on the first CTU. After the second CTU has been decoded, the decoder can simultaneously: a) start the reconstruction of the third CTU, b) send the second CTU to MP 105 so that MP 105 can perform its configured machine task, c) perform deblocking on the second CTU, and d) perform ALF filtering on the first CTU.

[0114] In one embodiment, device 103 may perform the following steps:

[0115] Step 1 : obtain a bitstream (the bitstream, which is typically a video bitstream, may for instance be obtained after transmitted over a network).

[0116] Step 2: derive post-reconstruction samples from the bitstream. This is typically done in the decoding process. The post-reconstruction samples come from a point in the decoding chain after the reconstruction but before the last loop filter. The post-reconstruction samples may comprise samples for a full picture or for a partial picture. In one version, the post-reconstruction samples may comprise samples for more than one picture coming from multiple points in the decoding chain after the reconstruction but before the last loop filter.

[0117] Step 3: filter the post-reconstruction samples using one or more loop filters to obtain last loop filter output samples. The one or more loop filters may for instance be an ALF, a SAO filter or an NN-based loop filter.

[0118] Step 4: store the last loop filter output samples in DPB 368.

[0119] Step 5: obtain an indicator value (the indicator value may be decoded from one or more syntax elements from the bitstream (e.g., from an SPS, PPS, PH or SEI message) or derived by external means).

[0120] Step 6: based on the value of the indicator value, do one of the following: i) output the post-reconstructed samples from the decoder, ii) output the last loop filter outputsamples from the decoder, iii) output both the post-reconstructed samples and the last loop filter output samples from the decoder.

[0121] Step 7: apply MP 105 with the samples outputted from the decoder as input.

[0122] FIG. 14 is a flowchart illustrating a process 1400 according to some embodiments which may be performed by device 103 (e.g., some or all of the steps may be performed by decoder 104). Process 1400 may begin in step sl402.

[0123] Step sl402 comprises deriving a first set of post-reconstruction samples (e.g., samples 870, samples 871, and / or samples 872) from a bitstream.

[0124] Step sl404 comprises filtering the first set of post-reconstruction samples using one or more loop filters (e.g., LF 811, LF 812, and / or LF 813) to produce a first set of last loopfilter output samples (e.g., samples 873), wherein the first set of last loop-filter output samples is not identical to the first set of post-reconstruction samples.

[0125] Step sl406 comprises storing the last loop-filter output samples in a DPB (e.g., DPB 368).

[0126] Step sl408 comprises outputting the first set of post-reconstruction samples (e.g., a copy of the first set of post-reconstruction samples may be output).

[0127] Step sl410 comprises performing a machine vision task using the postreconstruction samples.

[0128] In some embodiments, outputting the first set of post-reconstruction samples comprises providing the first set of post-reconstruction samples to a MP 105, which is configured to process the first set of post-reconstruction samples (e.g., the samples may be automatically sent directly to MP 105 or sent indirectly to MP 105 via, for example, a buffer (e.g., MPB, DPB), or the post-reconstruction samples may be stored in a buffer (e.g., DPB) and MP 105 may use an API to request the post-reconstruction samples).

[0129] FIG. 15 is a block diagram of an apparatus 1500 for performing any of the methods disclosed herein (i.e., for implementing encoder 102, decoder 104, device 103, and / or MP 105), according to some embodiments. As shown in FIG. 15, apparatus 1500 may comprise: processing circuitry (PC) 1502, which may include one or more processors (P) 1555 (e.g., one or more general purpose microprocessors and / or one or more other processors, such as anapplication specific integrated circuit (ASIC), field-programmable gate arrays (FPGAs), and the like), which processors may be co-located in a single housing or in a single data center or may be geographically distributed (e.g., apparatus 1500 may be a distributed computing apparatus comprising two or more computers or a monolithic computing apparatus consisting of a single computer); at least one network interface 1548 (e.g., a physical interface or air interface) comprising a transmitter (Tx) 1545 and a receiver (Rx) 1547 for enabling apparatus 1500 to transmit data to and receive data from other nodes connected to network 110 (e.g., an Internet Protocol (IP) network) to which network interface 1548 is connected (physically or wirelessly) (e.g., network interface 1548 may be coupled to an antenna arrangement comprising one or more antennas for enabling apparatus 1500 to wirelessly transmit / receive data); and a storage unit (a.k.a., “data storage system”) 1508, which may include one or more non-volatile storage devices and / or one or more volatile storage devices. In embodiments where PC 1502 includes a programmable processor, a computer readable storage medium (CRSM) 1542 may be provided. CRSM 1542 may store a computer program (CP) 1543 comprising computer readable instructions (CRI) 1544. CRSM 1542 may be a non-transitory computer readable medium, such as, magnetic media (e.g., a hard disk), optical media, memory devices (e.g., random access memory, flash memory), and the like. In some embodiments, the CRI 1544 of computer program 1543 is configured such that when executed by PC 1502, the CRI causes apparatus 1500 to perform steps described herein (e.g., steps described herein with reference to the flow charts). In other embodiments, apparatus 1500 may be configured to perform steps described herein without the need for code. That is, for example, PC 1502 may consist merely of one or more ASICs. Hence, the features of the embodiments described herein may be implemented in hardware and / or software.

[0130] Summary of Various Embodiments

[0131] Al. A method (1400) for processing an encoded picture, the method comprising: deriving (sl402) a first set of post-reconstruction samples (870, 871, 872) from a bitstream (118); filtering (si 404) the first set of post-reconstruction samples using one or more loop-filters (811, 812, 813) to produce a first set of last loop-filter output samples (873), wherein the first set of last loop-filter output samples (873) is not identical to the first set of post-reconstruction samples (870, 871, 872); storing (sl406) the first set of last loop-filter output samples (873) in a decoded picture buffer, DPB (368); outputting (si 408) the first set of post-reconstructionsamples; performing (sl410) a machine vision task using the first set of post-reconstruction samples.

[0132] A2. The method of embodiment Al, further comprising outputting the first set of last loop-filter output samples.

[0133] A3. The method of embodiment Al or A2, wherein the first set of postreconstruction samples is a decoded picture.

[0134] A4. The method of any one of embodiments A1-A3, wherein filtering the first set of post-reconstruction samples using one or more loop-filters comprises filtering the first set of post-reconstruction samples using a neural network loop-filter.

[0135] A5. The method of any one of embodiments A1-A4, wherein the machine vision task comprises usage of a neural network.

[0136] A6. The method of any one of embodiments A1-A5, wherein the first set of postreconstruction samples are samples of a particular type of pictures (I or P or B).

[0137] A7. The method of any one of embodiments A1-A6, further comprising obtaining the first set of last loop-filter output samples, wherein the machine vision task is performed using the post-reconstruction samples and the first set of last loop-filter output samples.

[0138] A7.1. The method of claim A7, further comprising post-filtering the first set of last loop-filter output samples to produce post-filtered output samples, wherein the post-filtered output samples are not used in the performance of the machine vision task.

[0139] A8. The method of any one of embodiments A1-A7.1, further comprising: deriving a second set of post-reconstruction samples from the bitstream; filtering the second set of post-reconstruction samples using the one or more loop-filters to produce a second set of last loop-filter output samples, wherein the second set of last loop-filter output samples is not identical to the second set of post-reconstruction samples; storing the second set of last loopfilter output samples in the DPB; and outputting the second set of post-reconstruction samples, wherein the machine vision task is performed using the first set of post-reconstruction samples and the second set of post-reconstruction samples.

[0140] A9. The method of embodiment A8, wherein the first set of post-reconstruction samples corresponds to a first portion of a partially decoded picture, and the second set of postreconstruction samples corresponds to a second portion of the partially decoded picture.

[0141] A10. The method of any one of embodiments A1-A9, wherein outputting the first set of post-reconstruction samples comprises providing the first set of post-reconstruction samples to a media processor (105) configured to perform the machine vision task using the first set of post-reconstruction samples.

[0142] Al l. The method of embodiment A10, wherein the method is performed by a decoder (104) and the media processor (105) is a component of the decoder (104).

[0143] A12. The method of any one of embodiments Al-Al l, wherein the first set of post-reconstruction samples correspond to a first picture.

[0144] Al 3. The method of any one of embodiments Al -Al 1, wherein the first set of post-reconstruction samples correspond to a portion of a picture.

[0145] A14. The method of any one of embodiments A1-A13, wherein filtering (sl006) the first set of post-reconstruction samples using one or more loop-filters (811, 812, 813) to produce the last loop-filter output samples (873) comprises: filtering the first set of postreconstruction samples using a first loop-filter (811) to produce first filtered samples (871); filtering the first filtered samples (871) using a second loop-filter (812) to produce second filtered samples (872); and filtering the second filtered samples (871) using a third loop-filter (813) to produce the last loop-filter output samples (873).

[0146] A15. The method of embodiment A14, further comprising: providing to a media processor (105) the first filtered samples (871); and / or providing to the media processor (105) the second filtered samples (872), wherein the media processor is configured to perform the machine vision task using the first set of post-reconstruction samples and at least one of the first filtered samples or the second filtered samples.

[0147] A16. The method of any one of embodiments A1-A15, wherein deriving (sl002) the first set of post-reconstruction samples (871, 872) comprises obtaining initial postreconstruction samples (870) and filtering the initial post-reconstruction samples using one or more filters (811, 812) to produce the first set of post-reconstruction samples (871, 872).

[0148] A17. The method of any one of embodiments A1-A15, wherein deriving (sl002) the first set of post-reconstruction samples (871, 872) comprises obtaining initial postreconstruction samples (870) and filtering the initial post-reconstruction samples using a first filter (811) to produce the first set of post-reconstruction samples (871, 872).

[0149] A18. The method of embodiment A17, wherein filtering (sl006) the first set of post-reconstruction samples using one or more loop-filters (812, 813) to produce the last loopfilter output samples (873) comprises: filtering the first set of post-reconstruction samples using a second loop-filter (812) to produce second filtered samples (872); and filtering the second filtered samples (872) using a third loop-filter (813) to produce the last loop-filter output samples (873).

[0150] A19. The method of any one of embodiments A1-A15, wherein deriving (sl002) the first set of post-reconstruction samples (871, 872) comprises: obtaining initial postreconstruction samples (870); filtering the initial post-reconstruction samples using a first filter (811) to produce first filtered samples (871); and filtering the first filtered samples using a second filter (812) to produce second filtered samples (872), wherein the first set of postreconstruction samples are the second filtered samples (872).

[0151] A20. The method of embodiment Al 9, further comprising: providing to a media processor (105) the initial post-reconstruction samples (870); and / or providing to the media processor (105) the first filtered samples (871), wherein the media processor is configured to perform the machine vision task using the first set of post-reconstruction samples and at least one of the first filtered samples or the second filtered samples.

[0152] A21. The method of any one of embodiments A1-A20, wherein the first set of post-reconstruction samples comprises samples from more than one picture.

[0153] A22. The method of any one of embodiments A1-A21, wherein the method further comprises providing to a media processor (105) both the first set of post-reconstruction samples and the last loop-filter output samples, and the media processor is configured to perform the machine vision task using the first set of post-reconstruction samples and the last loop-filter output samples.

[0154] A23. The method of any one of embodiments A1-A22, wherein the method is performed by a decoder (104) coupled to a media processor (105), and the first set of postreconstruction samples is output from the decoder to the media processor.

[0155] A24. The method of any one of embodiments A1-A23, wherein the one or more loop-filters comprises a deblocking filter, an ALF, a SAO filter, a bilateral filter, and / or an NN- based loop-filter.

[0156] A25. The method of embodiment A10 and any embodiment dependent on embodiment A10, wherein all or parts of the bitstream, and / or parameter values derived from the bitstream, are inputs to the media processor in addition to the post-reconstructed samples.

[0157] A26. The method of any of the previous embodiments, further comprising: deriving an indicator value from one or more syntax elements in the bitstream or by external means, the indicator value specifying the first set of post-reconstruction samples.

[0158] A27. The method of embodiment A26, further comprising, in response to the indicator value having a first value, outputting the first set of post-reconstruction samples.

[0159] A28. The method of embodiment A26, further comprising, in response to the indicator value having a second value, outputting the first set of post-reconstruction samples and also outputting the last loop-filter output samples.

[0160] A29. The method of any one of the previous embodiments, wherein performing the machine vision task using the first set of post-reconstruction sample comprises: performing an object detection task to detect an object; performing an object tracking task to track the object; and / or performing an event detection task to detect an event.

[0161] Bl. A computer program (1543) comprising instructions (1544) which when executed by processing circuitry (1502) of an apparatus (1500) causes the apparatus to perform the method of any one of the above embodiments.

[0162] B2. A carrier containing the computer program of embodiment Bl, wherein the carrier is one of an electronic signal, an optical signal, a radio signal, and a computer readable storage medium (1542).

[0163] Cl. An apparatus (1500) configured to perform the method of any one of the above embodiments.

[0164] While some of the terminology in this disclosure is described in terms of VVC, the embodiments of this disclosure also apply to any existing or future codec, which may use a different, but equivalent terminology.

[0165] While various embodiments are described herein, it should be understood that they have been presented by way of example only, and not limitation. Thus, the breadth and scope of this disclosure should not be limited by any of the above-described exemplary embodiments. Moreover, any combination of the above-described elements in all possible variations thereof is encompassed by the disclosure unless otherwise indicated herein or otherwise clearly contradicted by context.

[0166] Additionally, while the processes described above and illustrated in the drawings are shown as a sequence of steps, this was done solely for the sake of illustration. Accordingly, it is contemplated that some steps may be added, some steps may be omitted, the order of the steps may be re-arranged, and some steps may be performed in parallel.

Claims

CLAIMS1. A method (1400) for processing an encoded picture, the method comprising: deriving (sl402) a first set of post-reconstruction samples (870, 871, 872) from a bitstream (118); filtering (si 404) the first set of post-reconstruction samples using one or more loop-filters (811, 812, 813) to produce a first set of last loop-filter output samples (873), wherein the first set of last loop-filter output samples (873) is not identical to the first set of post-reconstruction samples (870, 871, 872); storing (sl406) the first set of last loop-filter output samples (873) in a decoded picture buffer, DPB (368); outputting (si 408) the first set of post-reconstruction samples; performing (sl410) a machine vision task using the first set of post-reconstruction samples.

2. The method of claim 1, further comprising outputting the first set of last loop-filter output samples.

3. The method of claim 1 or 2, wherein the first set of post-reconstruction samples is a decoded picture.

4. The method of any one of claims 1-3, wherein filtering the first set of postreconstruction samples using one or more loop-filters comprises filtering the first set of postreconstruction samples using a neural network loop-filter.

5. The method of any one of claims 1-4, wherein the machine vision task comprises usage of a neural network.

6. The method of any one of claims 1-5, wherein the first set of post-reconstruction samples are samples of a particular type of pictures (I or P or B).

7. The method of any one of claims 1-6, further comprising obtaining the first set of last loop-filter output samples, wherein the machine vision task is performed using the postreconstruction samples and the first set of last loop-filter output samples.

8. The method of claim 8, further comprising post-filtering the first set of last loop-filter output samples to produce post-filtered output samples, wherein the post-filtered output samples are not used in the performance of the machine vision task.

9. The method of any one of claims 1-8, further comprising: deriving a second set of post-reconstruction samples from the bitstream; filtering the second set of post-reconstruction samples using the one or more loop-filters to produce a second set of last loop-filter output samples, wherein the second set of last loopfilter output samples is not identical to the second set of post-reconstruction samples; storing the second set of last loop-filter output samples in the DPB; and outputting the second set of post-reconstruction samples, wherein the machine vision task is performed using the first set of post-reconstruction samples and the second set of post-reconstruction samples.

10. The method of claim 9, wherein the first set of post-reconstruction samples corresponds to a first portion of a partially decoded picture, and the second set of post-reconstruction samples corresponds to a second portion of the partially decoded picture.

11. The method of any one of claims 1-10, wherein outputting the first set of postreconstruction samples comprises providing the first set of post-reconstruction samples to a media processor (105) configured to perform the machine vision task using the first set of postreconstruction samples.

12. The method of claim 11, wherein the method is performed by a decoder (104) and the media processor (105) is a component of the decoder (104).

13. The method of any one of claims 1-12, wherein the first set of post-reconstruction samples correspond to a first picture.

14. The method of any one of claims 1-12, wherein the first set of post-reconstruction samples correspond to a portion of a picture.

15. The method of any one of claims 1-14, wherein filtering (si 006) the first set of post-reconstruction samples using one or more loop-filters (811, 812, 813) to produce the last loop-filter output samples (873) comprises: filtering the first set of post-reconstruction samples using a first loop-filter (811) to produce first filtered samples (871); filtering the first filtered samples (871) using a second loop-filter (812) to produce second filtered samples (872); and filtering the second filtered samples (871) using a third loop-filter (813) to produce the last loop-filter output samples (873).

16. The method of claim 15, further comprising: providing to a media processor (105) the first filtered samples (871); and / or providing to the media processor (105) the second filtered samples (872), wherein the media processor is configured to perform the machine vision task using the first set of post-reconstruction samples and at least one of the first filtered samples or the second filtered samples.

17. The method of any one of claims 1-16, wherein deriving (sl002) the first set of post-reconstruction samples (871, 872) comprises obtaining initial post-reconstruction samples (870) and filtering the initial post-reconstruction samples using one or more filters (811, 812) to produce the first set of post-reconstruction samples (871, 872).

18. The method of any one of claims 1-16, whereinderiving (sl002) the first set of post-reconstruction samples (871, 872) comprises obtaining initial post-reconstruction samples (870) and filtering the initial post-reconstruction samples using a first filter (811) to produce the first set of post-reconstruction samples (871, 872).

19. The method of claim 18, wherein filtering (si 006) the first set of post-reconstruction samples using one or more loop-filters (812, 813) to produce the last loop-filter output samples (873) comprises: filtering the first set of post-reconstruction samples using a second loop-filter (812) to produce second filtered samples (872); and filtering the second filtered samples (872) using a third loop-filter (813) to produce the last loop-filter output samples (873).

20. The method of any one of claims 1-16, wherein deriving (si 002) the first set of postreconstruction samples (871, 872) comprises: obtaining initial post-reconstruction samples (870); filtering the initial post-reconstruction samples using a first filter (811) to produce first filtered samples (871); and filtering the first filtered samples using a second filter (812) to produce second filtered samples (872), wherein the first set of post-reconstruction samples are the second filtered samples (872).

21. The method of claim 20, further comprising: providing to a media processor (105) the initial post-reconstruction samples (870); and / or providing to the media processor (105) the first filtered samples (871), wherein the media processor is configured to perform the machine vision task using the first set of post-reconstruction samples and at least one of the first filtered samples or the second filtered samples.

22. The method of any one of claims 1-21, wherein the first set of post-reconstruction samples comprises samples from more than one picture.

23. The method of any one of claims 1-22, wherein the method further comprises providing to a media processor (105) both the first set of post-reconstruction samples and the last loop-filter output samples, and the media processor is configured to perform the machine vision task using the first set of post-reconstruction samples and the last loop-filter output samples.

24. The method of any one of claims 1-23, wherein the method is performed by a decoder (104) coupled to a media processor (105), and the first set of post-reconstruction samples is output from the decoder to the media processor.

25. The method of any one of claims 1-24, wherein the one or more loop-filters comprises a deblocking filter, an ALF, a SAO filter, a bilateral filter, and / or an NN-based loopfilter.

26. The method of claim 11 and any claim dependent on claim 11, wherein all or parts of the bitstream, and / or parameter values derived from the bitstream, are inputs to the media processor in addition to the post-reconstructed samples.

27. The method of any one of the previous claims, further comprising: deriving an indicator value from one or more syntax elements in the bitstream or by external means, the indicator value specifying the first set of post-reconstruction samples.

28. The method of claim 27, further comprising, in response to the indicator value having a first value, outputting the first set of post-reconstruction samples.

29. The method of claim 27, further comprising, in response to the indicator value having a second value, outputting the first set of post-reconstruction samples and also outputting the last loop-filter output samples.

30. The method of any one of the previous claims, wherein performing the machine vision task using the first set of post-reconstruction sample comprises: performing an object detection task to detect an object; performing an object tracking task to track the object; and / or performing an event detection task to detect an event.

31. A computer program (1543) comprising instructions (1544) which when executed by processing circuitry (1502) of an apparatus (1500) causes the apparatus to perform the method of any one of the above claims.

32. A carrier containing the computer program of claim 31, wherein the carrier is one of an electronic signal, an optical signal, a radio signal, and a computer readable storage medium (1542).

33. An apparatus (1500) for processing an encoded picture, the apparatus being configured to: derive (sl402) a first set of post-reconstruction samples (870, 871, 872) from a bitstream (H8); filter (si 404) the first set of post-reconstruction samples using one or more loop-filters (811, 812, 813) to produce a first set of last loop-filter output samples (873), wherein the first set of last loop-filter output samples (873) is not identical to the first set of post-reconstruction samples (870, 871, 872); store (sl406) the first set of last loop-filter output samples (873) in a decoded picture buffer, DPB (368); output (si 408) the first set of post-reconstruction samples; perform (s 1410) a machine vision task using the first set of post-reconstruction samples.

34. The apparatus of claim 33, wherein the apparatus is further configured to perform the method of any one of claims 2-30.

35. A method (1600) for processing an encoded picture, the method comprising:a decoder (104) deriving (si 602) a middle signal (202) from a bitstream (118); the decoder deriving (si 604) a final output signal (204) using the middle signal (202); the decoder outputting (si 606) the middle signal (202); the decoder outputting (si 608) the final output signal (204); and performing (s 1610) a machine vision task using the middle signal (202) output by the decoder.

36. The method of claim 35, wherein deriving the final output signal (204) using the middle signal (202) comprises filtering the middle signal using a post-filter.

37. The method of claim 36, wherein the method further comprises providing the final output signal to a display device for displaying the final output signal.

38. The method of any one of claims 35-37, wherein the middle signal comprises a first set of loop-filter output samples (873), and the method further comprises storing the first set of loop-filter output samples (873) in a decoded picture buffer, DPB (368).

39. The method of any one of claims 35-38, wherein the machine vision task is performed using the middle signal (202) output by the decoder and the final output signal (204) output by the decoder.

Images