Coding and decoding video transmission concepts

Abstract

A video encoder, decoder, system and method for encoding and decoding video are presented. A video encoder for encoding a video is configured to encode frames of the video into a data stream using intra-frame coding and differential inter-frame coding; measuring quality loss caused by coding and decoding loss; and signaling information about the quality loss as side information in the data stream. Further, there is provided a video decoder configured to decode frames of a video from a data stream using intra decoding and differential inter decoding wherein the video is encoded into the data stream with a codec loss; estimating information of a quality loss caused by a codec loss; and outputting, at an output interface of the video decoder, metadata revealing information about the image loss, and / or modifying a reconstructed version of the video.

Description

Technical Field

[0001] The following describes different inventive embodiments and aspects relating to video encoding / decoding and the measurement of quality loss, specifically relating to frames of video encoded / decoded using intra-frame encoding / decoding and differential inter-frame encoding / decoding. Background Technology

[0002] Many technological applications require high video quality, such as optical measurement equipment, photo and video editing, and medical imaging. For medical diagnosis, the visual evaluation of medical images is crucial. To avoid false positives or false negatives in diagnosis, medical images must be presented on the monitor at the highest possible quality.

[0003] Typically, such presentations use either uncompressed images or images with mathematically lossless compression. In addition, the display must meet specific quality standards, such as IEC 62563-2 (“Medical electrical equipment – ​​Medical image display systems – Part 2: Acceptance and constancy tests formedical image displays”)[2] or follow the rules in AAPM Online Report No. 03 (“Assessment of Display Performance for Medical Imaging Systems”)[3].

[0004] One issue arises when a computer displays high-resolution images. When the computer and monitor are directly connected (threaded connection), this may not be a problem for some resolutions (e.g., displays with resolutions up to 4K) because uncompressed transmission occurs between the computer and monitor (see the HDMI specification). However, at 8K resolutions, the connection between the monitor and computer may already be using compression (see DSC compression, https: / / vesa.org / vesa-display-compression-codecs / dsc / ), which can negatively impact video quality.

[0005] Further issues arise when using the monitor remotely or from an even greater distance (e.g., via a bandwidth-limited Ethernet network such as 1G Ethernet). In such cases, it may be necessary to connect a data receiving computer to the monitor, which receives and buffers the received data, decodes it, and presents it to the monitor along with the user interface at the monitor's refresh rate.

[0006] Another solution might be to remotely transmit the full-screen image in real time at the monitor's refresh rate. This could only be achieved with higher compression. For example, a 3840 x 2160 monitor with 60 frames per second (fps), 10 bits per color component, and 3 components might require 3840 x 2160 x 60 x 10 x 3 = 15 Gbit / s transmission lines. For a 1G Ethernet interface, this might require 15:1 compression, which is generally not mathematically lossless.

[0007] Therefore, an improved concept is needed to achieve a better balance between transmission speed and ensuring a specific video quality. This is achieved through the subject matter of the independent claims of this application.

[0008] Other embodiments of the invention are defined by the subject matter of the dependent claims of this application. Summary of the Invention

[0009] According to one aspect, a video encoder for encoding video is configured to: encode frames of video into a data stream using intra-frame encoding and decoding and differential inter-frame encoding and decoding; measure the quality loss caused by encoding and decoding losses; and signal information about the quality loss as side information in the data stream.

[0010] In order to utilize differential inter-frame encoding and decoding, the video encoder at the transmitting end and the video decoder at the receiving end can use frame buffers to store the data that has been sent / received and reconstructed.

[0011] This application is based on the fact that video quality can be iteratively improved through successive differential inter-frame encoding and decoding steps, for example, for frames with small variations, such as static views. The improvement in video quality can be so gradual that the human eye may have difficulty discerning when the video quality reaches a sufficient level or a maximum (or stable or lossless quality). Since the video encoder has simultaneous access to the original video and its encoded version (or, in other words, a reconstructable version), it can use the original video as a reference to determine the quality loss resulting from the video encoding and decoding process, or to determine when the encoded video has reached sufficient quality. Because the video encoder is capable of differential inter-frame encoding and decoding, the frames (or portions of frames) encoded and decoded by inter-frame encoding and decoding also allow for quality loss estimation on the decoder side. For example, differential inter-frame encoding and decoding can represent the difference between these portions of a video / image and any previous or immediately preceding image, without any motion-compensated displacement between the two images / frames. This allows for both time-efficient and low-complexity encoding and decoding processes and easily implemented decoder-side quality loss estimation. Therefore, this paper describes two approaches: one where the encoder determines information about the quality loss and notifies the decoder via a data stream; and another where the decoder makes an estimate—possibly without any quality loss-specific side information from the encoder. Using tools that allow determining when video quality is sufficient, high-quality video data transmission can, for example, be extended across multiple frames (e.g., these frames facilitate iterative differential inter-frame encoding / decoding to improve video quality). Thus, video quality may no longer be limited by transmission speed. For example, for a video transmission system between a computer and a monitor on a transmission line with limited display frame rate and bandwidth, a compression codec (such as JPEG XS version 3) can be used that is capable of improving the quality of static scenes or images frame by frame, e.g., until lossless image quality is achieved. To enable reliable diagnosis based on encoded video (e.g., medical images), the image quality can be continuously monitored, and if a specific predefined quality or lossless quality is achieved, the viewer can be signaled (e.g., by signaling information about quality loss to the video decoder, which can then forward that information to the user). This allows for avoiding erroneous diagnoses caused by the transmission system. Furthermore, the requirements for the transmission system and codec can be reduced.

[0012] According to an aspect applicable to the determination of quality loss on the encoder side, a video decoder for decoding video is configured to: decode frames of video from a data stream using intra-frame decoding and differential inter-frame decoding; and derive information about the quality loss caused by encoding / decoding loss from side information of the data stream, wherein the video is encoded into the data stream with encoding / decoding loss.

[0013] The video decoder is capable of intra-frame decoding and differential inter-frame decoding, thus ensuring compatibility with the data stream encoded by the aforementioned video encoder. Therefore, the video encoder can perform encoding and decoding that progressively improves video quality and measure information about the quality loss resulting from encoding and decoding. Since the video decoder can derive information about the quality loss encoded into the video stream, it can forward this information to the user. For example, the video decoder can notify the user by modifying the decoded video (e.g., from red to green shading), and / or provide signals separate from the video, such as signals that can be received by acoustic, optical, or haptic feedback devices. The video decoder does not necessarily have to directly notify the user. Alternatively, the video decoder can provide (e.g., save) an image of the video when it derives from the information about the quality loss that sufficient video quality has been achieved. Information about the quality loss allows the video codec to continuously improve the encoding and decoding quality across multiple frames, which can improve the balance between video quality and data transmission.

[0014] According to another aspect of this application, a method for quality loss estimation on the decoder side is provided, in which a video decoder for decoding video is configured to: decode frames of video from a data stream using intra-frame decoding and differential inter-frame decoding, the video being encoded into the data stream with encoding / decoding loss; estimate information about the quality loss caused by the encoding / decoding loss; and output metadata revealing information about image loss at the output interface of the video decoder, and / or modify a reconstructed version of the video obtained by decoding the data stream to provide an indication of the stage during which the image loss meets a certain criterion for the reconstructed version of the video.

[0015] Because differential inter-frame decoding allows for iterative improvements in video quality, and the decoding of frames for such differential inter-frame encoding itself provides information related to video quality, estimates can be readily obtained without much effort. One example is the measurement of the energy of the signal in differential inter-frame encoding. This energy can be compared to the energy of previously reconstructed images in the frame buffer. Another example in codecs operating at a constant bit rate is measuring the padding zeros in the data packets of the bitstream. In variable bitstream encoding and decoding, the number of transmitted bits reduced from the maximum allowed bit rate can be used as the basis for determining a quality metric. Such measurements can indicate a loss in video quality without requiring the original video as a reference. In cases of improved video quality (e.g., due to smaller or reduced differences between subsequent frames), such properties of multiple frames can be processed to obtain values ​​that allow for an estimate of whether the quality loss has increased or decreased. It has been recognized that such video decoders can even estimate information about quality loss from the data stream of encoders that are unable to measure concurrent signaling information about quality loss (in other words, video encoders not specifically designed for this purpose). Therefore, the complexity of the video encoder can be reduced (or maintained), and existing video encoders—or in other words, bitstreams of such encoders—can be used.

[0016] While the embodiments of this application are also applicable to codecs using typical full-frame IP (where, within a group of images, one frame is intra-frame coded (I), and subsequent frames are predictively coded from past frames (P)) or IPB coding schemes (where, within a group of images, one frame is intra-frame coded (I), some frames are predictively coded from past frames (P), and some frames are bidirectionally predictively coded from past and future frames (B)), i.e., for bandwidth-constrained transmissions, a quality-degraded codec appears in the video sequence whenever a new I-frame is transmitted, this embodiment is most suitable for codecs such as JPEG XS version 3, where intra-frame coding is performed only on the small, changing portions of the image in each frame, while the rest is differentially coded (so-called progressive refresh). This ensures that, on the one hand, transmission errors are corrected over time, and on the other hand, the advantages of differential coding can be taken advantage of. More preferably, intra-frame coding is performed on only a small portion of the image in each frame / image, while differential inter-frame coding is performed on the main portion of the image. Attached Figure Description

[0017] Embodiments of the present invention will now be described with reference to the accompanying drawings, wherein:

[0018] Figure 1a shows a schematic diagram of an embodiment of a video encoder used for encoding video.

[0019] Figure 1b shows a schematic diagram of one embodiment of a video decoder for decoding video.

[0020] Figure 2a shows a schematic diagram of a frame divided into N subframes.

[0021] Figure 2b shows a schematic diagram of a frame divided into strip-shaped subframes.

[0022] Figure 3a shows a schematic diagram of the first decoded frame of the video.

[0023] Figures 3b to 3d show schematic diagrams of the second, third, and fourth frames.

[0024] Figure 4a shows a schematic diagram of the first decoded frame of the video.

[0025] Figure 4b shows a schematic diagram of the second decoded frame of the video.

[0026] Figure 4c shows a schematic diagram of the third decoded frame of the video.

[0027] Figure 4d shows a schematic diagram of the fourth decoded frame of the video.

[0028] Figure 5a shows a schematic diagram of one embodiment of a video decoder used for decoding video.

[0029] Figure 5b shows a schematic diagram of the relationship between frame number and peak signal-to-noise ratio.

[0030] Figure 6a shows a schematic diagram of a first system including a video encoder and a video decoder.

[0031] Figure 6b shows a schematic diagram of a second system, which includes a video encoder configured to encode video frames into a data stream using intra-frame encoding and differential inter-frame encoding and decoding, and a video decoder.

[0032] Figure 6c shows a schematic diagram of a third system, which includes a video decoder and a display for displaying the reconstructed video obtained by the video decoder through decoding.

[0033] Figure 7a shows a flowchart of a method for encoding video.

[0034] Figure 7b shows a flowchart of a method for decoding video.

[0035] Figure 8 A flowchart of a method for decoding video is shown. Detailed Implementation

[0036] In the following description, even if they appear in different figures, the same or equivalent elements or elements with the same or equivalent functions are indicated by the same or equivalent reference numerals.

[0037] In the following description, numerous details are set forth to explain embodiments of the invention more fully. However, it will be apparent to those skilled in the art that embodiments of the invention can be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring embodiments of the invention. Furthermore, unless otherwise specifically stated, features of the different embodiments described herein can be combined with each other.

[0038] In the following text, the term "encoding / decoding" will be used to address encoding and decoding. Therefore, the processes described in the context of encoding / decoding can be applied to both encoding and decoding. However, unless otherwise stated, the use of the term "encoding" does not mean that it is not applicable to "decoding" (and vice versa).

[0039] Figure 1a illustrates a schematic diagram of one embodiment of a video encoder 10 for encoding video 12. The video encoder 10 is configured to encode frames 14 of video 12 (e.g., frames or images with pure differential inter-frame coding, or frames or images with a small portion of intra-frame coding used for progressive decode refresh (GDR)) into data stream 16 using intra-frame coding and differential inter-frame coding (e.g., simultaneously, for the same frame, for example, not only intra-frame coding) and to measure the quality loss 18 caused by coding and decoding losses. The video encoder 10 is configured to signal information about the quality loss 18 in the data stream 16 as side information 20.

[0040] Differential inter-frame coding and decoding can be, or can include, temporal differential coding and decoding (TDC), for example, as defined in image and / or video coding and decoding standards, such as the JPEG XS standard (e.g., ISO / IEC 21122-1). Therefore, any differential inter-frame coding and decoding disclosed herein can be, or can include, temporal differential coding and decoding. For example, differential inter-frame coding and decoding can include using (e.g., subtracting on the encoder side and adding on the decoder side) the difference between sample values ​​(e.g., pixel values, such as one or more of luminance and chrominance values) of the current frame 14 (or a portion thereof) and a previously encoded frame 14 (or a portion thereof), and in a manner without relative displacement (e.g., without motion-compensated frame-by-frame inter-frame displacement, e.g., for sample values ​​at the same position in the current and previous frames).

[0041] Furthermore, differential inter-frame coding and decoding (e.g., TDC) can be performed partially (e.g., for a portion of the frame) or completely (e.g., for the entire frame) for frame 14. Similarly, intra-frame coding and decoding can be performed partially (e.g., for a portion of the frame) or completely for frame 14. Frame 14 can be encoded and decoded using a combination of partial differential inter-frame coding and partial intra-frame coding and decoding.

[0042] For multiple frames 14, each frame may be encoded and decoded using only one codec type (e.g., using only differential inter-frame codec for one set of frames and only intra-frame codec for another different set of frames), or each frame may be encoded and decoded partially (e.g., all frames are encoded and decoded partially using differential inter-frame codec and partially using intra-frame codec), or a combination thereof (e.g., some frames are encoded and decoded using only one codec type, and some frames are encoded and decoded partially using two codec types).

[0043] In the first example, frame 14 (e.g., the first frame) is fully encoded using intra-frame coding (e.g., without differential inter-frame coding), and subsequent frames of 14 are fully encoded using differential inter-frame coding (e.g., without intra-frame coding). In the second example, frame 14 (e.g., the first frame) is encoded using both differential inter-frame coding and intra-frame coding (e.g., for different portions of frame 14), and subsequent frames are also encoded using both differential inter-frame coding and intra-frame coding (e.g., but for different portions of frame 14). However, the first and second examples are not mutually exclusive. The invention can employ any combination of partial or full coding (e.g., using only intra-frame coding or only differential inter-frame coding).

[0044] For example, the video encoder 10 may include: an encoding unit (or device) 22 configured to encode frame 14 of video 12 into data stream 16 using intra-frame encoding and differential inter-frame encoding (e.g., using only differential inter-frame encoding for the same frame or using differential inter-frame encoding and intra-frame encoding, e.g., simultaneously, e.g., using different encoding modes for different sub-frame portions of the same frame); a measurement unit (or device) 24 configured to measure the quality loss 18 caused by encoding and decoding losses; and a signaling unit 26 configured to signal information about the quality loss 18 in data stream 16 as side information 20. Units 22, 24, and 26 are shown as independent units. However, at least two (e.g., all) functions of the video encoder 10 may be performed by the same device (completely or partially). For example, the video encoder 10 may include a processing unit configured to perform at least one of encoding, measurement, and signaling. For example, video encoder 10 may be a video camera, optical measurement equipment, medical imaging equipment, server, monitor (e.g., a device that forwards data streams to a monitor's display), computer, smartphone, or tablet device, or may be part of such devices. Intra-frame encoding and decoding can be performed independently of any previous frame; for example, encoding and decoding of frame differences or subframe differences compared to previous frames is not required.

[0045] For example, video encoder 10 can be configured to perform differential inter-frame decoding for a first portion of frame 14 (e.g., including one or more subframe portions) and intra-frame encoding / decoding for a second portion of frame 14 (e.g., including one or more subframe portions). In another example, video encoder 10 can be configured to perform differential inter-frame decoding only for one or more (or all) frames 14 (or portions thereof). Video encoder 10 can be configured to perform intra-frame encoding / decoding of frames (e.g., the first frame 14 of the video, for example, for multiple frames 14, for example, in regular frames or time intervals).

[0046] Video encoder 10 (or a device including video encoder 10) can be configured to generate data packets 28 that include encoded video data 30 and side information 20. Side information 20 may be included in the header, metadata, or payload of data packets 28. In the example shown in Figure 1a, each data packet includes side information 20. However, as will be further described below, not every data packet needs to signal side information 20. For example, a single frame may use multiple data packets to signal, where only one data packet may require side information 20. Signaling side information 20 may not occur for every frame (e.g., every two frames or every three frames). Furthermore, signaling side information 20 may be conditional (e.g., signaling at the beginning of an image invariance phase or during an image invariance phase), and therefore does not necessarily need to be done for multiple frames and / or multiple data packets. Alternatively, the data stream may not be packaged.

[0047] Data stream 16 may include wireless data transmission, such as via a wireless router and / or cellular network. Data streams may also include wired transmissions, such as between a server and a user, or between a server and another server, or between local devices, such as between a monitor and a computer or camera. Furthermore, data streams may be transmitted between different parts of the same device, such as between a monitor's data receiver and the monitor's display.

[0048] Figure 1b illustrates a schematic diagram of one embodiment of a video decoder 40 for decoding video. The video decoder 40 can be configured to decode video encoded by any video encoder disclosed herein (e.g., video encoder 10 shown in Figure 1a).

[0049] The video decoder 40 is configured to decode frame 14 of video 12 (or differentially encoded image, or inter-frame decoding for one or more frames only without intra-frame decoding) from data stream 16 using intra-frame decoding and differential inter-frame decoding, and to derive information about the quality loss 18 caused by encoding and decoding losses when video 12 is encoded into data stream 16 from side information 20 of data stream 16.

[0050] For example, video decoder 40 may include: a decoding unit (or device) 42 configured to decode frame 14 of video 12 from data stream 16 using intra-frame decoding and differential inter-frame decoding (e.g., or differential inter-frame encoding / decoding only for one or more frames); and a derivation unit (or device) 44 configured to derive information about quality loss 18 caused by encoding / decoding loss from side information 20 of data stream 16, with which video 12 is encoded into data stream 16. Units 42 and 44 are shown as independent units. However, at least two (e.g., all) functions of video decoder 40 may be performed by the same device (at least partially). For example, video decoder 40 may include a processing unit configured to perform at least one of decoding and derivation. For example, video decoder 40 may be a video camera (e.g., for playing back recorded video), a monitor, a server, a computer, a smartphone, or a tablet device, or may be part of such devices.

[0051] Figure 2a shows a schematic diagram of an example of a frame 14 divided into N subframe portions (or subframe regions) 32. In the example shown in Figure 2a, frame 14 is divided into 16 subframe portions 32, where N=16. However, frame 14 may include any other number of subframe portions 32, for example, N is between 2 and 200, or N is between 32 and 120, or N=60. Subframe portions 32 may have the same or different shapes (e.g., rectangles, squares, or polygons) or sizes. For multiple frames 14, the number of subframe portions in each frame may be the same or different.

[0052] Figure 2b illustrates a schematic diagram of frame 14 divided into striped subframe portions (or subframe regions) 32. Frame 14 can have any number N striped subframe portions 32. Each subframe portion 32 extends from the top to the bottom of frame 14. Alternatively, each subframe portion 32 can extend from the left to the right of frame 14. That is, the stripes can extend vertically or horizontally. Encoding and decoding of frame 14 can include using progressive refresh (which may be referred to here as Progressive Decoding Refresh (GDR), but should be understood to generally apply to encoding and decoding, not just decoding), in units of one or more subframe portions 32, which will be described in more detail below. Striped subframe portions 32 can facilitate the use of progressive decoding refresh.

[0053] When frame 14 is encoded, quality loss may occur between the lossless version of video 12 (e.g., the video before encoding) and the encoded video 12 (i.e., the version reconstructed from the encoded and decoded data stream). The encoded data stream may contain less video information than the lossless version of video 12, which may cause or contribute to quality loss. The information in data stream 16 may include a quality metric parameter that monotonically increases as quality loss decreases; in other words, it can be a quality metric. Alternatively, the information in data stream 16 may include a quality metric parameter that monotonically decreases as quality loss decreases; in other words, it can be a loss metric.

[0054] Video encoder 10 can be configured to perform measurements and signaling at a spatial granularity corresponding to a frame or subframe portion. For example, video encoder 10 can be configured to perform measurements and signaling at a temporal granularity corresponding to the frame rate of video 12. For example, in the case of a frame rate of 30 frames per second, video encoder 10 can be configured to also perform measurements and signaling at 30 frames per second (or 30 Hz), or alternatively, in an example having 60 subframe portions 32 and spatial granularity corresponding to the subframe portions, video encoder 10 can be configured to perform measurements and signaling 60 times per frame (or 30 x 60 = 1800 times per second). Alternatively, video encoder 10 can be configured to perform measurements and signaling at a rate lower than the frame rate (e.g., half or one-third of the frame rate, i.e., every two or three frames). Video encoder 10 can be configured to perform measurements and / or signaling continuously (e.g., at the rate described herein) or conditionally (e.g., for frames belonging to the phase of image invariance, e.g., at the rate described herein).

[0055] Video decoder 40 can be configured to derive information about quality loss from the data stream at a spatial granularity corresponding to frame 14 or subframe portion 32. For example, video decoder 40 can be configured to derive information about quality loss from the data stream at the same spatial granularity as video encoder 10. Alternatively, video decoder 40 can be configured to derive information about quality loss from the data stream at a spatial granularity different from that of video encoder 10 (e.g., half the spatial granularity). Video decoder 40 can have any spatial granularity of video encoder 10 as described above.

[0056] The video encoder 10 can be configured to measure quality loss by determining the average of a per-pixel deviation measure between a lossless version of video 12 (e.g., lossless frame 14 or one or more lossless subframe portions 32 of frame 14) and a reconstructable version of video 12 reconstructable according to data stream 16 (e.g., reconstructed frame 14 or one or more reconstructed subframe portions 32 of frame 14).

[0057] Therefore, the video decoder 40 can derive information about the quality loss, which is formed by the average of the pixel-by-pixel deviation measure between the lossless version of the video and the reconstructable version that can be reconstructed from the data stream 16, or by the average of the pixel-by-pixel deviation measure between the lossless version of the video and the reconstructable version that can be reconstructed from the data stream 16.

[0058] For example, video encoder 10 can be configured to determine a deviation metric between the lossless version and the reconstructed version of frame 14 for each (or selected) pixel of frame 14. The reconstructed version of frame 14 can be (or based on) frame 14 or one or more portions of one or more frames 14 (e.g., subframe portion 32) reconstructed from encoded frame 14 in data stream 16.

[0059] For example, video encoder 14 can be configured to encode frame 14 using differential inter-frame coding and then determine the frame difference between the current frame and the previous (e.g., the immediately preceding) frame. Video encoder 14 can then be configured to reconstruct the current frame 14 based on this frame difference and the previous frame 14. However, the reconstructed and / or lossless version of video 12 is not limited to the entirety and a single frame 14. As will be further described below, the reconstructed and / or lossless version of video 12 can be formed from a portion of the first frame 14, such as a portion not refreshed using progressive decoding (e.g., using intra-frame coding) (e.g., subframe portion 32) (e.g., using a portion of the first frame 14 that undergoes differential inter-frame coding). The reconstructed and / or lossless version of video 12 may be additionally formed from a portion of a previous or subsequent second frame 14 (e.g., subframe portion 32), such as one or more portions of which progressive decoding refresh was not used in the first frame 14 but was used in the second frame 14 (e.g., subframe portion 32) (e.g., intra-frame coding was used in the first frame and inter-frame coding was used in the second frame 14).

[0060] A measure of bias can be based on the bias between frames of the reconstructed and lossless video in terms of sample values ​​and / or (e.g., transform coefficients assigned to pixels) in terms of linear, absolute, squared, or cubic differences. A measure of bias can also be based on the sum (or formed by) the sum of (or derived from) the differences (e.g., linear, absolute, squared, or cubic differences) of sample values ​​(e.g., at least one of luminance and chrominance values) between the lossless and reconstructed versions of frame 14 (or a portion of frame 14). In addition to or alternative to the sum of differences, an average, weighted average, or median can also be formed. A measure of bias can be based on or derived from the peak signal-to-noise ratio (PSNR).

[0061] Alternatively, the measure of the deviation can be determined in the transform domain (e.g., wavelet domain). For example, in the case of intra-frame encoding / decoding (e.g., for a portion of frame 14 when using progressive decoding refresh), the quantized transform coefficients of the original frame's transform can be encoded / decoded; in the case of differential inter-frame encoding / decoding, the quantized transform coefficients of the transform of the frame difference between the current frame and the previous frame can be encoded / decoded. The video encoder 10 can be configured to encode frame 14 in the transform domain and measure the quality loss by averaging (e.g., using an arithmetic mean, median, or weighted average) or summing the transform coefficients (or their absolute values) in the transform domain (e.g., by averaging all coefficients belonging to the unrefreshed region of the frame, e.g., the average absolute value or average square value of the transform coefficients). For example, the video encoder 10 can be configured to determine the square or absolute value of the coefficients of all (or part of) subframe portions 32 that are not refreshed in this frame (e.g., those encoded by differential inter-frame coding rather than intra-frame coding), and form the sum or average of the square or absolute values ​​of these coefficients.

[0062] The video encoder 10 can be configured to measure the quality loss of frame 14 by averaging all coefficients belonging to the unrefreshed area of ​​the current frame, for example, based on or using the average absolute value or average square value of the transform coefficients.

[0063] The video encoder 10 can be configured to signal information about quality loss in the form of or based on mean squared error (MSD) or mean absolute difference (MAD) values. The video decoder 40 can be configured to derive information about quality loss from the data stream 16 in the form of MSD or MAD values.

[0064] The video encoder 10 can be configured to signal information about quality loss as binary information with two states, where the first state corresponds to sufficient quality and the second state corresponds to insufficient quality; or, to signal information about quality loss using an integer or floating-point scale. The video decoder 40 can be configured to derive information about quality loss from the data stream 16 as binary information with two states, where the first state corresponds to sufficient quality and the second state corresponds to insufficient quality; or, to derive information about quality loss using an integer or floating-point scale. These two states can be signaled using binary flags or bits, where a value "1" indicates sufficient quality and "0" indicates insufficient quality (or vice versa). Integer or floating-point scales can indicate quality loss in any unit (e.g., from 1 to 1000), as an MSD value, as a MAD value, as decibels (e.g., for PSNR), as a ratio (e.g., a quotient or percentage) relative to a threshold (e.g., between adequate and inadequate quality), or as a ratio (e.g., a quotient or percentage) relative to a lossless version of the video or relative to a previous (e.g., immediately preceding) decoded frame. The threshold used to determine adequate or inadequate quality (e.g., for quality loss as binary information with two states) can be defined as an absolute value, or as a value relative to a lossless version of video 12 or relative to a previous (e.g., immediately preceding) decoded frame. The threshold can be defined in any unit (e.g., "900") as an MSD value, MAD value, or decibel (e.g., PSNR).

[0065] Video encoder 10 can be configured to perform differential inter-frame encoding / decoding by subtracting the current frame 14 from previously encoded / decoded frames 14 (e.g., in terms of encoding and rendering order) to obtain a difference and encoding / decoding that difference into the data stream (e.g., using quantization, e.g., without quantization; e.g., forming a difference between quantized values, e.g., a difference between quantized transform coefficients). The difference can be obtained in the spatial domain or the transform domain (or wavelet domain). The difference can be, or can be based on, the difference between sample values ​​(in the spatial domain) and / or (e.g., the difference between quantized transform coefficients).

[0066] The video decoder 40 can be configured to perform differential inter-frame decoding by decoding the difference (e.g., quantized or unquantized) between the current frame 14 and previously decoded (e.g., in terms of decoding and rendering order) frames 14 from the data stream 16, and adding the difference (e.g., quantized or unquantized) between the previously decoded (e.g., in terms of decoding and rendering order) frames 14 (e.g., where the difference can be the difference between quantized transform coefficients). The video decoder 40 can also be configured to decode the quantized difference between the current frame 14 and the immediately preceding frame 14 in the transform domain from the data stream 16, perform addition in the transform domain to obtain the transform of the current frame 14, and re-transform the transform.

[0067] The video encoder 10 can be configured to subtract between the current frame 14 (or a portion thereof) and a previously encoded / decoded frame 14 (or a portion thereof) in a mutually displacement-free manner (e.g., without motion-compensated inter-frame displacement). The video decoder 40 can be configured to add between the (e.g., quantized or unquantized) difference and the previously decoded frame 14 in a mutually displacement-free manner (e.g., without motion-compensated inter-frame displacement).

[0068] For example, referring to FIG1a, when encoding the current frame 14c, the video encoder 10 can be configured to perform a subtraction between the current frame 14c and the previously encoded / decoded frame 14b (or frame 14a). For example, the first transform coefficient or sample value of the previous frame 14b is subtracted from the first transform coefficient or sample value of the upper left corner of the current frame 14c. Subsequently, the second transform coefficient or sample value to the right of the first transform coefficient of the previous frame 14b is subtracted from the second transform coefficient or sample value to the right of the first transform coefficient of the current frame 14c. Similarly, all (or selected portions, such as only unrefreshed portions) of the transform coefficients or sample values ​​of frames 14c and 14b can be subtracted.

[0069] As a result, it is unnecessary to determine or send information related to the displacement (e.g., motion vector) of frames 14b and 14c. Differential inter-frame encoding and decoding is compatible with the use of frame buffers, as defined in the standard JPEG XS version 3. Furthermore, differential inter-frame encoding and decoding without displacement allows for low-complexity and fast implementation of frame buffers, which store the values ​​of the current frame (e.g., transform coefficients or sample values), and these values ​​can be easily updated based on subtraction. Such frame buffers can be used in at least one of the video encoder 10 and the video decoder 40.

[0070] Video encoder 10 can be configured to sequentially encode frame 14 using an encoding / decoding order consistent with the frame's presentation order. Video decoder 40 can be configured to sequentially decode frames using an encoding / decoding order consistent with the frame's presentation order. Therefore, video encoder 10 and / or video decoder 40 may not skip frames or go back in the presentation order (e.g., unlike an MPEG encoder, they cannot encode / decode frames in a different frame order than the presentation order). As a result, the storage size for the data used for frames can be reduced, for example, only for storing one or more previous frames. Such storage has low complexity and allows for fast processing of video 12.

[0071] Video encoder 10 can be configured to perform differential inter-frame encoding / decoding by subtracting the current frame from the immediately preceding frame to obtain a difference and encoding / decoding that difference into the data stream (e.g., with quantization, e.g., without quantization; e.g., the difference between quantized values, e.g., the difference between quantized transform coefficients). Video decoder 40 can be configured to perform differential inter-frame decoding by decoding the difference (e.g., quantized or unquantized) between the current frame and the immediately preceding frame from the data stream and adding the difference (e.g., quantized or unquantized) to the current frame. As a result, the storage size of the data used for the frame can be further reduced, e.g., reduced to storing only the data of the immediately preceding frame 14 (e.g., sample values ​​or transform coefficients) and updating the frame buffer for the current frame by adding the difference. Such storage complexity is lower and allows for faster encoding / decoding of video 12.

[0072] The video encoder 10 can be configured to switch between encoding and decoding frames per frame using intra-frame codec and using differential inter-frame codec. For example, the video encoder 10 can be configured to associate one of a plurality of codec modes with each frame, wherein the codec modes may include intra-frame codec and differential inter-frame codec, such that each frame associated with intra-frame codec is encoded and decoded entirely using intra-frame codec, and each frame associated with differential inter-frame codec is encoded and decoded entirely using differential inter-frame codec. The video decoder 40 can be configured to switch between decoding frame 14 per frame using intra-frame decoding and decoding frame 14 using differential inter-frame decoding (in other words, the video decoder associates one of a plurality of decoding modes with each frame, including intra-frame decoding and differential inter-frame decoding, so that each frame associated with intra-frame decoding is decoded entirely using intra-frame decoding, and each frame associated with differential inter-frame decoding is decoded entirely using differential inter-frame decoding).

[0073] For example, differential inter-frame encoding / decoding can be useful for encoding video 12 captured by a camera device that moves little or no. For instance, a medical professional might keep the camera device in a fixed position, pointing it at a patient or inside a patient's body (e.g., inside a body cavity before surgery). When the camera device is fixed in position, the frames subsequently recorded by the camera show little or no difference (e.g., except for background noise or minor tissue movement). Because the differences between frames are small or nonexistent, inter-frame encoding / decoding requires very little data transmission to signal the differences between frames. Video quality can be progressively improved with each frame 14 of differential inter-frame encoding / decoding, where higher quality video (or images) can be obtained after encoding / decoding multiple frames 14, but with very little data transmission (due to the small differences between frames 14). As a result, data transmission may no longer be a bottleneck for encoding / decoding high-quality images, and the video quality decoded by the decoder 40 may be improved. It should be noted that the encoding / decoding methods disclosed herein are not limited to video captured by a fixed camera device, which naturally may exhibit differences (e.g., due to noise). Video 12 may include two or more identical frames 14 (e.g., copies of previous frames 14). For example, a user may capture high-quality images stored in a camera device, where video encoder 10 may be configured to encode video 12 with multiple copies of high-quality images.

[0074] Figure 3a shows a schematic diagram of the first decoded frame 14a of video 12. Frame 14 exemplarily shows a medical image of a patient that a user (e.g., a medical professional) may wish to study for diagnosis or surgery. Therefore, the user may require high-quality (e.g., 4K, 8K, or higher) video 12. However, the transmission speed of data stream 16 may be insufficient to provide the first frame 14a at the required quality. However, subsequent frames 14b, 14c, 14d of the patient may not differ significantly from the first frame 14a, for example, due to the fixed position of the recording camera (e.g., fixed by a stationary device or held steadily by the user) or static diagnostic images in the video. The first frame 14a may be intra-frame encoded (e.g., using intra-frame encoding and not using differential inter-frame encoding).

[0075] Figures 3b to 3d show schematic diagrams of the second frame 14b, the third frame 14c, and the fourth frame 14d of the video, which appear immediately after the first frame 14a of Figure 3a (in terms of presentation and encoding / decoding order).

[0076] When encoding the second frame 14b, the video encoder 10 can be configured to perform differential inter-frame encoding and decoding by subtracting the current frame 14b from the previously encoded frame 14a (e.g., in terms of encoding order and presentation order) to obtain a difference and encoding and decoding that difference into the data stream 12 (e.g., with or without quantization, e.g., the difference between quantized transform coefficients). For example, the video encoder 10 can be configured to perform the subtraction between the current frame 14b and the previously encoded frame 14a in a mutually non-displaced manner.

[0077] Since frames 14a and 14b are not significantly different, the difference does not require much data to be transmitted. In the example shown in Figures 3a-d, the video encoder 10 is configured to perform differential inter-frame encoding and decoding by subtracting the current frame 14b from the immediately preceding frame 14a to obtain the difference and encoding and decoding the difference into the data stream 16 (e.g., with or without quantization).

[0078] The video decoder 40 can be configured to perform differential inter-frame decoding by decoding the difference (e.g., quantized or unquantized) between the current frame 14b and the previously decoded frame 14a (e.g., in terms of encoding and rendering order) from the data stream 16 and adding the difference (e.g., quantized or unquantized) to the previously decoded frame 14a (e.g., in terms of encoding and rendering order). For example, the video decoder 40 can be configured to decode the difference (e.g., quantized or unquantized) between the current frame 14b and the immediately preceding frame 14a in the transform domain from the data stream 16, perform addition in the transform domain to obtain the transform of the current frame 14b, and then re-transform the current frame 14b.

[0079] Therefore, as shown in Figure 3b, the quality of the second frame 14b is better than that of the first frame 14a.

[0080] Similarly, the video encoder 10 and the video decoder 40 can be configured to perform inter-frame differencing on the third frame 14c based on the difference (e.g., quantized or unquantized) between the third frame 14c and the second frame 14b. As shown in Figure 3c, the quality of the third frame 14c is higher than that of the second frame 14b.

[0081] The process continues to the fourth frame 14c, which has a higher quality than the third frame 14c. The quality of the fourth frame 14c is sufficient, which can be determined by the video encoder 10, for example, by measuring the quality loss caused by encoding / decoding loss and determining whether the quality loss meets one or more criteria.

[0082] In the example shown in Figure 3a, the video encoder 10 is configured to signal information about quality loss as binary information with two states, where the first state corresponds to sufficient quality and the second state corresponds to insufficient quality. The video decoder 40 can be configured to derive the information about quality loss from the data stream 12.

[0083] The video decoder 40 can be configured to modify the reconstructed version of the video 12 obtained by decoding the data stream 16 in order to provide the reconstructed version of the video 12 with an indication 46 of the stage during which the image loss meets one or more criteria (e.g., if the quality loss is below a threshold that may depend on the display quality of the display, such that the total quality loss (e.g., the sum of the display-induced quality loss and the quality loss signaled in the data stream) is below a certain threshold).

[0084] In Figures 3a-d, indicator 46 is exemplarily shown as an icon whose appearance varies depending on whether the quality loss meets the criteria. When the video quality is insufficient (see Figures 3a-c), indicator 46 has a first color (e.g., red); when the video quality is sufficient (see Figure 3d), indicator 46 has a second color (e.g., green). Alternatively or additionally, the icon may have a first shape and a second shape (e.g., a cross and a checkmark). Indicator 46 may include a border for frame 14 (e.g., with a different color).

[0085] Indicator 46 is not necessarily visible in both cases of sufficient and insufficient quality. In other words, video decoder 40 can be configured to modify the reconstructed version of video 12 obtained by decoding data stream 16 so that indicator 46 is provided only for the reconstructed version of video 12 when the quality is sufficient or only when the quality is insufficient. For example, indicator 46 may display an icon and / or border only during periods of insufficient video quality (or only during periods of sufficient video quality).

[0086] The video decoder 40 can be configured to provide binary indications as described above, for example, when information about quality loss is derived from the data stream 16 as binary information with two states, where the first state corresponds to sufficient quality and the second state corresponds to insufficient quality.

[0087] When video decoder 40 is configured to derive information about quality loss from the data stream as non-binary information (e.g., using integer or floating-point scales), video decoder 40 can be configured to determine two states, where a first state corresponds to sufficient quality and a second state corresponds to insufficient quality. Video decoder 40 can be configured to determine the two states as described above with reference to video encoder 10 (e.g., using a threshold). For example, video encoder 10 can signal the information as a peak signal-to-noise ratio (PSNR) value, and video decoder 40 can be configured to compare the PSNR value with a threshold. Subsequently, video decoder 40 can provide a binary indication 46 as described above.

[0088] However, the video decoder 40 can be configured to provide non-binary indications 46 with more than two states, for example, when information about quality loss is derived from the data stream 16 as non-binary information (e.g., by using integer or floating-point scales).

[0089] Indicator 46 may include displaying a number corresponding to or based on an integer or floating-point scale (e.g., rescaling to a percentage). Alternatively or additionally, indicator 46 may include an icon and / or a border, wherein at least one of color, brightness, and opacity represents an integer or floating-point scale. For example, indicator 46 may include a border that gradually switches from a first color (e.g., red) to a second color (e.g., yellow) based on an integer or floating-point scale, and may switch to a third color (e.g., green) when a target quality is reached.

[0090] Alternatively or additionally, video decoder 40 may be configured to output metadata revealing information about quality loss (or image loss) at its output interface. The output interface of video decoder 40 may (communicatively) couple to or be coupleable to an indication device other than the display used to display the reconstructed video 12. For example, the indication device may include at least one of a speaker, a haptic feedback device, another display (independent of the display), and a computer. Such an indication device allows the user to be indicated to the quality loss without the user seeing the reconstructed video 12. For example, the user may be standing too far from the display of the reconstructed video 12, or may have to focus on the camera device (e.g., for security reasons or because the camera device is handheld). The indication device may indicate to the user that the quality of video 12 is sufficient (e.g., by outputting sound, vibration of the handheld device, or a close-up display, such as on the camera device).

[0091] The examples shown in Figures 3a-d are implemented using only differential inter-frame coding and decoding, where each frame 14a-d is encoded and decoded entirely using differential inter-frame coding and decoding (without intra-frame coding and decoding). However, the video encoder 10 and video decoder 40 can be configured to use progressive decoding refresh to spatially reposition the intra-frame coding and decoding portion of frame 14 of video 12 in the frame region containing the coding and decoding portion of frame 14 between frames, and to use differential inter-frame coding and decoding outside the intra-frame coding and decoding portion. The video decoder 40 can be configured to use progressive decoding refresh to spatially reposition the intra-frame decoding portion of frame 12 of video 12 in the frame region containing the decoding portion of frame 14 between frames, and to use differential inter-frame decoding outside the intra-frame decoding portion.

[0092] Figure 4a shows a schematic diagram of the first decoded frame 14a of video 12. The first frame 14a is divided into four striped subframe portions 32a-d. Alternatively, the first frame 14a may include any number N striped subframe portions 32 (e.g., N=60) and subframe portions 32 of any other shape described herein. These four subframe portions 32a-d cover the entire area of ​​the first frame 14a. Alternatively, the subframe portions 32 may cover only a portion of the first frame 14a.

[0093] It should be noted that indicator 46 is not shown in Figures 4a-d. However, any indicator 46 disclosed herein can be implemented in the examples of Figures 4a-d.

[0094] The video encoder 10 can be configured to perform intra-frame coding / decoding (e.g., partial intra-frame coding / decoding) and differential inter-frame coding / decoding (e.g., partial differential inter-frame coding / decoding) on ​​different portions of frame 14. For each frame 14, one of the four subframe portions 32 forms a portion of frame 14 that is intra-frame coded / decoded (hereinafter referred to as the "intra-frame coding / decoding portion"), and three of the four subframe portions 32 form a portion of frame 14 that is differentially coded / decoded (hereinafter referred to as the "differential inter-frame coding / decoding portion"). However, the intra-frame coding / decoding subframe portion 32 varies with each frame. It should be noted that in the example shown in Figures 4a-d, the intra-frame coding / decoding portion is only a single subframe portion 32. However, the intra-frame coding / decoding portion may include more than one subframe portion 32 (e.g., a column of four intra-frame coding / decoding portions 32 shown in Figure 2a). In Figures 4a-d, the subframe portions 32 that form the intra-frame coding / decoding portion are indicated by the black bars below, as a visual indication to the reader, and do not constitute part of frames 14a-d.

[0095] Furthermore, it should be noted that, referring to Figures 4a-d, an example is shown where the video quality of subframe portion 32 degrades after intra-frame encoding / decoding. However, there may be scenarios where intra-frame encoding / decoding may improve video quality at a rate similar to or faster than differential inter-frame encoding / decoding (e.g., for the first frame or the first few frames). In this simplified example, it is assumed that subframe portion 32 of intra-frame encoding / decoding reaches maximum video quality after two differential inter-frame encoding / decoding sessions (e.g., cannot be significantly improved by further inter-frame encoding / decoding). In more realistic scenarios, such maximum or asymptotic video quality may require a large number of iterative differential inter-frame encoding / decoding sessions.

[0096] In Figure 4a, the first subframe portion 32a on the left side of frame 14a forms the intra-frame coding / decoding portion (indicated by the black bar below). The second, third, and fourth subframe portions 32c-d form the differential inter-frame coding / decoding portion.

[0097] The video encoder 10 can be configured to perform a subtraction between the differential inter-frame encoding / decoding portion of the first frame 14a and the (co-located) second, third, and fourth sub-frame portions of the previously encoded / decoded frame (not shown in FIG. 4a) in a manner without mutual displacement. The video encoder 10 can then encode this difference into the data stream 16 (e.g., using quantization, e.g., without quantization; e.g., the difference between quantized values, e.g., the difference between quantized transform coefficients).

[0098] The video decoder 40 can be configured to perform differential interframe decoding by decoding (e.g., quantized or unquantized) the difference between the first frame 14a (the differential interframe codec portion) and the previously decoded frames (the second, third, and fourth subframe portions) from the data stream 16, and reconstructing the second, third, and fourth subframe portions 32c-d of the first frame 14 by adding (e.g., quantized or unquantized) the difference to the previously decoded frames.

[0099] Similarly, the video encoder 10 can be configured to perform intra-frame encoding and decoding of the first subframe portion 32a and encode the encoding and decoding information (e.g., transform coefficients or sample values) used for intra-frame encoding and decoding of the first subframe portion 32a into the data stream 16. The video decoder 40 can be configured to perform intra-frame encoding and decoding of the first subframe portion 32a based on the encoding and decoding information extracted from the data stream 16 for intra-frame encoding and decoding of the first subframe portion 32a. Intra-frame encoding and decoding may not require any encoding and decoding information from previous frames and may allow for a completely new reconstruction of the corresponding subframe portion 32. Therefore, intra-frame encoding and decoding can form a frame portion reset within the subframe portion 32, which can be improved by differential inter-frame encoding and decoding in subsequent frames.

[0100] Figure 4a may represent a stage of the video where frame 14 has high video quality (e.g., due to iterative differential inter-frame coding), where the first subframe portion 32a is intra-coded without using information from previous frames. Therefore, the video quality of the first subframe portion 32a is lower than that of the second, third, and fourth subframe portions 32b-c. The fourth subframe portion 32d has undergone intra-frame coding in previous frames (similar to the intra-frame coding described below with reference to Figure 4d), but has already undergone one differential inter-frame coding in Figure 4a. Therefore, the fourth subframe portion 32d has better video quality than the "new" intra-frame coded first subframe portion 32a, but lower video quality compared to the second and third subframe portions 32c and 32d, which have undergone at least two inter-frame codings and thus achieved maximum video quality.

[0101] Figure 4b shows a schematic diagram of the second decoded frame 14b of video 12. In the second frame 14b, the intra-frame coding / decoding portion is formed by the second subframe portion 32b (as shown by the black bar below), and the differential inter-frame coding / decoding portion is formed by the first subframe portion 32a, the third subframe portion 32c, and the fourth subframe portion 32d. As a result, the intra-frame coding / decoding portion is shifted to the right by one subframe portion 32. However, any other mode can be used to achieve the shift of the intra-frame coding / decoding portion, such as from right to left or from top to bottom (or vice versa).

[0102] When comparing the first frame 14a and the second frame 14b, the video quality of the differential inter-frame encoding / decoding portion has improved. The fourth subframe portion 32d has now undergone two inter-frame encoding / decoding operations and has reached the maximum video quality. Since the first subframe portion 32a only underwent one inter-frame encoding / decoding operation, it has not yet reached the maximum video quality. The intra-frame encoding / decoding portion (or the second subframe portion 32b) has undergone intra-frame encoding / decoding operations, where the video quality has decreased (or reset) to a quality similar to that of the first subframe portion 32a in the first frame 14a (see Figure 4a).

[0103] Figure 4c shows a schematic diagram of the third decoded frame 14c of video 12. In the third frame 14c, the intra-frame coding / decoding portion is formed by the third subframe portion 32c (as shown by the black bar below), and the differential inter-frame coding / decoding portion is formed by the first subframe portion 32a, the second subframe portion 32b, and the fourth subframe portion 32d. As a result, the intra-frame coding / decoding portion is shifted to the right by one subframe portion 32.

[0104] When comparing the second frame 14b and the third frame 14c, the video quality of the differential inter-frame encoding / decoding portion has improved. The first subframe portion 32a has now performed two inter-frame encoding / decoding operations and has reached the maximum video quality. Since the second subframe portion 32b has only performed one inter-frame encoding / decoding operation, it has not yet reached the maximum video quality. The intra-frame encoding / decoding portion (or the third subframe portion 32c) has performed intra-frame encoding / decoding operations, in which the video quality has decreased (or reset).

[0105] Figure 4d shows a schematic diagram of the fourth decoded frame 14d of video 12. In the fourth frame 14d, the intra-frame coding / decoding portion is formed by the fourth subframe portion 32d (as shown by the black bar below), and the differential inter-frame coding / decoding portion is formed by the first subframe portion 32a, the second subframe portion 32b, and the third subframe portion 32c. As a result, the intra-frame coding / decoding portion has shifted to the right by one subframe portion 32. Therefore, during the coding / decoding of the four frames 14a-d, the intra-frame coding / decoding portion has moved through the entire frame 14.

[0106] The second subframe portion 32b has now undergone two inter-frame encoding / decoding operations and has reached maximum video quality. Since the third subframe portion 32c has only undergone one inter-frame encoding / decoding operation, it has not yet reached maximum video quality. The intra-frame encoding / decoding section (or the fourth subframe portion 32d) has undergone intra-frame encoding / decoding operations, during which video quality is degraded (or reset).

[0107] In frames following the fourth frame 14 (not shown), the first subframe portion 32a again forms an intra-frame encoding / decoding portion, corresponding to a scene similar to the first frame 14a shown in Figure 4a.

[0108] As shown in Figures 4a-d, the intra-frame encoding / decoding portion moves through the entire frame over a four-frame period. Each subframe portion 32 is ultimately intra-coded once and differentially encoded / decoded three times. By dividing the frame into more subframe portions, the length of the period and the number of times the subframe portion is differentially encoded / decoded can be increased. For example, if frame 14 is divided into N (e.g., N=60) subframe portions 32, the intra-frame encoding / decoding portion moves through the entire frame 14 after a period of N frames, where each subframe portion 32 is intra-coded once and differentially encoded / decoded N-1 times (e.g., 59 times).

[0109] Intra-frame coding and decoding allows for the coding and decoding of a portion of a frame without the coding and decoding information (e.g., sample values ​​or transform coefficients) from previous frames. Therefore, intra-frame coding and decoding is less susceptible to error propagation caused by potential errors in previous frames. For example, if video 12 includes errors (e.g., due to transmission errors or coding / decoding errors) or significant scene changes (e.g., accidental obstruction of the camera lens), intra-frame coding and decoding provides the basic information that can subsequently be improved through differential inter-frame coding and decoding. Thus, this type of video coding and decoding is more robust while reducing the data rate. While progressive decoding refresh may result in poor video quality in the first frame (or a portion of a subframe), differential inter-frame coding and decoding allows for iterative improvement in video quality (similar to progressive coding and decoding systems).

[0110] Frames may also exist with fully differential inter-frame coding, where quality information is derived directly from the aforementioned averaging / cumulative / summing of the frame's transform coefficients (or their squares, etc.). For example, a GDR sequence can be placed at specific temporal intervals, comprising N consecutive GDR-coded frames, where N is the number of sub-regions. As described above, these sub-regions are spatially traversed for intra-frame coding and decoding, with frames in between being fully differentially inter-frame coded and decoded.

[0111] It is important to note that Figures 3a-d illustrate frames 14 that may only undergo differential inter-frame coding (e.g., one or more frames may only undergo intra-frame coding, e.g., the first frame, e.g., multiple frames, e.g., after a certain number of frames and / or periodically after a quality degradation is detected), while Figures 4a-d illustrate all frames 14 that are partially differentially inter-frame coded and partially intra-frame coded. However, partial coding (e.g., part of the same frame is coded using differential inter-frame coding and part is coded using intra-frame coding) or dedicated coding (e.g., a frame is coded using only differential inter-frame coding or only intra-frame coding) are not mutually exclusive. For example, the examples shown in Figures 4a-d may include frames 14 that are only differentially inter-frame coded (e.g., if a certain image quality has been achieved and / or the data bandwidth for image data transmission is decreasing) or frames 14 that are only intra-frame coded (e.g., for the first frame). Similarly, the examples shown in Figures 3a-d may include frames that are partially encoded and decoded intra-frame and partially encoded and decoded using differential inter-frame encoding and decoding.

[0112] The video encoder 10 can be configured to encode frame 14 of video 12 using bit rate control, according to which a constant bit rate (in other words, the number of bits per frame region) is followed in units of subframe portions 32 (e.g., N=60, where N is the number of subframe regions 32, where they can be of equal size), wherein the frame regions are divided into these subframe portions.

[0113] For example, each of the N subframe portions 32 sequentially becomes the currently refreshed intra-frame codec portion, such as a subframe portion in frame 14, the next subframe portion in the next frame, and so on (e.g., as described with reference to Figures 4a-d). Within each frame 14, the remaining N-1 subframe portions 32 form the unrefreshed portion and are all encoded using differential inter-frame codec. For example, bitrate control may require that the difference in bitrate between each subframe portion 32 does not exceed a predetermined threshold (e.g., less than 10% or 5%, or less than 2%).

[0114] For subframes 32 whose length is less than the length defined by a constant bit rate, bit padding (e.g., filling some or all unused bits with zeros that would have been available given a target bit rate) can be used to achieve the subframe size defined by the constant bit rate.

[0115] The video encoder 10 can be configured to continuously measure the quality loss caused by encoding / decoding loss and continuously signal information about the quality loss as side information in the data stream 16, for example for each frame 14 (or a set of frames 14) or each subframe portion 32 (or a set of subframe portions).

[0116] When a user operates the camera device capturing video 12, frames may differ significantly during the movement of the camera device (e.g., when moving the camera towards a target location), resulting in significant differences in subsequent frames of the video. Consequently, continuous measurements of quality loss due to encoding / decoding losses performed by the video encoder 10 may indicate a large quality loss, e.g., insufficient video quality. The video encoder 10 continuously signals information about the quality loss to the video decoder 40, which can then provide the user with information about insufficient video quality. However, once the user positions the camera device in a fixed location, the differences in subsequent frames 14 become smaller, and the video encoder 10 may begin measuring a lower quality loss, which is then signaled to the video decoder 40. The video decoder 40 can then indicate to the user that the video quality is sufficient (and / or begin saving the video on a storage device).

[0117] The video decoder 40 can be configured to save a still frame (e.g., a freeze frame) based on the video 12 once the video quality is determined to be sufficient. Subsequently, the user may be able to move the camera device (e.g., to capture additional video or to remove the camera device from the surgical site). The video decoder 40 can be configured to provide output (e.g., visually, audibly, or tactilely) to indicate to the user that the still frame has been saved (or stored).

[0118] When the user begins to move the camera device, the content of the recorded frames may vary more significantly, resulting in encoding / decoding loss. The video encoder 10 is configured to measure and signal this loss to the video decoder 40. As a result, the user can be informed whether the video quality meets the target quality or falls below it.

[0119] Instead of continuous measurement and signaling notifications, the video encoder 10 can be configured to conditionally (e.g., based on user input) perform measurement and signaling notifications. The video encoder 10 can be configured to signal the start of a phase of image invariance (in other words, a phase of still image content) in the data stream.

[0120] For example, the video encoder 10 can be configured to receive signals indicating a phase of image invariance. A user can input (e.g., via mouse click, keyboard, or voice command) to signal the start of the image invariance phase (e.g., after the camera device has been positioned). Alternatively, the camera mount used to position the camera device can be configured to send a signal to the video encoder 10 upon receiving the camera device (e.g., by triggering a switch on the camera mount upon receiving the camera device).

[0121] The video encoder 10 can be configured to measure and signal frames belonging to a phase of image invariance (e.g., not to apply this to frames not belonging to any phase of image invariance, or in other words, to determine for each frame of the video whether the corresponding frame belongs to any phase of image invariance, and if so, to apply measurement and signaling to that corresponding frame). Signaling may include sending a binary flag (or bit) during a phase of image invariance. Alternatively, the video encoder 10 can be configured to signal a message indicating the start of a phase.

[0122] The duration of the image invariance phase can be a predetermined time (e.g., 5 seconds) or determined based on a criterion related to the measured quality loss. For example, the video encoder 10 can be configured to determine the end of image invariance when the quality loss exceeds a predetermined threshold. As a result, the video encoder 10 can be configured to signal the end of image invariance in the data stream (e.g., by not signaling a binary flag or by signaling a message indicating the end of the phase).

[0123] Video decoder 40 can be configured to derive an indication of the start of the phase of image invariance from data stream 16, for example, in the form of a start-indicating message or a series of binary bits. Video decoder 40 can be configured to output metadata revealing the start of the phase of invariance at the video decoder's output interface. The output interface can be coupled to or is coupleable to a monitor, speaker, haptic feedback device, or computer. The monitor, speaker, or haptic device can be configured to indicate to the user that the phase of image invariance has begun (e.g., and provide confirmation of successful user registration input).

[0124] In the examples shown in Figures 1a and 1b, the video encoder 10 is the entity that measures the quality loss caused by the encoding / decoding loss. The video encoder 10 then signals information about the quality loss as side information. However, the decoder may be able to at least estimate information about the quality loss without requiring the encoder to additionally signal this information.

[0125] Figure 5a shows a schematic diagram of one embodiment of a video decoder 50 used for decoding video.

[0126] Video decoder 50 is configured to decode frame 14 of video 12 from data stream 16 using intra-frame decoding and differential inter-frame decoding (e.g., using intra-frame decoding and differential inter-frame decoding for the same frame; e.g., when using progressive decoding refresh; e.g., simultaneously; or using differential inter-frame decoding only for one or more frames), wherein video 12 is encoded into data stream 16 with encoding / decoding loss, and to estimate information about the quality loss caused by the encoding / decoding loss. Video decoder 50 is also configured to output metadata 58 at output interface 56 of video decoder 50 that reveals information about image loss, and / or modify the reconstructed version of video 12 obtained by decoding the data stream to provide an indication of the stage during which the image loss of the reconstructed version of video 12 meets a certain criterion.

[0127] The criteria may include one or more thresholds for estimating the quality loss. For example, indicator 60 may indicate whether the estimated quality loss is higher than the threshold (e.g., indicating insufficient video quality) or lower than the threshold (e.g., indicating sufficient video quality). Indicator 60 may be implemented as indicator 46 as described above (the difference being that it depends on the estimated quality loss, rather than on information about the quality loss caused by encoding / decoding losses when encoding the video into the data stream).

[0128] Alternatively, the criterion may depend on both: the estimated quality loss and the information about the quality loss signaled by the encoding device 10. For example, the criterion may be met if the (estimated) quality loss is below a certain threshold (which may depend on the display quality of the display), such that the total quality loss (e.g., the sum of the quality loss caused by the display and the quality loss signaled in the data stream) is below another threshold.

[0129] Alternatively or additionally, the standard may include a measure of padding bits (e.g., as zeros or ones). A larger number of padding bits may indicate a smaller difference between subsequent frames and / or subframe portions, thus achieving higher video quality. On the other hand, a smaller number of padding bits may indicate a larger difference between subsequent frames and / or subframe portions (e.g., because the video encoder encodes more information, and because the video quality could still be significantly improved), thus the video quality is not yet sufficient. The standard may include one or more thresholds for the number or density of padding bits. The standard may include a function of the measure of padding bits and the video quality. Padding bits may, for example, be used in constant bitrate encoding / decoding operations where bits that would provide for transmitting encoding / decoding information (due to the constant bitrate) are unused, and therefore padding bits or zeros are used.

[0130] Alternatively or additionally, the standard may include the deviation (e.g., difference) between the maximum allowed bitrate and the current bitrate (e.g., the current and / or average bitrate of the current frame). A larger deviation (e.g., a lower bitrate) may indicate a smaller difference between frame and / or subframe portions, thus achieving higher video quality. Conversely, a smaller deviation may indicate lower video quality. The standard may include one or more thresholds for the current and / or average bitrate (e.g., absolute values ​​or relative to the maximum bitrate). The standard may include a measure of the current and / or average bitrate as a function of video quality. The maximum bitrate may, for example, be used in conjunction with variable bitstream coding / decoding.

[0131] For example, video decoder 50 may include decoding unit (or device) 52 configured to decode frame 14 of video 12 from data stream 16 encoded into it with encoding / decoding loss, using intra-frame decoding and differential inter-frame decoding (e.g., using intra-frame decoding and differential inter-frame decoding for the same frame; e.g., when using progressive decoding refresh; e.g., simultaneously; or using differential inter-frame decoding only for one or more frames). Video decoder 50 may include estimation unit (or device) 54 configured to estimate information about the quality loss 64 caused by encoding / decoding loss.

[0132] Units 52 and 54 are shown as independent units. However, at least two (e.g., all) functions of video decoder 50 may be performed (at least partially) by the same device. For example, video decoder 50 may include a processing unit configured to perform at least one of decoding and estimation. For example, video decoder 50 may be a video camera (e.g., for playing back recorded video), a monitor, a server, a computer, a smartphone, or a tablet device, or may be part of such devices. For example, video decoder 50 may estimate the quality loss itself and forward it to the user.

[0133] Video decoder 50 can be configured to derive an indication of the start of the image invariance phase from the side information of data stream 16, and to estimate and output and / or modify the image invariance phase based on the indication of the start of the image invariance phase, so as to estimate and output the image invariance phase. Video decoder 50 may include any features of the image invariance phase disclosed in reference to video decoder 40 above, or may be configured to perform any steps of the image invariance phase disclosed in reference to video decoder 40 above.

[0134] The video decoder 50 can be configured to estimate information 64 about the quality loss caused by encoding / decoding losses when the video 12 is encoded into the data stream 16, based on the average level (or, in other words, the average amplitude) or average energy of the unrefreshed frame portion (in other words, the portion decoded using differential inter-frame decoding). Alternatively, at least one of the sum, variance, and median of the level or energy of the unrefreshed frame portion or frame can be determined. For example, information about the quality loss can be, or can be based on, the sum of the linear, absolute, or squared values ​​(or any other measure) of sample values ​​or transform coefficients of the encoded frames or frame differences.

[0135] For example, if the entire frame 14 is differentially inter-decoded, the average level or energy of the entire frame 14 can be determined. If only a portion of frame 14 is differentially inter-decoded, the average level or energy can be determined based on that portion of frame 14. The average level or energy (or sum, variance, or median) of multiple decoded frames can be determined. Alternatively or additionally, the average level or energy of multiple differences between decoded frames can be determined based on the encoding / decoding information in data stream 16. For example, as video quality improves, the differences between decoded frames 14 may gradually decrease (e.g., because the difference between decoded frame 14 and the original frame decreases). Therefore, a decrease in the average value of frame differences may indicate an improvement in video quality.

[0136] Alternatively, the video decoder 50 may be configured to estimate information about the quality loss caused by encoding / decoding losses during video encoding to a data stream based on a combination (e.g., summation) of the following: the average level (or, in other words, average amplitude) or average energy of the unrefreshed frame portion of the frame for which information about the quality loss is to be estimated (in other words, the portion decoded using differential inter-frame decoding), and the average level (or, in other words, average amplitude) or average energy of the co-bit unrefreshed frame portion of the subsequent frame or the preceding frame for which information about the quality loss is to be estimated, wherein the co-bit unrefreshed frame portion is co-bit with the refreshed portion of the frame for which information about the quality loss is to be estimated.

[0137] It should be noted that the video encoder can be configured to encode video 12 using progressive decoding refresh based on the details described herein.

[0138] The video decoder 50 can be configured to use progressive decoding refresh by performing intra-frame encoding / decoding on the intra-frame encoding / decoding portion of frame 14 and differential inter-frame decoding on another portion of frame 14 (e.g., the remainder). By avoiding determining the average level or average energy of the refreshed frame portion, the overall average value can better represent the current improvement in video quality achieved through differential inter-frame encoding / decoding.

[0139] The intra-frame codec portion of frame 14 can be differentially decoded in an immediately following frame or an earlier frame. Therefore, the video decoder 50 may be able to obtain the "missing" level or energy from the intra-frame codec portion that co-occurs with the refresh portion of the frame from which information about quality loss is to be estimated. Thus, an average of all frame portions can be obtained. As a result, the overall average may provide a better basis for comparisons between frames 14.

[0140] Alternatively, the refresh portion of the current frame 14 can be omitted to estimate information about quality loss, and an average can be formed only for the unrefreshed areas in frame 14 (e.g., differentially encoded inter-frame codecs that are not intra-frame codecs). As a result, the complexity of the estimation process can be reduced.

[0141] Information about quality loss can be determined based on at least one of the average, sum, median, and weighted average of the energy or level of the frame portion as described above.

[0142] The video decoder 50 can be configured to: at the output interface of the video decoder, output binary information, as metadata 58 revealing information about image loss, indicating whether the quality loss meets a predetermined minimum quality level, wherein if the average level or average energy is below a predetermined threshold, the binary information is set to indicate that the quality loss meets the predetermined minimum quality level; or, modify the reconstructed version of the video obtained by decoding the data stream to provide an indication of the stage during which the image loss meets a certain criterion, such that the indication provides binary information about whether the quality loss meets the predetermined minimum quality level, thereby indicating that the quality loss meets the predetermined minimum quality level if the average level or average energy is below the predetermined threshold.

[0143] Optionally, any step described above for estimating information about quality loss in the reference video decoder 50 can be used by any video encoder described herein (e.g., video encoder 10; for example, for measuring quality loss caused by encoding / decoding loss).

[0144] The video decoder 50 can be configured to output metadata 48 as described above in the reference to the video decoder 40 configured to output metadata (e.g., displaying icons or borders or providing acoustic output).

[0145] Video decoder 50 can be configured to perform differential inter-frame decoding by decoding the difference (e.g., difference between quantized values; e.g., difference between quantized transform coefficients) between the current frame 14 and previously decoded (e.g., in terms of encoding and rendering order) frames 14 from data stream 16, and by adding the difference (e.g., quantized or unquantized) to the previously decoded (e.g., in terms of encoding and rendering order) frames 14. Video decoder 50 can be configured to decode the difference (e.g., quantized or unquantized) between the current frame 14 and the immediately preceding frame 14 in the transform domain from data stream 16, add in the transform domain to obtain the transform of the current frame, and re-transform the transform. Video decoder 50 can be configured to perform differential inter-frame decoding as described above with reference to video decoder 40.

[0146] Video decoder 50 can be configured to add (e.g., quantized or unquantized) differences between previously decoded frames 14 in a manner without mutual displacement (e.g., without motion-compensated inter-frame displacement). Video decoder 50 can be configured to form additions as described above with reference to video decoder 40.

[0147] Video decoder 50 can be configured to sequentially decode frames using an encoding / decoding order consistent with the frame presentation order. Video decoder 50 can be configured to perform differential inter-frame decoding by decoding the difference (e.g., difference between quantized values; e.g., difference between quantized transform coefficients) between the current frame and the immediately preceding frame from the data stream, and adding the difference (e.g., quantized or unquantized) to the current frame. Video decoder 50 can be configured to decode the difference (e.g., quantized or unquantized) between the current frame 14 and the immediately preceding frame 14 in the transform domain from the data stream 16, perform addition in the transform domain to obtain the transform of the current frame, and re-transform the transform. Video decoder 50 can be configured to decode frames, perform differential inter-frame decoding, and decode (e.g., quantized or unquantized) differences as described above with reference to video decoder 40, at least one of these.

[0148] Video decoder 50 can be configured to switch between decoding frames per frame using intra-frame decoding and decoding frames using differential inter-frame decoding (in other words, the video decoder associates each frame with one of several decoding modes, including intra-frame decoding and differential inter-frame decoding, such that each frame associated with intra-frame decoding is decoded entirely using intra-frame decoding, and each frame associated with differential inter-frame decoding is encoded entirely using differential inter-frame decoding), and / or use progressive decoding refresh to spatially reposition the intra-frame decoding portion of a video frame in a frame region having the decoding portion of the frame between frames, and to use differential inter-frame decoding outside of the intra-frame decoding portion. Video decoder 50 can be configured to switch and use at least one of progressive decoding refresh as described above with reference to video decoder 40.

[0149] Figure 5b shows a schematic diagram of the relationship between frame number and peak signal-to-noise ratio (PSNR).

[0150] PSNR can be an example of a metric for video quality. Low noise generally indicates good video quality. Therefore, a large ratio between the signal peak and noise may indicate high video quality. PSNR can form the basis for information about quality loss, or form the basis for measuring or determining information about quality loss.

[0151] The video encoder 10 can be configured to determine the PSNR of the current frame. The video encoder 10 itself can determine a measure of quality loss based on the PSNR, or it can signal the PSNR as side information to the video decoders 40 and 50, where the video decoders 40 and 50 can be configured to determine a measure of quality loss.

[0152] For example, video encoder 10 can be configured to determine whether the PSNR exceeds a predetermined level. In the example shown in Figure 6, a threshold 66 of 90 dB is used to distinguish between sufficient video quality (equal to or higher than 90 dB) and insufficient video quality (lower than 90 dB). Test measurement 68 (indicated by black squares) shows that the PSNR is low in the first frame (approximately 33 dB), where the PSNR gradually increases with increasing frame number. The smallest frame number with a PSNR higher than 90 dB is frame number 17 (approximately 95 dB).

[0153] For example, video encoder 10 can signal information about quality loss as binary information with two states: insufficient quality for frames 1 through 16, and sufficient quality for frame 17. Signaling binary information can improve signaling efficiency. Alternatively, video encoder 10 can signal more complex information, such as the PSNR value. Such complex information allows video decoders 40 and 50 to adapt better. For example, video decoders 40 and 50 can acquire more appropriate thresholds (e.g., adapted to the monitor type or a specific task, such as displaying specific medical or technical videos).

[0154] Different combinations of video encoders and video decoders can form different systems for video encoding and decoding.

[0155] Figure 6a shows a schematic diagram of a first system 70, which includes the video encoder 10 and video decoder 40 described herein.

[0156] The video encoder 10 is configured to encode frames of video into a data stream using intra-frame coding and differential inter-frame coding (e.g., using intra-frame coding and differential inter-frame coding for the same frame when using progressive decoding refresh, or using only differential inter-frame coding; e.g., for one or more frames or all frames), measure the quality loss caused by coding and decoding losses, and signal information about the quality loss as side information in the data stream.

[0157] The video decoder 40 is configured to decode video frames from the data stream using intra-frame decoding and differential inter-frame decoding (e.g., using intra-frame encoding and differential inter-frame encoding for the same frame when using progressive decoding refresh, or using only differential inter-frame encoding; e.g., for one or more frames or all frames) and derive information about the quality loss caused by the encoding and decoding losses when the video is encoded into the data stream from the side information of the data stream.

[0158] The video encoder 10 and the video decoder 40 may include any features and / or may be configured to perform any of the steps disclosed herein.

[0159] The video encoder 10 is the entity that measures quality loss. Therefore, the complexity of the decoder 40 can be reduced. Furthermore, the video encoder 10 can obtain a better reference for measuring quality loss in the form of the original video.

[0160] Figure 6b shows a schematic diagram of a second system 72, which includes a video encoder 74 and a video decoder 50 as described herein. The video encoder 74 is configured to encode frames of video into data stream 16 using intra-frame coding and differential inter-frame coding (e.g., using intra-frame coding and differential inter-frame coding for the same frame when using progressive decoding refresh, or using only differential inter-frame coding; e.g., for one or more frames or all frames).

[0161] The video encoder can be any video encoder 10 described herein, but it can also be a video encoder that lacks the ability to measure the quality loss caused by encoding / decoding loss and to signal information about the quality loss as side information in the data stream. The video encoder does not need to measure the quality loss because the video decoder 50 is configured to estimate information about the quality loss caused by encoding / decoding loss.

[0162] Video decoder 50 is the entity that estimates quality loss. Therefore, video encoders with lower complexity (e.g., those without the ability to measure quality loss) can be used. Consequently, video decoder 50 is compatible with a wider range of video encoders.

[0163] Figure 6c shows a schematic diagram of a third system 76, which includes the video decoder 40 or 50 described herein and a display 78 for displaying the reconstructed video decoded by the video decoder 40 or 50.

[0164] Decoder 40 or 50 may be part of display 78, or coupled to or combustible with display 78. Display 78 may be configured to display at an ultra-high definition television resolution (e.g., 4K UHD, 8K UHD, or higher).

[0165] Even at lower transmission speeds, decoders 40 or 50 allow for gradual improvements in video quality.

[0166] Display 78 can be configured to modify a reconstructed version of video 12, thereby presenting a reconstructed version of the video modified with indications that the image loss during the process meets a certain criterion. For example, display 78 can be configured to modify the reconstructed version of the video so as to present a reconstructed version of the video modified with indications of stages using indication 46 as described above. Stages can, for example, indicate whether the video quality is sufficient or insufficient. As described above, these stages can indicate information about quality loss from the data stream using integer or floating-point scales.

[0167] The display 78 can be configured to modify a reconstructed version of video 12 to present a reconstructed version of video 12 modified with indications of a phase of image invariance. The indications of the phase of image invariance can be implemented as described above (e.g., refer to video encoder 10 and video decoder 40).

[0168] According to one embodiment, system 70 includes: a video encoder 10 for encoding video 12, configured to encode frames 14 of video 12 into data stream 16 using intra-frame decoding and differential inter-frame decoding, measure quality loss 18 caused by encoding / decoding loss, and signal information about quality loss 18 as side information 20; and a video decoder 50, 40 for decoding video, configured to decode frames 14 of video 12 from data stream 16 using intra-frame decoding and differential inter-frame decoding, and derive information about quality loss 18 caused by encoding / decoding loss from side information 20 of data stream 16, video 12 being encoded into data stream 16 with encoding / decoding loss.

[0169] According to one embodiment, system 72 includes: a video encoder 74 configured to encode frame 14 of video 12 into data stream 16 using intra-frame encoding and differential inter-frame encoding; and a video decoder 50 for decoding video 12, configured to decode frame 14 of video 12 from data stream 16 where video 12 is encoded with encoding and decoding loss via intra-frame decoding and differential inter-frame decoding, estimate information about quality loss 64 caused by encoding and decoding loss, and output metadata 58 at output interface 56 of video decoder 50 revealing information about image loss, and / or modify the reconstructed version of video 12 obtained by decoding data stream 16 to provide an indication 60 of the stage during which image loss meets a certain criterion for the reconstructed version of video 12.

[0170] According to one embodiment, the system includes: a video decoder 40, 50 for decoding video 12, configured to decode frame 14 of video 12 from data stream 16 using intra-frame decoding and differential inter-frame decoding, and to derive information about quality loss 18 caused by encoding / decoding loss when video 12 is encoded into data stream 16 from side information 20 of data stream 16; or a video decoder 50 for decoding video 12, configured to decode frame 14 of video 12 from data stream 16 where video 12 is encoded with encoding / decoding loss using intra-frame decoding and differential inter-frame decoding, estimate information about quality loss 64 caused by encoding / decoding loss, and output metadata 58 at output interface 56 of video decoder 50 revealing information about image loss, and / or modify the reconstructed version of video 12 obtained by decoding data stream 16 to provide an indication of the stage during which image loss meets a certain criterion for the reconstructed version of video 12; and a display 78 for displaying the reconstructed video 12 obtained by video decoder 50 through decoding.

[0171] Figure 7a shows a flowchart of a method 100 for encoding video.

[0172] Method 100 includes, in step 102, encoding frame 14 of video 12 into data stream 16 using intra-frame coding and differential inter-frame coding (e.g., using intra-frame coding and differential inter-frame coding for the same frame when using progressive decoding refresh, or using only differential inter-frame coding; e.g., for one or more frames or all frames). Method 100 includes, in step 104, measuring the quality loss caused by coding and decoding losses.

[0173] Method 100 can be performed by video encoder 10. Method 100 may include any steps performed by the video encoder 10 disclosed herein.

[0174] Figure 7b shows a flowchart of method 110 for decoding video.

[0175] Method 110 includes, in step 112, decoding frame 14 of video 12 from data stream 16 using intra-frame decoding and differential inter-frame decoding (e.g., using intra-frame encoding and decoding and differential inter-frame decoding for the same frame when using progressive decoding refresh, or using only differential inter-frame decoding; e.g., for one or more frames or all frames). Method 110 also includes, in step 114, deriving information about quality loss due to encoding / decoding loss from side information of data stream 16, wherein video 12 is encoded into data stream 16 with encoding / decoding loss.

[0176] Method 110 can be performed by video decoder 40. Method 110 may include any steps performed by the video decoder 40 disclosed herein.

[0177] Figure 8 A flowchart of method 120 for decoding video is shown.

[0178] Method 120 includes, in step 122, decoding frame 14 of video 12 from data stream 16 using intra-frame decoding and differential inter-frame decoding (e.g., using intra-frame encoding and differential inter-frame encoding for the same frame when using progressive decoding refresh, or using only differential inter-frame encoding; e.g., for one or more frames or all frames), wherein video 12 is encoded into the data stream with encoding / decoding loss. Method 120 includes, in step 124, estimating information about the quality loss caused by the encoding / decoding loss. Method 120 includes, in step 126, outputting metadata revealing information about image loss at output interface 56 of video decoder 50, and / or modifying the reconstructed version of the video obtained by decoding the data stream to provide an indication of the stage during which the image loss met a certain criterion for the reconstructed version of the video.

[0179] Method 120 can be performed by video decoder 50. Method 120 may include any steps performed by the video decoder 50 disclosed herein.

[0180] According to an embodiment of the present invention, a computer program is also provided, which has program code that, when run on a computer, performs method 100 (and optionally, performs any method steps disclosed herein in conjunction with method 100).

[0181] According to an embodiment of the present invention, a computer program having program code is also provided for performing method 110 (and optionally, performing any method steps disclosed herein in conjunction with method 110) when it is run on a computer.

[0182] According to an embodiment of the present invention, a computer program is also provided, which has program code that, when run on a computer, performs method 120 (and optionally, performs any method steps disclosed herein in conjunction with method 120).

[0183] Any computer program disclosed herein (e.g., a computer program for performing methods 100, 110, and 120) may be stored on a non-transient digital storage medium on which the computer program is stored.

[0184] According to embodiments of the present invention, a data stream generated by a method for video coding is also provided, the method comprising encoding frames of a video into the data stream using intra-frame coding and differential inter-frame coding (e.g., using intra-frame coding and differential inter-frame coding for the same frame when using progressive decoding refresh, or using only differential inter-frame coding; e.g., for one or more frames or all frames); measuring quality loss due to coding and decoding losses; and signaling information about the quality loss as side information in the data stream. The data stream may be generated by any other method steps included herein (e.g., refer to video encoder 10). A non-transient digital storage medium may have the data stream stored thereon.

[0185] In implementations of this invention, a compression codec (e.g., at least one of the video encoder 10 and video decoder 40 or 50 described herein) is used in a video transmission system between a computer and a display (with a display frame rate), for example, via a bandwidth-constrained Ethernet connection. The compression codec can improve the quality of static image content from frame to frame until lossless display of the video is achieved (or until a target video quality is reached). For example, the video encoder can signal the (current) achieved video quality, allowing the decoder to indicate to the user when or whether the target video quality has been reached. To enable reliable diagnoses for image evaluation in the medical field, the quality of the image (or video) can be measured and signaled to the viewer at the receiving end, thereby reducing the risk of misdiagnosis. Simultaneously, the bandwidth requirements of the transmission line can be reduced.

[0186] In the JPEG XS version 3, which is being standardized in 2023, temporal differential prediction was proposed to improve the encoding and decoding of screen content or static backgrounds. The video decoder includes a frame buffer and receives data from intra-frame encoding and decoding or data from inter-frame encoding and decoding (residual). Unlike the MPEG standard with fixed IPB encoding and decoding, JPEG XS uses a different mechanism called progressive refresh, in which very small portions of an image (e.g., frame 14) (e.g., one or more subframe portions 32) are refreshed sequentially (intra-frame encoding and decoding), while the rest of the image is only updated (e.g., differential inter-frame encoding and decoding), which means that the difference between the contents of the frame buffer and the original image is transmitted (in the wavelet domain in the case of JPEG XS). This allows for a significant reduction in data rate, but may come at the cost of poorer video quality in the first frame and iterative improvement of image quality for static content (similar to a frame-by-frame progressive encoding and decoding system). The update mechanism (e.g., differential inter-frame encoding and decoding) can progressively improve the image quality of the static image in the frame buffer until it reaches the original image. An example related to continuous quality improvement is shown in the SMPTE paper [3]. The effect of continuously improving image quality can be exemplified in Figure 5b. Some algorithms can improve image quality to infinity, for example, by filling the rectangular curve 68 up to frame number 20 (120dB is the upper limit of the quality shown in Figure 5b, but it can also be infinitely large). If some movement occurs in the image sequence, the quality may degrade.

[0187] The proposed invention can be implemented in a system with an encoder, a bandwidth-limited transmission line (optionally including wireless transmission), and a decoder: the encoder encodes the uncompressed image using a frame-by-frame progressive encoding / decoding system that can reproduce a defined high quality or mathematically lossless quality on the decoder side; the decoder can iteratively improve the image quality of static content (e.g., using differential inter-frame encoding / decoding) while transmitting video or image sequences at the display frame rate. This is combined with a signaling mechanism, for example, signaling if the presented image reaches a specific quality threshold (e.g., dashed line 66 in Figure 5b) or even a mathematically lossless quality. This could be particularly beneficial for medical applications where doctors evaluate medical images. Typically, doctors need at least a few seconds to evaluate an image. In the first few frames after a newly presented image is displayed, the decoded image may not have the necessary quality, but this is not important because a person cannot evaluate the image at the display frame rate. After a short time, the desired quality is achieved. The signaling of quality achievement can be done, for example, directly on the display via red / green lights or externally on a separate device (e.g., a device for optical, acoustic, or haptic feedback).

[0188] In an exemplary embodiment, quality measurement may be combined with the quality characteristics of the display (e.g., having a maximum displayable video quality and / or having frame rate-dependent video quality). For example, signaling notification can be performed not only based on codec quality, but also when the combined quality of the decoded image and the monitor reaches a defined quality.

[0189] In other words, the following content has already been presented above:

[0190] 1. A system for transmitting video to a display at a display frame rate, the system comprising a compression unit at the transmitting end, a bandwidth-limited transmission line, and a decoder unit at the display end, wherein the compression unit encodes an uncompressed image using a frame-by-frame progressive encoding / decoding system, the encoding / decoding system being able to achieve a defined high quality or mathematically lossless quality at the decoder end after several frames of still images, and the decoding unit being able to improve the image quality of the still image content frame by frame. This is combined with a signaling mechanism on the decoder side indicating whether a specific quality has been achieved.

[0191] 2. The system can be applied to medical images to ensure non-destructive quality.

[0192] 3. The signaling system can combine the decoded image quality with the monitor quality to signal the overall quality achieved.

[0193] Alternative solutions for implementation

[0194] Although some aspects are described in the context of an apparatus, they clearly also represent a description of the corresponding method, where modules or apparatus correspond to method steps or features of method steps. Similarly, aspects described in the context of method steps also represent a description of a corresponding module, item, or feature of the corresponding apparatus.

[0195] Typically, embodiments of the present invention can be implemented in the form of a computer program product including program code that operates to perform one of the methods when the computer runs the computer program product. The program code may, for example, be stored on a machine-readable medium.

[0196] Other embodiments include a computer program for performing one of the methods described herein, the computer program being stored on a machine-readable medium.

[0197] In other words, therefore, one embodiment of the method of the present invention is a computer program having program code for performing one of the methods described herein when the computer program is run on a computer.

[0198] Therefore, another embodiment of the method of the present invention is a data carrier (or digital storage medium, or computer-readable medium) having a computer program recorded thereon for performing one of the methods described herein. The data carrier, digital storage medium, or recording medium is typically tangible and / or non-transitory.

[0199] The above embodiments are merely illustrative of the principles of the present invention. It should be understood that modifications and variations to the arrangements and details described herein will be apparent to those skilled in the art. Therefore, the present invention is to be construed only by the scope defined in the forthcoming patent claims, and not by the specific details presented herein through the description and explanation of the embodiments. References: [1]ISO / IEC DIS 21122-1 Ed. 3, Information technology — JPEG XS low-latency lightweight image coding system — Part 1: Core coding system [2]IEC 62563-2 (“Medical electrical equipment – ​​Medical image displaysystems – Part 2: Acceptance and constancy tests for medical image displays”) [3]AAPM On-Line Report No.03 (“Assessment of Display Performance for Medical Imaging Systems”) [4]S. Foessel and T. Richter, "Proposed Extensions to the ThirdEdition of JPEG XS (ISO / IEC 21122) Standard," in SMPTE Motion ImagingJournal, vol. 132, no. 4, pp. 26-32, May 2023, doi: 10.5594 / JMI.2023.3261399.

Claims

1. A video encoder (10) for encoding video (12), configured as follows: The frames (14) of the video (12) are encoded into the data stream (16) using intra-frame encoding and differential inter-frame encoding; Measure the quality loss caused by encoding / decoding loss (18); and In the data stream (16), it is used as edge information (20) to signal information about the quality loss (18).

2. The video encoder (10) according to claim 1 is configured as follows: Measurements and signaling are performed at a spatial granularity corresponding to frame (14) or subframe portion (32); and / or in, The video encoder (10) is configured as follows: Measurements and signaling are performed at a time granularity corresponding to the frame rate of the video (12).

3. The video encoder (10) according to claim 1 or 2 is configured as follows: The quality loss (18) is measured by determining the average of the pixel-by-pixel deviation measure between the lossless version of the video (12) and the reconstructable version that can be reconstructed from the data stream (16).

4. The video encoder (10) according to any one of the preceding claims is configured to: The frame (14) is encoded in the transform domain, and the quality loss (18) is measured by averaging the transform coefficients in the transform domain.

5. The video encoder (10) according to any one of the preceding claims is configured to: As binary information with two states, it is used to signal information about the quality loss (18), where, The first of the two states corresponds to sufficient mass, and the second of the two states corresponds to insufficient mass; or Information about quality loss (18) is signaled using integer or floating-point scales.

6. The video encoder (10) according to any one of the preceding claims is configured to: Differential inter-frame encoding and decoding is performed by subtracting the current frame from the previous encoded / decoded frame to obtain the difference and encoding / decoding the difference into the data stream (16).

7. The video encoder (10) according to any one of the preceding claims is configured to: Progressive decoding refresh is used to spatially reposition the intra-frame codec portion of the video (12) of the frame (14) in the frame region of the codec portion of the frame (14) between frames, and differential inter-frame codec is used outside the intra-frame codec portion.

8. The video encoder (10) according to claim 7 is configured as follows: The frames (14) of the video (12) are encoded using bit rate control, and a constant bit rate is maintained in units of subframe portions (32) into which the frame region is divided, according to the bit rate control.

9. A video decoder (40, 50) for decoding video (12), configured as follows: Frames (14) of the video (12) are decoded from the data stream (16) using intra-frame decoding and differential inter-frame decoding. Information about the quality loss (18) caused by encoding / decoding loss is derived from the side information (20) of the data stream (16), and the video (12) is encoded into the data stream (16) with encoding / decoding loss.

10. The video decoder (40, 50) according to claim 9 is configured as follows: Information about quality loss (18) is derived from the data stream at a spatial granularity corresponding to frame (14) or subframe portion (32); and / or in, The video decoders (40, 50) are configured as follows: Information about quality loss (18) is derived from the data stream (16) at a time granularity corresponding to the frame rate of the video (12).

11. The video decoder (40, 50) according to claim 9 or 10, wherein, Information regarding quality loss (18) represents the average of the pixel-by-pixel deviation measure between the lossless version of video (12) and the reconstructable version that can be reconstructed from the data stream (16).

12. The video decoder (40, 50) according to any one of claims 9 to 11 is configured to: Information about the quality loss (18) is derived from the data stream (16) as binary information with two states, where, The first of the two states corresponds to sufficient mass, and the second of the two states corresponds to insufficient mass; or Information about quality loss (18) is obtained from the data stream (16) using integer or floating-point scales.

13. The video decoder (40, 50) according to any one of claims 9 to 12 is configured to: Differential frame decoding is performed by decoding the difference between the current frame and the previously decoded frame from the data stream (16) and adding the difference to the previously decoded frame.

14. The video decoder (40, 50) according to any one of claims 9 to 13 is configured as follows: Decode the difference between the current frame and the immediately preceding frame in the transform domain from the data stream (16), perform addition in the transform domain to obtain the transform of the current frame, and re-transform the transform.

15. The video decoder (40, 50) according to any one of claims 9 to 14 is configured to: Progressive decoding refresh is used to spatially reposition the intra-frame decoding portion of the video (12) of the frame (14) in the frame region of the decoding portion of the frame (14) between frames, and differential inter-frame decoding is used outside the intra-frame decoding portion.

16. The video decoder (40, 50) according to any one of claims 9 to 15 is configured to: Output metadata (48) revealing information about image loss at the output interface (56) of the video decoder (40); and / or Modify the reconstructed version of the video (12) obtained by decoding from the data stream (16) to provide an indication (46) of the stage during which the image loss meets a certain criterion for the reconstructed version of the video (12).

17. A video decoder (50) for decoding video (12), configured to: Frame 14 of video (12) is decoded from data stream (16) using intra-frame decoding and differential inter-frame decoding, and video (12) is encoded into data stream (16) with encoding / decoding loss; Estimate information about the quality loss caused by encoding / decoding loss (64); and Output metadata (58) revealing information about image loss at the output interface (56) of the video decoder (50), and / or modify the reconstructed version of the video (12) obtained by decoding from the data stream (16) to provide an indication (46) of the stage during which the image loss meets a certain criterion for the reconstructed version of the video (12).

18. The video decoder (50) according to claim 17 is configured to: The following information is used to estimate the quality loss (64) caused by the encoding and decoding loss of the video (12) into the data stream (16): The average level or average energy of the unrefreshed frame portion; or The following combinations: The average level or average energy of the unrefreshed frame portion of the frame whose quality loss (64) is to be estimated; and The average level or average energy of the co-bit unrefreshed frame portion of the subsequent frame or the previous frame after the frame of the information to be estimated (64), wherein the co-bit unrefreshed frame portion is co-bit with the refreshed portion of the frame of the information to be estimated (64).

19. The video decoder (50) according to claim 18 or 17 is configured to: At the output interface (56) of the video decoder (50), as metadata (58) revealing information about image loss, binary information is output regarding whether the quality loss (18) meets a predetermined minimum quality level. If the average level or average energy is below a predetermined threshold, the binary information is set to indicate that the quality loss (18) meets the predetermined minimum quality level; or Modify the reconstructed version of the video (12) obtained by decoding from the data stream (16) to provide an indication (60) for the reconstructed version of the video (12) of a stage in which the image loss during the period meets a certain standard, such that the indication (60) provides binary information about whether the quality loss (18) meets a predetermined minimum quality level, thereby indicating that the quality loss (18) meets the predetermined minimum quality level if the average level or average energy is below a predetermined threshold.

20. The video decoder (50) according to any one of claims 17 to 20 is configured to: Differential frame decoding is performed by decoding the difference between the current frame and the previously decoded frame from the data stream (16) and adding the difference to the previously decoded frame.

21. The video decoder (50) according to claim 20 is configured to: Decode the difference between the current frame and the immediately preceding frame in the transform domain from the data stream (16), perform addition in the transform domain to obtain the transform of the current frame, and re-transform the transform.

22. The video decoder (50) according to any one of claims 17 to 21 is configured to: Progressive decoding refresh is used to spatially reposition the intra-frame decoding portion of the video (12) of the frame (14) in the frame region of the decoding portion of the frame (14) between frames, and differential inter-frame decoding is used outside the intra-frame decoding portion.

23. A system (70) comprising: The video encoder (10) according to any one of claims 1 to 8; and The video decoder (50, 40) according to any one of claims 9 to 16.

24. A system (72) comprising: The video encoder (74) according to any one of claims 1 to 8; and The video decoder (50) according to any one of claims 17 to 22.

25. A system (76) comprising: The video decoder according to any one of claims 9 to 22; and A display (78) is used to display the reconstructed video (12) obtained by the video decoder (40, 50) through decoding.

26. The system according to claim 25, wherein, The display is configured to modify a reconstructed version of the video (12) to present a reconstructed version of the video (12) modified with instructions (46, 60) that the image loss during the period meets a certain criterion; and / or The display is configured to modify a reconstructed version of the video (12) so as to present a reconstructed version of the video (12) modified with the instructions (46, 60) of the phase of image invariance.

27. A method performed by an encoder according to any one of claims 1 to 8.

28. A method performed by a decoder according to any one of claims 9 to 22.

Images