Video coding system and method
The video coding system addresses bandwidth limitations by transmitting encoded frames with high-frequency components, allowing super-resolution processing at the client device to improve video quality efficiently.
Patent Information
- Application Number
- GB2025002690
- Authority / Receiving Office
- GB · GB
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2025-02-25
- Publication Date
- 2026-11-18
AI Technical Summary
The distribution of high-quality video content is hindered by large file sizes and bandwidth requirements, particularly due to limitations in client internet speeds and content distribution server burdens, which are exacerbated by demands for higher resolution and frame rates.
A video coding system that transmits encoded video frames along with high-frequency components, using a convolutional neural network to extract and encode these components, enabling super-resolution processing at the client device to enhance image quality without significantly increasing bandwidth.
Enhances video quality by applying super-resolution processing at the client device, reducing latency and data overhead through efficient transmission and processing of high-frequency components.
Smart Images

Figure 00000000_0000_ABST
Abstract
Description
BACKGROUND OF THE INVENTION Field of the invention This disclosure relates to a video coding system and method. Description of the Prior Art The "background" description provided herein is for the purpose of generally presenting the context of the disclosure. Work of the presently named inventors, to the extent it is described in this background section, as well as aspects of the description which may not otherwise qualify as prior art at the time of filing, are neither expressly nor impliedly admitted as prior art against the present invention. In recent years there has been a growing demand for video content having a higher quality; this includes both pre-generated content, such as television programmes and movies, and content being rendered in real-time such as video games. This is often reflected in demands for higher resolution, higher frame rates, or improved colour reproduction, for example. While such content can be made available, difficulties may arise in the distribution of that content -retaining the original level of quality can mean that file sizes or required streaming bandwidths become larger than is practical for distribution via a network. This can be due to limitations on a client's internet speeds, for example, or a significant burden being placed upon a content distribution server. It is therefore desirable that improvements to the quality of video content are provided without introducing a significant data overhead. One method for addressing this is to perform post-processing to upscale video content at the client device prior to display. One example of such a process is super-resolution processing; this is a process which generates a higher-resolution image from an input low-resolution image. This is typically achieved via use of a trained machine learning model, such as a convolutional neural network, implemented at the client device. This therefore enables a higher-resolution image to be displayed than the one which is transmitted, increasing video quality without a corresponding increase to the bandwidth requirements for transmitting the video. It is in the context of the above discussion that the present disclosure arises. SUMMARY OF THE INVENTION This disclosure is defined by claim 1. Further respective aspects and features of the disclosure are defined in the appended claims. It is to be understood that both the foregoing general description of the invention and the following detailed description are exemplary, but are not restrictive, of the invention. BRIEF DESCRIPTION OF THE DRAWINGS A more complete appreciation of the disclosure and many of the attendant advantages thereof will be readily obtained as the same becomes better understood by reference to the following detailed description when considered in connection with the accompanying drawings, wherein: Figure 1 schematically illustrates an entertainment system; Figure 2 schematically illustrates a networked system; Figure 3 schematically illustrates an image frame coding process; Figure 4 schematically illustrates a method corresponding to the process described with reference to Figure 3; Figure 5 schematically illustrates an encoding device; Figure 6 schematically illustrates a decoding device; Figure 7 schematically illustrates an encoding method; and Figure 8 schematically illustrates a decoding method. DESCRIPTION OF THE EMBODIMENTS Referring now to the drawings, wherein like reference numerals designate identical or corresponding parts throughout the several views, embodiments of the present disclosure are described. Referring to Figure 1, an example of an entertainment system 10 is a computer or console. The entertainment system 10 comprises a central processor or CPU 20. The entertainment system also comprises a graphical processing unit or GPU 30, and RAM 40. Two or more of the CPU, GPU, and RAM may be integrated as a system on a chip (SoC). Further storage may be provided by a disk 50, either as an external or internal hard drive, or as an external solid state drive, or an internal solid state drive. The entertainment device may transmit or receive data via one or more data ports 60, such as a USB port, Ethernet® port, Wi-Fi® port, Bluetooth® port or similar, as appropriate. It may also optionally receive data via an optical drive 70. Audio / visual outputs from the entertainment device are typically provided through one or more A / V ports 90 or one or more of the data ports 60. Where components are not integrated, they may be connected as appropriate either by a dedicated data link or via a bus 100. An example of a device for displaying images output by the entertainment system is a head mounted display 'HMD' 120, worn by a user 1. Interaction with the system is typically provided using one or more handheld controllers 130, and / or one or more VR controllers (130A-L,R) in the case of the HMD. Figure 2 schematically illustrates a networked system in accordance with implementations of the present disclosure. In this Figure, a local device 200 is shown in communication, via a network represented by the line, with a remote device 210. The local device 200 may be the entertainment system 10 of Figure 1, for example, or may be any other device suitable for receiving video content for display via a network. For instance, the local device 200 may be a television or other display (such as an HMD) which is configured to obtain video content over a network directly, or another computing device such as a personal computer, laptop, portable games console, or mobile phone. The remote device 210 may be any suitable processing device, such as a personal computer, games console, or server, which is configured to stream video content to the local device 200. This may be as a part of a remote video hosting service, for instance, or as a part of a user's home media distribution setup. The video content may be a video, such as a television programme or a movie, or it may be a video game stream or the like (such as in the case that the remote device 210 is part of a cloud gaming arrangement). In implementations according to the present disclosure, video content is transmitted in conjunction with information about high frequency components of one or more of the image frames in the video content. This additional information is used in addition to a super-resolution process performed by the client device which is receiving the video content for display; the result is a display image with an improved quality, with only a minimal overhead being introduced in the video distribution. Figure 3 schematically illustrates a process in accordance with implementations according to the present disclosure. In this Figure, the input video frame, hnput 300, is encoded to obtain the encoded video frame I* 310. This encoding may be in accordance with any suitable video coding scheme, such as HEVC, as no change to the coding is required to implement this process. Separately, the high frequency components 320 are extracted from the input video frame 300. The encoded video frame 310 and high frequency components 320 are each transmitted to the client device. 3 Once received, the encoded video frame 310 is decoded to obtain the decoded image frame l# 330. The decoded image frame 330 is then subjected to super-resolution processing and any other postprocessing that is desired (such as deblocking processing). Separately to these processes, the high frequency components 320 are added to the processed decoded frame to obtain the output frame loutput 340. The high frequency components represent discontinuities in the image frame hnput, such as sharp edges or colour changes within an image. These can be encoded spatially, for instance as an image which comprises only edges, in a manner that allows the high frequency components to be represented efficiently (for instance, through an efficient compression). This may be a single channel of information, or may comprise three channels of information in some implementations. Each of these channels corresponds to a different component of the image - in a single-channel implementation this may be based upon luminance, while in a three-channel implementation the additional channels can correspond to information derived from respective colour channels. The high frequency components can be obtained from an image frame using any suitable techniques; it is preferable that the extraction technique that is selected is one which is fast, so as to reduce any latency that may be introduced. A convolutional neural network may be utilised for the extraction of high frequency features, with examples of suitable techniques including Sobel filters, image derivatives, and Laplacian operators. Figure 4 schematically illustrates a method corresponding to the process described with reference to Figure 3. This method includes the encoding of a frame of video content and extraction of the high frequency component, and then a reconstruction of the frame at a client device. As such, this method is typically implemented by an encoding device and a decoding device (such as a server and a client) in communication with one another. A step 400 comprises encoding the image frame; this may be in accordance with any preferred video coding scheme. The image frame may be a part of any video output that is to be provided to the client -this includes both pre-existing video content (such as television programmes) and interactive content such as video games. In the latter case, this may be as a part of a cloud gaming arrangement, for example, or a local streaming arrangement in which a games console is transmitting video to another device (such as a mobile phone or portable games console). A step 410 comprises extracting high frequency components from the image frame and encoding them. As noted above, this is typically performed by a trained neural network that is configured to perform this function - this may be a convolutional neural network that is based upon a Sobel filter or Laplacian operator, for instance. This step may be performed before or after the encoding of the image frame, or 4 it may be performed substantially simultaneously in some instances. The extracted high frequency components may subsequently be processed to generate data for output - for instance, the components may be compressed to reduce the corresponding data overhead for transmitting them. A step 420 comprises transmitting both the encoded image frame and the high frequency components. The high frequency components may be transmitted in any suitable manner; for instance, as a separate stream to the encoded image frames, or as extra packages which form a part of the same stream. The transmission may be via an internet connection, for instance, or via a local network for a local streaming implementation. A step 430 comprises decoding, at the client device, the encoded image frame which is received. A step 440 comprises applying processing the decoded image frame; this can include a super-resolution or other upsampling process. Any other post-processing on the image frame may also be performed, as desired. A step 450 comprises applying high frequency components to the processed image frame that is output by step 440. Finally, the generated images may be presented to one or more viewers via a display device associated with the client. Alternatively, the decoded video content may be stored for use at a later time. Such a method may be implemented in a dynamic fashion, with some frames being supplemented with the high frequency component data as discussed while others are not. The decision about which frames should be supplemented can be made on the basis of a number of different criteria. A first example is that of when the latency of the network connection is high, such that a significant delay is present between the transmission and display of video content. In such a case, it may be preferred that the process is switched off so as to reduce the latency; while the latency associated with implementations of this method is considered to be small, any reduction in overall latency may be particularly desirable when the latency exceeds a threshold value. Similar concerns can also stem from the client device having limited processing power, in which case as processing load exceeds a threshold level the processing required to apply the high frequency components can be omitted. A second example is that of when the network bandwidth is low; while the extra data associated with the transmission of the high frequency components is expected to be very small, in cases where bandwidth is particularly low it may be preferred to minimise the amount of data that is being transmitted. A third example is in the case that the high frequency components do not comprise a significant amount of information - such as when viewing a scene with little or no fine detail. In this case, the improvement to the final image may be sufficiently low so as to not justify the additional processing and data transmission. It is also considered that the high frequency component data can be generated and applied on a per-region basis; for instance, a region corresponding to a user's gaze direction in an image should such information be available, or a presumed / estimated gaze location in the case that it is not. Similarly, a saliency map or the like may be used to identify regions likely to be of particular interest to a viewer (and therefore more likely to be viewed) to enable the targeting of the transmitted high frequency data. Information may also be transmitted on the basis of the high frequency component data itself; by dividing this information into regions, any region with a below-threshold amount of high frequency data (such as a region having few edges) may be omitted from the encoded / transmitted high frequency component data. Figure 5 schematically illustrates an encoding device in accordance with implementations of the present disclosure. The system for encoding video content 500 comprises an encoding unit 510, an extraction unit 520, and a transmission unit 530. The system 500 may be embodied by any processing device which is configured to transmit content to another device; this may be a media or cloud gaming server, for instance. The encoding unit 510 is configured to encode an image frame of the video content, as a part of a video encoding process in which a plurality of image frames (that is, a video) will be encoded. This encoding may be in accordance with any preferred video codec; no limitations are placed upon the encoding in the present disclosure as the operations are performed upon the decoded image content (and therefore are not responsive to a particular coding scheme). The extraction unit 520 is configured to extract, using a convolutional neural network, high frequency component information from the image frame. The extraction unit 520 is also configured to encode the extracted information into a preferred format for transmission. Optionally, the extraction unit 520 may compress the high frequency component information prior to its transmissions - this may be before encoding, as a part of the encoding, and / or a compression of the encoded high frequency component information. The format of the high frequency component information may be determined freely; in some instances this is encoded spatially (for example, as an image or map showing only those components), while in others an alternative representation may be used. The high frequency component information may comprise a single channel, or may use more channels (such as three). This may be determined on a per-6 implementation, per-video, or per-frame basis. For example, this determination may be responsive to the amount of high frequency component information that is generated (in that if the amount of information is lower, more channels may be used without generating a significant burden upon the transmission of the information). The convolutional neural network may use a Sobel filter or Laplacian operator to extract the high frequency component information from the image frame; however any appropriate filter may be used. The threshold at which the frequencies are considered to be high may be determined freely for a given implementation; this may be dependent upon characteristics of the coding scheme used for the image frames, for instance, or the super-resolution process that is to be performed. Specific implementation details may be selected so as to balance the data size of the high frequency component information and the benefit to the image frame upon reconstruction by a decoder. As described above, the operation of the extraction unit 520 may be managed in a dynamic fashion, such that some image frames are subjected to this processing while others are not. This may be due to any of a number of factors which affect a given implementation, examples of which include: • the extraction unit 520 being configured to extract high frequency component information when a latency associated with the network is below a threshold value; • the extraction unit 520 being configured to extract high frequency component information when a bandwidth associated with the network is above a threshold value; and / or • the extraction unit 520 being configured to extract high frequency component information when the processing load upon the decoding device is above a threshold value (for instance, responsive to information output by the decoding device about its processing load). In some cases, the extraction may still be performed despite the conditions being met, but instead the high frequency component information is not transmitted - while this is less efficient than not extracting the information, this enables the decision of whether to transmit the information to be made at a later time. This therefore results in a more responsive implementation. In some implementations, the extraction unit 520 may be configured to extract high frequency component information for a portion of the image frame which is less than the whole image frame. This portion of the image frame may be one which corresponds to a user's gaze direction (for instance, determined from images captured by inwards-facing cameras in an HMD worn by the user), for example. Alternatively, or in addition, the portion may correspond to one or more image features or regions identified as being of particular interest within a scene on the basis that these are likely to be gazed at by viewers. The transmission unit 530 configured to transmit the encoded image frame and the high frequency component information to a decoding device via a network. This may be a local network, for example, or via the internet, and accordingly the transmission unit 530 comprises hardware for enabling such communication. The arrangement of Figure 5 is an example of a processor (for example, a GPU and / or CPU located in a games console, server, or any other computing device) that is operable to encode video content, and in particular is operable to: encode an image frame of the video content; extract, using a convolutional neural network, high frequency component information from the image frame; encode the extracted information; and transmit the encoded image frame and the high frequency component information to a decoding device via a network. The arrangement of Figure 5 may be embodied by the remote device 210 of Figure 2, which may comprise any suitable arrangement of CPUs and / or GPUs for providing the described functionality. Figure 6 schematically illustrates a decoding device in accordance with implementations of the present disclosure. The system for decoding video content 600 comprises a reception unit 610, a decoding unit 620, a super-resolution unit 630, and a processing unit 640. The system 600 may be embodied by any processing device which is configured to receive content from another device; the system 600 may be a games console, personal computer, or display device, for instance. The reception unit 610 is configured to receive an encoded image frame and corresponding high frequency component information via a network; this is the output of a system for encoding video content as discussed with reference to Figure 5, for example. The network may be a local network, for example, or the internet, and accordingly the reception unit 610 comprises hardware for enabling such communication. The decoding unit 620 is configured to decode the encoded image frame and the high frequency component information. The operation of the decoding unit 620 is defined in accordance with the specific video coding scheme used for the encoded image frame. The decoding unit 620 may also be configured to perform decompression of the high frequency component information where required, either before, during, or after the decoding of this information as appropriate. The super-resolution unit 630 is configured to apply a super-resolution process to the decoded image frame to obtain an enhanced image frame. The super-resolution unit 630 may also be configured to perform other post-processing on the decoded image, either before or after the application of the super-resolution processing as appropriate. The processing unit 640 is configured to apply the high frequency component information to the enhanced image frame. This may be performed by adding the high frequency component information to the enhanced image frame as an extra layer, for instance. The arrangement of Figure 6 is an example of a processor (for example, a GPU and / or CPU located in a games console or any other computing device) that is operable to decode video content, and in particular is operable to: receive an encoded image frame and corresponding high frequency component information via a network; decode the encoded image frame and the high frequency component information; apply a super-resolution process to the decoded image frame to obtain an enhanced image frame; and apply the high frequency component information to the enhanced image frame. The arrangement of Figure 6 may be embodied by the local device 200 of Figure 2, which may comprise any suitable arrangement of CPUs and / or GPUs for providing the described functionality. Figure 7 schematically illustrates an encoding method in accordance with implementations of the present disclosure. This process may be implemented by the system of Figure 5, for instance. A step 700 comprises encoding an image frame of the video content. A step 710 comprises extracting, using a convolutional neural network, high frequency component information from the image frame. A step 720 comprises encoding the extracted information. A step 730 comprises transmitting the encoded image frame and the high frequency component information to a decoding device via a network. Figure 8 schematically illustrates a decoding method in accordance with implementations of the present disclosure. This process may be implemented by the system of Figure 6, for instance. A step 800 comprises receiving an encoded image frame and corresponding high frequency component information via a network. A step 810 comprises decoding the encoded image frame and the high frequency component information. A step 820 comprises applying a super-resolution process to the decoded image frame to obtain an enhanced image frame. A step 830 comprises applying the high frequency component information to the enhanced image frame. The techniques described above may be implemented in hardware, software or combinations of the two. In the case that a software-controlled data processing apparatus is employed to implement one or more features of the embodiments, it will be appreciated that such software, and a storage or transmission medium such as a non-transitory machine-readable storage medium by which such software is provided, are also considered as embodiments of the disclosure. Thus, the foregoing discussion discloses and describes merely exemplary embodiments of the present invention. As will be understood by those skilled in the art, the present invention may be embodied in other specific forms without departing from the spirit or essential characteristics thereof. Accordingly, the disclosure of the present invention is intended to be illustrative, but not limiting of the scope of the invention, as well as other claims. The disclosure, including any readily discernible variants of the teachings herein, defines, in part, the scope of the foregoing claim terminology such that no inventive subject matter is dedicated to the public.
Claims
1. A system for encoding video content, the system comprising:an encoding unit configured to encode an image frame of the video content;an extraction unit configured to extract, using a convolutional neural network, high frequency component information from the image frame and to encode the extracted information; anda transmission unit configured to transmit the encoded image frame and the high frequency component information to a decoding device via a network.
2. A system according to claim 1, wherein the convolutional neural network uses a Sobel filter or Laplacian operator to extract high frequency component information.
3. A system according to any preceding claim, wherein the extraction unit is configured to compress the high frequency component information prior to transmission.
4. A system according to any preceding claim, wherein the high frequency component information comprises single-channel or three-channel information.
5. A system according to any preceding claim, wherein the extraction unit is configured to extract high frequency component information when a latency associated with the network is below a threshold value.
6. A system according to any preceding claim, wherein the extraction unit is configured to extract high frequency component information when a bandwidth associated with the network is above a threshold value.
7. A system according to any preceding claim, wherein the extraction unit is configured to extract high frequency component information when the processing load upon the decoding device is above a threshold value.
8. A system according to any preceding claim, wherein the extraction unit is configured to extract high frequency component information for a portion of the image frame which is less than the whole image frame.
9. A system for decoding video content, the system comprising:a reception unit configured to receive an encoded image frame and corresponding high frequency component information via a network;a decoding unit configured to decode the encoded image frame and the high frequency component information;a super-resolution unit configured to apply a super-resolution process to the decoded image frame to obtain an enhanced image frame; anda processing unit configured to apply the high frequency component information to the enhanced image frame.
10. A method for encoding video content, the method comprising:encoding an image frame of the video content;extracting, using a convolutional neural network, high frequency component information from the image frame;encoding the extracted information; andtransmitting the encoded image frame and the high frequency component information to a decoding device via a network.
11. A method for decoding video content, the method comprising:receiving an encoded image frame and corresponding high frequency component information via a network;decoding the encoded image frame and the high frequency component information;applying a super-resolution process to the decoded image frame to obtain an enhanced image frame; andapplying the high frequency component information to the enhanced image frame.
12. Computer software comprising instructions which, when the software is executed by a computer, causes the computer to carry out the method of claim 10 or claim 11.
13. A non-transitory machine-readable storage medium which stores computer software according to claim 12.
Citation Information
Patent Citations
ViewUS2025/0005708A1onEspacenetopensinnewtab
ViewWO2019/093234A1onEspacenetopensinnewtab
