Reducing latency in networked games by reducing i-frame size

Abstract

Techniques are described for reducing latency in networked games by reducing (304) I frame size (which also results in an automatic increase in P frame size) to reduce the total amount of video being transmitted (308). The size of other frames is increased (306) by using a low pass filter (LPF) (402), such as a Gaussian filter that reduces the sharpness that a decoder may attempt to recover, or the reduced size of I frames is compensated by using a lower resolution (502). The I-frame may be reduced (602) by rotating the I-frame or flipping / mirroring the I-frame to produce a smaller encoded frame, a flag being sent to signal the orientation.

Description

Technical Field

[0001] This application relates to technically inventive, unconventional solutions that must be rooted in computer technology and produce specific technical improvements, and more specifically to reducing latency in online games by reducing the size of I-frames. Background Technology

[0002] Computer simulation videos, such as computer game videos, can be streamed over a network to end-user terminals. Summary of the Invention

[0003] As understood herein, network regulations imposed by network conditions and / or rules regarding bandwidth limitations for energy conservation can restrict the network channels available for transmitting video such as computer game videos. As further understood herein, latency is a primary concern under these conditions, particularly in the case of computer game players, because gamers prefer near-instantaneous responses to their inputs, such as when shooting game weapons. Therefore, video quality may be less important compared to transmitting video with little or no latency.

[0004] In a first aspect, an apparatus includes at least one processor component configured to identify at least one condition of at least one network. The processor component is configured to, in response to the condition, decrease the size of at least one I-frame of video and optionally increase the size of at least one non-I-frame of video, then send the I-frame and non-I-frame to a transmitter for transmission over the network.

[0005] Videos can include computer game videos.

[0006] In the example, conditions may include latency and / or bandwidth.

[0007] In some embodiments, the processor component may be configured to reduce the size of the I-frame by at least partially processing it through a filter. Alternatively, the processor component may be configured to reduce the size of the I-frame by at least partially reducing its resolution. Furthermore, or alternatively, the processor component may be configured to reduce the size of the I-frame by at least partially changing its orientation, and may signal the orientation of the I-frame if necessary.

[0008] In a non-limiting implementation, the I-frame can be a first I-frame, and the processor component can be configured to reduce the size of the first I-frame in response to network conditions and in response to the first I-frame having a first importance, and not reduce the size of the second I-frame in response to network conditions and in response to the second I-frame having a second importance.

[0009] On the other hand, the apparatus includes at least one computer medium that is not a transient signal, and further includes instructions executable by at least one processor component to reduce a first portion of the video before encoding using at least a first filtering parameter. The instructions are executable to reduce a second portion of the video before encoding using at least a second filtering parameter not used to reduce the first portion, and to transmit these portions via a network to at least one receiver.

[0010] In this respect, in the example, the instruction could be executable signaling to transmit the portion via a network and to indicate the first and second filter parameters. The first and second portions of the video could be keyframes of the video, such as I-frames or Instant Decoding Refresh (IDR) frames. Alternatively, the first and second portions could be parts of a keyframe of the video.

[0011] In the example embodiment, the first filter parameter and the second filter parameter may include a corresponding first filter type and a corresponding second filter type, and / or a corresponding first filter strength and a corresponding second filter strength.

[0012] In another aspect, a method includes receiving, via a network, at least a portion of at least a first keyframe of at least one video. The method also includes receiving, via the network, signaling instructing reconstruction of that portion of the keyframe to be applied. The method includes reconstructing that portion of the keyframe based on the signaling before presenting it on a video display.

[0013] In some examples, the reconstruction includes magnifying that portion of the keyframe, and / or processing that portion of the keyframe using at least one sharpening filter, and / or redirecting that portion of the keyframe.

[0014] The details of this disclosure regarding its structure and operation can be best understood with reference to the accompanying drawings, wherein like reference numerals denote like parts, and wherein: Attached Figure Description

[0015] Figure 1 It is a block diagram of an example system that includes examples consistent with this principle;

[0016] Figure 2 An example encoder-decoder system is shown;

[0017] Figure 3 The example coding logic is shown in the format of an example flowchart;

[0018] Figure 4 An example filtering technique for reducing keyframe size is shown in the form of an example flowchart;

[0019] Figure 5 An example resolution reduction technique for reducing keyframe size is shown in an example flowchart format;

[0020] Figure 6 An example orientation rotation technique for reducing keyframe size is shown in an example flowchart format;

[0021] Figure 7 It shows the relationship with Figure 6 Rotate keyframes in a consistent manner;

[0022] Figure 8 The example flowchart illustrates the process for implementing it. Figure 4 Decoder-side technology for video;

[0023] Figure 9 The example flowchart illustrates the process for implementing it. Figure 5 Decoder-side technology for video;

[0024] Figure 10 The example flowchart illustrates the process implemented on it. Figure 6 Decoder-side technology for video;

[0025] Figure 11 The example flowchart illustrates how to reduce the keyframe size based on the importance of each frame;

[0026] Figure 12 The pipeline for reducing keyframes piece by piece and encoding each piece once it has been reduced for transmission is shown in the example flowchart format.

[0027] Figure 13A The flowchart is shown in the example flowchart format. Figure 12 The technology is consistent with the decoder-side pipeline used for processing video;

[0028] Figure 13B The flowchart is shown in the example flowchart format. Figure 12 The technology is consistent with that of an alternative decoder-side pipeline for processing video;

[0029] Figure 14 The technique for reducing video segments based on pre-configuration of video files is illustrated in the example flowchart format;

[0030] Figure 15 The technique for reducing video clips based on signaling from a computer game engine is illustrated in an example flowchart format.

[0031] Figure 16 The example flowchart illustrates the selection of filters for each video segment based on its content.

[0032] Figure 17The flowchart is shown in the example flowchart format. Figure 16 The decoder-side technology used for processing video is consistent with the technology used in the process.

[0033] Figure 18 It shows the relationship with Figure 17 Consistent tables;

[0034] Figure 19 The reduction techniques, which involve reducing only chromaticity information, are illustrated in an example flowchart format; and

[0035] Figure 20 The flowchart is shown in the example flowchart format. Figure 19 The technology is consistent with the decoder-side technology used for processing video. Detailed Implementation

[0036] This disclosure generally relates to the computer ecosystem, including various aspects of consumer electronics (CE) device networks, such as, but not limited to, computer gaming networks. Systems herein may include server and client components that can be networked to allow data exchange between client and server components. Client components may include one or more computing devices, including game consoles such as the Sony PlayStation® or game consoles manufactured by Microsoft, Nintendo, or other manufacturers; extended reality (XR) headsets such as virtual reality (VR) headsets; augmented reality (AR) headsets; portable televisions (e.g., smart TVs, internet-enabled televisions); portable computers such as laptops and tablets; and other mobile devices including smartphones and other examples discussed below. These client devices may operate in a variety of operating environments. For example, by way of example, some client computers may run on a Linux operating system, an operating system from Microsoft, or a Unix operating system, or an operating system manufactured by Apple or Google, or a Berkeley Software distribution or Berkeley Standard Distribution (BSD) OS (including descendants of BSD). These operating environments may be used to execute one or more browsing programs, such as browsers or other browser programs manufactured by Microsoft, Google, or Mozilla, which can access websites hosted by internet servers discussed below. Furthermore, the operating environment based on this principle can be used to execute one or more computer game programs.

[0037] A server and / or gateway may be used, which may include one or more processors executing instructions that configure the server to receive and send data over a network such as the Internet. Alternatively, the client and server may connect via a local intranet or virtual private network. The server or controller may be instantiated from a game console such as a Sony PlayStation®, a personal computer, etc.

[0038] Information can be exchanged between clients and servers over a network. For this purpose, and for security, servers and / or clients may include firewalls, load balancers, temporary storage, and proxies, as well as other network infrastructure for reliability and security. One or more servers can form a means of implementing methods to provide network members with a secure community, such as an online social networking site or a gaming network.

[0039] A processor can be a single-chip or multi-chip processor that executes logic via various lines such as address lines, data lines, and control lines, as well as registers and shift registers. A processor that includes a digital signal processor (DSP) can be an embodiment of a circuit. A processor assembly can include one or more processors.

[0040] Components included in one embodiment may be used in other embodiments in any suitable combination. For example, any of the various components described herein and / or depicted in the accompanying drawings may be combined, interchanged, or excluded from other embodiments.

[0041] "A system having at least one of A, B and C" (similarly, "a system having at least one of A, B or C" and "a system having at least one of A, B and C") includes systems having only A, only B, only C, A and B together, A and C together, B and C together and / or A, B and C together.

[0042] Now for reference Figure 1 An example system 10 is illustrated, which may include one or more example devices described above and further below according to the principles of the invention. A first example device included in system 10 is a consumer electronics (CE) device, such as an audio-visual device (AVD) 12, which may be, for example, but is not limited to, a projector-based cinema display system, or an internet-enabled television with a television tuner (equivalent to a set-top box for controlling the television). Alternatively, the AVD 12 may also be a computerized internet-enabled (“smart”) phone, tablet computer, laptop computer, head-mounted device (HMD) and / or a headset such as smart glasses or VR headsets, another wearable computerized device, a computerized internet-enabled music player, a computerized internet-enabled headset, a computerized internet-enabled implantable device, such as an implantable skin device, etc. In any case, it should be understood that the AVD 12 is configured to implement the principles of the invention (e.g., to communicate with other CE devices to implement the principles of the invention, to perform the logic described herein, and to perform any other functions and / or operations described herein).

[0043] Therefore, to implement this principle, the AVD 12 can be constructed from some or all of the components shown. For example, the AVD 12 may include one or more touch-enabled displays 14, which may be implemented using a high-definition or ultra-high-definition "4K" or higher flat panel screen. The touch-enabled displays 14 may include, for example, a capacitive or resistive touch sensing layer having an electrode grid for touch sensing consistent with this principle.

[0044] AVD 12 may also include one or more speakers 16 for outputting audio according to these principles, and at least one additional input device 18, such as an audio receiver / microphone, for inputting audible commands to control AVD 12. Example AVD 12 may also include one or more network interfaces 20 for communicating over at least one network 22 (e.g., the Internet, WAN, LAN, etc.) under the control of one or more processors 24. Thus, interface 20 may be, but is not limited to, a Wi-Fi transceiver, which is an example of a wireless computer network interface, such as, but not limited to, a mesh network transceiver. It should be understood that processor 24 controls AVD 12 to perform these principles, including other elements of AVD 12 described herein, such as controlling display 14 to present images thereon and receiving input from it. Furthermore, it should be noted that network interface 20 may be a wired or wireless modem or router, or other suitable interface, such as a wireless telephone transceiver or a Wi-Fi transceiver as described above.

[0045] In addition to the above, the AVD 12 may also include one or more input and / or output ports 26, such as an HDMI port or a USB port, for physical connection to another CE device and / or a headphone port for connecting headphones to the AVD 12 to present audio from the AVD 12 to the user via headphones. For example, input port 26 may be connected via a cable or satellite source 26a to audio / video content, either wired or wirelessly. Therefore, source 26a may be a standalone or integrated set-top box or satellite receiver. Alternatively, source 26a may be a game console or disc player containing content. When implemented as a game console, source 26a may include some or all of the components described below with respect to CE device 48.

[0046] AVD 12 may also include one or more computer memory / computer-readable storage media 28, such as non-transient signal disk storage or solid-state storage, which in some cases are embodied in the chassis of the AVD as a standalone device or personal video recording device (PVR) or video disk player, located inside or outside the AVD chassis for playing AV programs, or as a removable storage medium or a server as described below. Furthermore, in some embodiments, AVD 12 may include a location or place receiver, such as, but not limited to, a mobile phone receiver, a GPS receiver, and / or an altimeter 30, configured to receive geolocation information from a satellite or mobile phone base station and provide that information to the processor 24 and / or determine the height at which the AVD 12 is placed with the processor 24.

[0047] Continuing the description of AVD 12, in some embodiments, AVD 12 may include one or more cameras 32, which may be thermal imaging cameras, digital cameras such as webcams, IR sensors, event-based sensors, and / or cameras integrated into AVD 12 and controllable by processor 24 to collect pictures / images and / or videos in accordance with these principles. AVD 12 may also include a Bluetooth® transceiver 34 and other near-field communication (NFC) elements 36 for communicating with other devices using Bluetooth and / or NFC technologies, respectively. An example NFC element may be a radio frequency identification (RFID) element.

[0048] In addition, the AVD 12 may include one or more auxiliary sensors 38 that provide input to the processor 24. For example, the one or more auxiliary sensors 38 may include one or more pressure sensors forming a layer of the touch-enabled display 14 itself, and may be, but are not limited to, piezoelectric pressure sensors, capacitive pressure sensors, piezoresistive strain gauges, optical pressure sensors, electromagnetic pressure sensors, etc. Other sensor examples include pressure sensors, motion sensors such as accelerometers, gyroscopes, cyclometers, or magnetometers, infrared (IR) sensors, optical sensors, speed and / or rhythm sensors, event-based sensors, and gesture sensors (e.g., for sensing gesture commands). Thus, the sensor 38 may be implemented by one or more motion sensors, such as separate accelerometers, gyroscopes, and magnetometers and / or inertial measurement units (IMUs), which typically include a combination of accelerometers, gyroscopes, and magnetometers to determine the position and orientation of the AVD 12 in three dimensions, or by event-based sensors, such as event detection sensors (EDS). An EDS conforming to this disclosure provides an output light sensing array that indicates changes in light intensity sensed by at least one pixel. For example, if the light sensed by the pixels is decreasing, the EDS output can be -1; if it is increasing, the EDS output may be +1. Outputting a binary signal of 0 can indicate that the light intensity below a certain threshold has not changed.

[0049] AVD 12 may also include an over-the-air (OTA) television broadcast port 40 for receiving OTA television broadcasts that provide input to processor 24. In addition to the foregoing, it should be noted that AVD 12 may also include an infrared (IR) transmitter and / or an IR receiver and / or an IR transceiver 42, such as an IR data association (IRDA) device. A battery (not shown) may be provided to power AVD 12, such as a kinetic energy harvester that can convert kinetic energy into electrical energy to charge the battery and / or power AVD 12. A graphics processing unit (GPU) 44 and a field-programmable gate array (FPGA) 46 may also be included. One or more tactile / vibration generators 47 may be provided to generate tactile signals that can be felt by a person holding or touching the device. Thus, the tactile generator 47 may use an electric motor to vibrate all or part of AVD 12, the motor being connected via a rotatable shaft to an eccentric and / or unbalanced weight, such that the shaft can rotate under the control of the motor (which may in turn be controlled by a processor such as processor 24), to generate vibrations of various frequencies and / or amplitudes, as well as force simulations in various directions.

[0050] It may also include a light source such as a projector (such as an infrared (IR) projector).

[0051] In addition to AVD 12, system 10 may include one or more other CE device types. In one example, the first CE device 48 may be a computer game console that can be used to send computer game audio and video to AVD 12 via commands sent directly to AVD 12 and / or via a server described below, while the second CE device 50 may include components similar to the first CE device 48. In the example shown, the second CE device 50 may be configured as a computer game controller operated by a player or a head-mounted display (HMD) worn by a player. The HMD may include transparent or opaque heads-up displays for presenting AR / MR content or VR content (more generally, extended reality (XR) content), respectively. The HMD may be configured as a glasses-type display or a larger VR-type display sold by a computer game device manufacturer.

[0052] In the example shown, only two CE devices are illustrated; it should be understood that fewer or more devices may be used. The devices described herein may implement some or all of the components shown for AVD 12. Any components shown in the following figures may be combined with some or all of the components shown in the case of AVD 12.

[0053] Referring now to at least one server 52 described above, it includes at least one server processor 54, at least one tangible computer-readable storage medium 56 (such as a disk-based or solid-state storage device), and at least one network interface 58, which, under the control of the server processor 54, allows communication with other illustrated devices via network 22 and, in practice, facilitates communication between server and client devices according to this principle. Note that the network interface 58 may be, for example, a wired or wireless modem or router, a Wi-Fi transceiver, or other suitable interface, such as a wireless telephone transceiver.

[0054] Therefore, in some embodiments, server 52 may be an internet server or an entire server "group" and may include and perform "cloud" functionality, enabling devices of system 10 to access a "cloud" environment via server 52 in the example embodiment for, for example, online gaming applications. Alternatively, server 52 may be implemented by one or more game consoles or other computers in the same room or nearby as the other devices shown.

[0055] The components shown in the following figures may include some or all of the components shown herein. Any user interface (UI) described herein may be combined and / or extended, and UI elements may be mixed and matched between UIs.

[0056] This principle can be applied to various machine learning models, including deep learning models. Machine learning models consistent with this principle can be trained using a variety of algorithms, including supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, feature learning, self-learning, and other forms of learning. Examples of such algorithms that can be implemented by computer circuits include one or more neural networks, such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), and RNN types known as long short-term memory (LSTM) networks. Generative pre-trained transformers (GPTTs) can also be used. Support vector machines (SVMs) and Bayesian networks can also be considered examples of machine learning models. In addition to the network types described above, the models in this paper can be implemented using classifiers.

[0057] As understood in this paper, performing machine learning can therefore involve accessing and then training a model on training data so that the model can process further data to make inferences. Thus, an artificial neural network / AI model trained via machine learning can include an input layer, an output layer, and multiple hidden layers in between, which are configured and weighted to make inferences about the appropriate output.

[0058] Figure 2 A system including a video encoder 200 for encoding / compressing video 202 is shown. A video decoder 204 can receive encoded video and decode / decompress it into output video 206.

[0059] As used in this article, "keyframe" includes video frames that do not reference other video frames, including inner frames (I-frames) and I-frame types known as Instant Decoding Refresh (IDR) frames.

[0060] Figure 3 Example encoding logic is illustrated. Starting at state 300, one or more conditions of the computer network, such as the video of a computer game being streamed from the transmitter to the receiver via that network, are identified if necessary. For example, if in state 302 the condition is identified as meeting a throttling threshold, the bandwidth and / or latency and / or packet loss of the network path on which the video will be streamed can be identified, and in state 304, the size of at least one keyframe of the video, such as an I-frame like an IDR, can be reduced. For example, if the bandwidth is determined to be below a threshold, the keyframe size can be reduced, or if the network latency is above a latency threshold, the keyframe size can be reduced. The identification of network conditions can occur in real time, or can be predetermined or preset, and the transmitter signals to reduce the keyframe size.

[0061] State 306 indicates that in response to a reduction in the keyframe size due to network constraints, the size of the predicted frame (P-frame) may be increased. This may occur automatically by the P-frame generation algorithm or engine in response to the reduction in the keyframe size. The reduced keyframe and the enlarged P-frame are encoded at state 308 for transmission over the network to the receiver and / or storage at state 310.

[0062] Figures 4 to 6 Example filtering techniques for reducing keyframe size are illustrated. Note that in each technique, the reduced frame may be transmitted to the receiver along with a signal indicating the type of reduction performed and / or the type of reconstruction required on the receiver side, so as to alert the receiver with the reconstruction signal that must be performed on the frame. Further note that the techniques described herein can be used alone or in combination with other techniques described herein.

[0063] exist Figure 4 In the process, at state 400, a keyframe of an I-frame of a first size (“A”) is received, and at state 402, the keyframe is processed by a filter to render a reduced I-frame of size A-Δ at state 404. Note that because the output of the low-pass filter is an image of the same resolution, “size” here refers to image complexity, not pixel count.

[0064] The filter can be implemented as a low-pass filter (LPF) such as a Gaussian filter, or it can be implemented as a cyclic filter, a band-pass filter, or other filters such as an averaging filter, which reduces a small subset of data, such as a block, to the average of the values ​​of the elements in that block. This reduces the sharpness of the image while reducing its size. Note that if a Gaussian filter is used, a radius can be used. Also note that the intensity of the filter may vary from I-frame to I-frame, or even within the same I-frame, from slice to slice; that is, the filter may be the same, but the intensity may differ between I-frames or between slices within the same I-frame. However, the resolution of the reduced I-frame remains the same as the resolution of the input I-frame.

[0065] on the other hand, Figure 5 The instruction is to receive an I-frame of size A in state 500 and reduce its resolution in state 502 to present a reduced I-frame of size A-△ in state 504. For example, a high-definition (HD) keyframe can be reduced to standard-definition (SD).

[0066] again, Figure 6 The instruction is to receive an I-frame with a first orientation at state 600, and process it by a view converter at state 602 to change the orientation of the I-frame by, for example, rotating it by 90°, so that at state 604, I-frames with different orientations presenting a smaller data coverage area are output. For example, the I-frame can be changed from a lateral orientation ( Figure 7 Rotate 700 in the middle to the longitudinal orientation ( Figure 7 (702 in the original text). This frame may be sent along with a flag or other signal that indicates its orientation and the original orientation to which it will return at the receiver.

[0067] Figure 8 The following diagram illustrates the process implemented on it. Figure 4 This is a decoder-side technology for video. Starting at state 800, a downsized (filtered) keyframe (e.g., an I-frame) and signaling regarding the filtering are received from the transmitter via the network. It is decoded at state 802. After decoding, based on the signaling at state 804, a sharpness filter is applied to sharpen the I-frame to recover some content lost when its size was reduced on the encoder side. At state 806, the sharpened frame is sent to the frame buffer for rendering on the video display.

[0068] Figure 9 The following diagram illustrates the process implemented on it. Figure 5The technology involves a decoder-side technique for video. Starting at state 900, reduced-size (scaled down) keyframes (e.g., I-frames) are received from the transmitter over the network. It is decoded at state 902. After decoding, at state 904, the frame is upscaled using techniques such as super-resolution, for example, upscaling from SD to HD. At state 906, the frame is sent to a frame buffer for rendering on the video display.

[0069] Figure 10 The following diagram illustrates the process implemented on it. Figure 6 This is a decoder-side technology for video. Starting at state 1000, a redirected keyframe (e.g., an I-frame) is received from the transmitter over the network. It is decoded at state 1002. After decoding, at state 1004, the frame is redirected back to its original direction using signaling from the transmitter. At state 1006, the frame is sent to a frame buffer for rendering on the video display.

[0070] Figure 11 This illustrates reducing the keyframe size based on the importance of each frame. If a keyframe (such as an I-frame) or a portion of a keyframe is determined to be important at state 1100, for example, as part of a region of interest, a significant action sequence, or other heuristic, its size may not be reduced at state 1102, and the frame / frame portion may be encoded and transmitted without reduction or with a first, smaller reduction. On the other hand, if a keyframe (such as an I-frame) or a portion of a keyframe is determined to be unimportant at state 1100, for example, as part of a background or skybox region, or other heuristic, its size may be reduced at state 1102, and the frame / frame portion may be encoded and transmitted with a reduction greater than any reduction applicable at state 1102.

[0071] Figure 12A pipeline for reducing keyframes piecewise and encoding each piece for transmission before reducing the entire keyframe to reduce latency is illustrated. Starting at state 1200, keyframes, such as I-frames, are divided into slices. At state 1202, the first slice is reduced, for example, by applying a first filter to the first slice with a first strength. The logic then splits in one branch to encode and transmit the reduced first slice at state 1204, and in another branch at state 1206 to reduce a second slice of the frame. This reduction can be accomplished by applying the same first filter with the same first reduction strength as applied to the first slice, or by applying the first filter to the second slice with a second strength, or by applying a second filter of a second type different from the first filter type to the second slice. Then, at state 1208, the second slice is encoded and transmitted, and subsequently, slices of frames are reduced and transmitted in a similar manner in parallel pipeline workflow branches. It should be understood that slices are transmitted along with appropriate signaling indicating the frame to which they belong and the filter type and strength used to reduce the slice.

[0072] Figure 13A It shows the relationship with Figure 12 The technology is consistent with the decoder-side pipeline used for processing video. Starting at state 1300, the first slice of a keyframe is received and decoded at state 1302. In parallel, at state 1304, the second slice of the frame is received. Once the first slice is decoded, it can be sent to the frame buffer for display at state 1306, or otherwise presented on the video display at state 1308 while the second slice is being decoded. Then, once the second slice is decoded, it can similarly be sent to the frame buffer for display at state 1310, or otherwise presented on the video display as subsequent slices of the frame are received and decoded.

[0073] Figure 13B It shows the relationship with Figure 12 The technology is consistent with an alternative decoder-side pipeline used for processing video. Starting at state 1312, the first slice of a keyframe is received and decoded at state 1314. At state 1316, a sharpening filter is applied to the first slice. In parallel, at state 1318, the second slice of the frame is received. Once the first slice is decoded and sharpened, it can be sent to the frame buffer for display at state 1320, or rendered on the video display when the second slice is decoded at state 1322 and sharpened at state 1316. Then, once the second slice is decoded and sharpened, it can also be sent to the frame buffer for display at state 1320, or rendered on the video display when subsequent slices of that frame are received, decoded, and sharpened.

[0074] Figure 14Techniques for reducing video segments based on pre-configuration of video files are illustrated. Starting with state 1400, the pre-configured video can be processed segment by segment. For example, game developers can program their game engines to indicate the appropriate amount of keyframe reduction to be implemented on each segment under state 1402, and apply different levels of reduction to different segments as needed. Alternatively, video as a movie can be similarly pre-configured by human experts or by a machine learning (ML) model trained on a series of movie segments, where a base-fact reduction is indicated for each movie segment.

[0075] For example, in Figure 15 The diagram illustrates a technique for reducing video segments based on signaling from a computer game engine, wherein at state 1500, a signal is received from the game engine that an upcoming scene contains a dense or complex video sequence. In response, at state 1502, keyframe reduction is applied to each segment based on its intensity or complexity, with larger reduction applied to more complex segments and smaller reduction applied to less complex segments. Alternatively, in other embodiments, larger reduction may be applied to less complex segments, and smaller reduction may be applied to more complex segments.

[0076] Figure 16 Another example is shown, where a filter is selected for each segment based on its content. In state 1600, the content of keyframes of the segment is identified by signaling accompanying keyframes, or by, for example, an ML model trained on a training set of keyframes, where each keyframe is appended with a basic fact content classification. In state 1602, the filter type and filter strength (if needed) are selected based on the content of the keyframes and / or their associated video segments. The process moves to state 1604, where one or more keyframes of the segment are filtered according to the selection in state 1602. The reduced frames are then encoded in state 1606, and in state 1608, they are transmitted and / or stored using signaling indicating the filter type and strength.

[0077] Figure 17 It shows the relationship with Figure 16 The technology used is consistent with the decoder-side technology for processing video. In state 1700, keyframes of the segment are received from the transmitter via the network. Moving to state 1702, the frames are decoded. Proceeding to state 1704, based on the signaling of the accompanying frames, appropriate sharpening filters and filter strengths are selected to reconstruct as many frames as possible in state 1706 so that they can be presented on the video display in state 1708.

[0078] Figure 18 It shows the relationship with Figure 17A consistent table. The first column (1800) represents signaling data indicating the filter type used for a specific segment of video, and the second column (1802) represents the reconstruction filter applied to the corresponding filter in the same row of the first column. Therefore, the first column indicates multiple corresponding filter types used to reduce a given segment of video, while the second column represents the corresponding type of reconstruction filter. Note that signaling from the transmitter can indicate the reduction filter type (and strength) and leave it to the receiver to find the appropriate reconstruction filter, or signaling from the transmitter can indicate to the receiver which reconstruction filter (and strength) should be used.

[0079] Figure 19 This illustrates a reduction technique that reduces only chroma information. Starting at state 1900, chroma data is identified in the keyframe, distinct from luminance data. Chroma-only reduction is performed at state 1902, encoded along with luminance at state 1904, and transmitted at state 1906 along with appropriate signaling indicating that only chroma has been reduced and requires reconstruction.

[0080] Figure 20 It shows the relationship with Figure 19 The technology used is consistent with the decoder-side technology for processing video. In state 2000, keyframe data is received from the transmitter via a computer network. In state 2002, the frame is decoded. Then, based on signaling from the transmitter, in state 2004, a sharpness filter is applied only to the chroma of the frame, and in state 2006, the frame is displayed on a video monitor.

[0081] It is now understood that, among other things, a portion of the video can be given higher priority and encoded / decoded first as described above. After decoding, before being sent to the display, encoded noise is removed, and in some embodiments, resolution is enhanced, while the frame rate is increased if necessary. Lost frames can be reconstructed using interpolation between frames before and after the lost frame. The artifacts to be eliminated are caused by the quantization increase of the transform coefficients during encoding. The decoder addresses this by processing the decoded coefficients and can post-process only the pixels of a subset of frames.

[0082] While specific techniques are illustrated and described in detail herein, it should be understood that the subject matter covered by this application is limited only by the claims.

Claims

1. An apparatus comprising: At least one processor component is configured as follows: Identify at least one condition for at least one network; In response to the condition, reduce the size of at least one I-frame of the video; and The I-frame and at least one non-I-frame are sent to the transmitter for transmission over the network.

2. The apparatus according to claim 1, wherein, The videos include computer game videos.

3. The apparatus according to claim 1, wherein, The conditions include time delay.

4. The apparatus according to claim 1, wherein, The conditions include bandwidth.

5. The apparatus according to claim 1, wherein, The processor component is configured to reduce the size of the I-frame by at least partially processing it through a filter.

6. The apparatus according to claim 1, wherein, The processor component is configured to reduce the size of the I-frame at least in part by reducing the resolution of the I-frame.

7. The apparatus according to claim 1, wherein, The processor component is configured to reduce the size of the I-frame at least in part by changing its orientation.

8. The apparatus according to claim 7, wherein, The processor component is configured to signal the orientation of the I-frame.

9. The apparatus according to claim 1, wherein, The I-frame is a first I-frame, and the processor component is configured to reduce the size of the first I-frame in response to the condition and in response to the first I-frame having a first importance, and not reduce the size of the second I-frame in response to the condition and in response to the second I-frame having a second importance.

10. An apparatus comprising: At least one computer medium, which is not a transient signal, and includes instructions executable by at least one processor component to: Reduce the first portion of the video before encoding using at least the first filter parameter; Before encoding, a second portion of the video is reduced using at least a second filtering parameter that is not used to reduce the first portion; as well as The portion is transmitted to at least one receiver via a network.

11. The apparatus according to claim 10, wherein, The instruction can be executed to: The portion, along with signaling indicating the first and second filter parameters, is transmitted over the network.

12. The apparatus according to claim 10, wherein, The first part and the second part are keyframes of the video.

13. The apparatus according to claim 10, wherein, The first part and the second part are portions of a keyframe of the video.

14. The apparatus according to claim 10, wherein, The first filtering parameter and the second filtering parameter include corresponding first filtering type and second filtering type.

15. The apparatus according to claim 10, wherein, The first filtering parameter and the second filtering parameter include corresponding first filtering strength and second filtering strength.

16. The apparatus according to claim 10, wherein, The video includes at least one computer game.

17. A method comprising: Receive at least a portion of at least the first keyframe of at least one video over a network; The network receives signaling indicating the reconstruction of the portion of the keyframe to be applied. as well as Before the portion of the keyframe is presented on the video display, the portion of the keyframe is reconstructed based on the signaling.

18. The method according to claim 17, wherein, The reconstruction includes magnifying the portion of the keyframe.

19. The method of claim 17, wherein, The reconstruction includes processing the portion of the keyframe using at least one sharpening filter.

20. The method of claim 17, wherein, The reconstruction includes redirecting the portion of the keyframe.

Images