MACHINE LEARNING MODEL FOR VIDEO WITH REAL-TIME RATE CONTROL

Abstract

Methods and systems for rate control include determining (322) an encoding parameter value for use with an input set of video frames based on a current channel capacity using a machine learning model that accepts the input set of video frames and the current channel capacity as inputs. The input set of video frames is encoded using the encoding parameter (324) to produce encoded video that has a bitrate below the current channel capacity. The encoded video is transmitted (326).

Description

INFORMATION ABOUT RELATED REGISTRATIONS

[0001] This application claims priority over U.S. patent application no. 63 / 535,406, filed on August 30, 2023, and U.S. patent application no. 18 / 816,444, filed on August 27, 2024, which are incorporated herein in their entirety by reference. BACKGROUND Technical area

[0002] The present invention relates to the transmission of streaming video and in particular to real-time rate control for video encoding. Description of the state of the art

[0003] Channel quality for transmitting real-time information, especially when using wireless communication, can vary as environmental conditions change. For example, a user with a mobile device might move quickly from one environment to another, causing transmission paths to change drastically in an instant. Even between a stationary transmitter and receiver, channel conditions can change as objects like people and vehicles move through the environment. SUMMARY

[0004] A rate control method involves determining an encoding parameter value for use with an input set of video frames based on a current channel capacity, using a machine learning model that accepts the input set of video frames and the current channel capacity as inputs. The input set of video frames is encoded using the encoding parameter to produce encoded video with a bitrate below the current channel capacity. The encoded video is then transmitted.

[0005] A rate control system comprises a hardware processor and memory that stores a computer program. When executed by the hardware processor, the computer program instructs the hardware processor to determine an encoding parameter value for use with an input set of video frames based on a current channel capacity, using a machine learning model that accepts the input set of video frames and the current channel capacity as inputs; to encode the input set of video frames using the encoding parameter to produce encoded video with a bitrate below the current channel capacity; and to transmit the encoded video.

[0006] These and other features and advantages will become apparent from the following detailed description of illustrative embodiments, which should be read in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The disclosure will provide details in the following description of preferred embodiments with reference to the following figures, wherein: Fig. 1 a diagram of a communication system that transmits video over a channel according to a maximum capacity of the channel, wherein a coding parameter is dynamically adjusted according to real-time information about the channel quality, in accordance with an embodiment of the present invention; Fig. 2 a block diagram of a real-time rate control which dynamically adjusts a coding parameter according to real-time information about the channel quality using a machine learning model, in accordance with an embodiment of the present invention; Fig. 3 a block / flow diagram of a method for training and using a machine learning model for transmitting video with real-time rate control, in accordance with an embodiment of the present invention; Fig. 4 is a block diagram of a healthcare facility that uses video surveillance with real-time rate control to assist in medical decision-making, in accordance with an embodiment of the present invention; Fig. 5 a block diagram of a computer device capable of performing real-time rate-controlled video encoding in accordance with an embodiment of the present invention; Fig. 6 is a diagram of an exemplary neural network architecture that can be used to implement part of the predictive head in accordance with an embodiment of the present invention; and Fig. Figure 7 shows a diagram of an exemplary deep neural network architecture that can be used to implement part of the prediction head in accordance with an embodiment of the present invention. DETAILED DESCRIPTION OF PREFERRED EXECUTION FORMS

[0008] Channel quality information, such as a Channel Quality Index (CQI), can be reported to a transmitter by a user device (UE) on a near-instantaneous basis. This channel quality information represents the potential bandwidth limit for the channel between the transmitter and the UE. The UE can be used to dynamically update the encoding quality of transmitted video so that as much of the available bandwidth as possible is used without exceeding the channel limits.

[0009] To achieve this, real-time rate control is used, processing incoming video clips, which consist of a set of video frames, to generate features representing the video clips. These features are processed by a predictive head, along with a current bandwidth limit for the channel, to generate an estimated quantization parameter (QP) for the channel. The estimated QP is then used to encode the set of video frames, which are then transmitted to the UE.

[0010] The estimated QP is chosen to be just below the maximum bitrate for the channel. When the next set of video frames is provided, the estimate can be updated to reflect changing channel conditions. For rapidly changing channel conditions, the number of video frames in a set can be reduced, resulting in more frequent updates to the QP estimate.

[0011] With reference to Fig. Figure 1 shows an exemplary video transmission system. A video camera 102 captures frames of a scene within its field of view and outputs image data. Each captured frame is encoded by an encoder 104 into a compressed image that will consume less bandwidth during transmission and can be decoded by a UE 108. The encoder 104 has the ability to adjust the encoding according to a QP, where higher QP values ​​correspond to lower bit rates and lower visual quality of the decoded image at the UE 108.

[0012] A transmitter 106 transmits the encoded images to the UE 108, which receives and decodes them, making them suitable for viewing. The transmission is illustrated as occurring via a wireless medium; however, it should be understood that the principles presented here can apply to any suitable wired or wireless communication medium and protocol. The transmission is susceptible to channel effects, for example, characterized by a signal-to-noise ratio, which describes how environmental factors affect the UE 108's ability to receive the transmission without errors.

[0013] The UE 108 can identify a CQI, or any other suitable representation of channel quality, based on the received transmission. The UE 108 can then return the CQI to a real-time rate controller 110. In some cases, the UE 108 can return the channel quality information over the same channel as the transmission, for example, using time-division multiplexing to schedule channel usage between the transmitter 106 and the UE 108. In other cases, the UE 108 can transmit the channel quality information over a separate channel with its own specific transmission characteristics.

[0014] The Real-Time Rate Control 110 accepts the channel quality information and determines a QP (Quality Point). After one or more frames have been encoded and transmitted to the UE 108 (Universal Encoder), the UE 108 provides a current estimate of the channel conditions. This new estimate is used by the Real-Time Rate Control 110 to update the QP for use when encoding the next set of one or more frames. In this way, the system can quickly adapt to changing channel conditions.

[0015] With reference to Fig. Section 2 shows additional details regarding the Real-Time Rate Control 110. Frames from camera 102 are provided as input for the Real-Time Rate Control 110, along with channel quality information, to generate a QP as output. The Real-Time Rate Control 110 can accept a group of such frames together, so the QP output is used to encode all frames in the group.

[0016] When encoding a set of video frames with a given QP, the actual bitrate of the encoded video stream is difficult to predict. While higher QP values ​​generally correspond to higher bitrates, the visual and dynamic complexity of a group of video frames can affect the size of the resulting video. Thus, an analytical or formulaic approach to QP selection can lead to a situation where the bitrate exceeds the maximum capacity of the channel, resulting in dropped frames and other errors. Such errors are easily visible to a user, whereas the slight degradation in quality resulting from selecting a slightly higher QP value might not be noticeable. However, an overly conservative QP selection leads to unused channel capacity.

[0017] The QP selection can instead be performed using machine learning, with video features being extracted from the video frames by a feature extractor 202. These features are then processed by a predictor head 204, which is trained to generate an encoding parameter to provide bitrates close to the channel capacity without exceeding it. While the predictor head 204 is specifically designed to generate a QP value, it should be understood that any suitable encoding parameter in accordance with an encoding standard used by the encoder 104 can be used. The predictor head 204's machine learning model is trained to recognize the bitrate requirements of the input video frames and balance them against the limits imposed by the channel.

[0018] The channel quality information is thus used to identify a bitrate limit of 206, which represents the currently available bandwidth of the channel. If more information than this bitrate limit of 206 is transmitted, some of it is likely unrecoverable by the UE 108.

[0019] The encoder 104 can process a group of video frames V to produce encoded video Ṽ with a specific height H, width W, and frame rate fps. The bit rate of Ṽ is expressed as b(V, QP) and depends on the quantization parameter QP. The encoded video Ṽ can then be encapsulated in container packets for transmission. For a channel with Rayleigh fading, which has a channel coefficient h that is a circularly symmetric complex Gaussian process with mean zero and variance σn2 The supported data transfer rate is: r(h)=log(1+P|h|2σ2G) where G models the capacity gap due to channel coding, which is the difference between a theoretical rate given by the above and a physical model, and P represents the average power. G can be assumed to be a fixed number, and any dependence on the modulation and coding scheme can be ignored.

[0020] Channel coherence time represents how much time it takes for channel conditions to change significantly. In most circumstances, the channel coherence time is relatively long compared to the duration of the set of video frames, and thus the bitrate limit of 206 can be assumed to be suitable for the entire duration of the transmission of the set of video frames. In some circumstances, such as in a rapidly changing environment or when the UE 108 is moving quickly, the channel coherence time may be shorter than the duration of the set of video frames. In such circumstances, the number of video frames processed at one time can be reduced to increase the frequency with which the QP estimation is updated.

[0021] If the encoded bitrate for a set of video frames is below the bitrate limit of 206, then all container packets should be received without errors. However, if the encoded bitrate exceeds the bitrate limit of 206, some packets (and therefore some video frames) may be dropped. The variability of the channel bitrate r(h) is due to the fading nature of channel h, and variations in the video bitrates of sets of video frames must be taken into account to ensure that the limit is not exceeded.

[0022] Feature extractor 202 can incorporate convolutional neural network layers to recognize video content with dimensions of [B, T, W, H] and outputs features with an example dimension of [B, 196, T, W / 16, H / 16], where B is the batch size, T is the number of video frames in a segment, W is the width in pixels of the video frames, and H is the height in pixels of the video frames. Each batch can thus contain T frames, with a total of B batches used for training.

[0023] The predictor head 204 can be implemented as a deep neural network comprising multiple convolutional layers, each followed by a conditional group normalization (CGN) block. The CGN blocks normalize the output from the previous layer. The predictor head can define the bitrate ceiling 206 as a log tensor. 10 (BR maxLet B,1, be a condition factor. Each element in the tensor represents the bitrate limit for each video in the batch.

[0024] The predictor head can process the condition factor through three linear layers, each with a Gaussian Error Linear Unit (GELU) activation function that transforms the tensor size to [2B,1]. The tensor can then be split into two tensors, γ and β, of size [B,1]. The γ and β tensors are used to fit the normalized output from the preceding layer using the formula γ × output + β, in order to train the video feature to distinguish QP values ​​across different input video examples with varying bitrate ceilings.

[0025] Training data for training the predictor head 204 can be sets of a group of video frames V, a specified target quantization parameter QP targetand a corresponding bitrate BR target include those associated with the QP target The video frames V are used as input for the feature extractor 202 with BR. target This is used as the condition factor for the CGN blocks. The result generated by the predictor head is an estimate QP. To train the predictor head, the following loss function can be optimized: L=LCE(QP˜,QPtarget) where LCE a cross-entropy loss.

[0026] During training, the target bitrate BR targetThe QP is not strictly treated as an upper bitrate limit. Therefore, during testing, the estimated QP may be slightly lower than the QP value that guarantees a bitrate below the 206 kbps upper limit. This rounding effect can be mitigated by using slightly higher QP values ​​(corresponding to lower-quality encodings) than the estimated QP value during data encoding. This adjustment effectively prevents the occurrence of unwanted and destructive artifacts on the UE 108 while adding only a negligible drop in video quality. The QP adjustment can be fine-tuned by running samples using different video feeds to identify a typical adjustment amount.

[0027] The exact structure of the predictor head 204 may depend on the specific dimensions and type of information in the video frames. In one example, the predictor head 204 can generate a three-dimensional (3D) convolution layer with dimensions (196, 196, kernel = (3,3,3)), a CGN layer with dimensions (number of groups = 4, condition), a GELU layer, a 3D convolution layer with dimensions (196, 196, kernel = (1,1,1)), a CGN layer with dimensions (number of groups = 4, condition), a GELU layer, a 3D convolution layer with dimensions (196, 196, kernel = (3,3,3)), a CGN layer with dimensions (number of groups = 4, condition), a GELU layer, a 3D convolution layer with dimensions (196, 52, kernel = (1,1,1)), a CGN layer with dimensions (number of groups = 4, condition), a GELU layer, a fully bonded layer with dimensions (12480,3120) and a GELU activation, a fully bonded layer with dimensions (3120,780) and a GELU activation, a fully connected layer with dimensions (780,195) and a GELU activation, and a fully connected layer with dimensions (195,52) to produce an output QP value between 1 and 52. The number 52 can be replaced by any suitable number of QP subdivisions for a given coding standard.

[0028] Each CGN layer of the predictor head 204 can process the condition (e.g., the bit rate limit 206) with a fully connected layer with dimensions (1,196) and a GELU activation, a fully connected layer with dimensions (196,392) and a GELU activation, and a fully connected layer with dimensions (392,392) to generate the tensor with dimensions [2B,1]. This tensor can be decomposed, as described above, into γ- and β-tensors, each with one dimension of 196. Group normalization (e.g., setting the number of groups to four) is then performed on the output of the previous layer (e.g., the output of a 3D convolution layer), and this normalized output is multiplied by γ, with the product being added to β.

[0029] With reference to Fig. Figure 3 presents a procedure for training and using a real-time rate control model. Block 300 trains the predictor 204 as described above, using a set of training data. The training data can be any suitable set of videos, where sets of video frames are encoded according to a variety of different QP values. To prepare the training dataset, each video can be divided into uniform blocks according to a number of frames (e.g., eight consecutive frames). The videos divided into blocks can then be encoded using, for example, 52 different QP values. The set of encoded video blocks can be sampled to create a training set and a test set, and the optimization of the predictor 204 parameters can be performed in accordance with the loss function described above.Training 300 takes an input pair comprising a video block from the training dataset and a corresponding encoded bitrate, and generates a predicted QP value. This predicted QP value is then compared to the QP value used to encode the video block from the input pair.

[0030] The trained model is then used in block 310. This can include the use of the model as software in a camera device 102, for example as part of an encoding software module that operates on raw camera data.

[0031] During operation, Block 320 transmits video data encoded with real-time rate control. This involves an iterative process in which, for each successive set of video frames, Block 322 determines the QP to use, Block 324 encodes the video frames using the determined QP value, and Block 326 transmits the encoded video to the UE 108. Block 328 recites and processes channel quality information from the UE 108, which is used in Block 322 to determine a new QP value for the next iteration.

[0032] With reference to Fig. Figure 4 shows a diagram of information extraction in the context of a healthcare facility 400. Video surveillance with real-time rate control 408 can be used to monitor patients in the healthcare facility 400 to ensure proper treatment and prevent hazardous conditions. For example, video surveillance in conjunction with action detection can be used to determine whether a patient is adhering to a treatment regimen or engaging in hazardous activities. Wireless transmission conditions within a healthcare facility 400 can change rapidly, for example, when doors open or close and when diagnostic equipment is switched on and off, with accompanying electromagnetic interference.

[0033] The healthcare facility may include one or more healthcare professionals 402 who review information extracted from a patient's medical records 406 to determine their health and treatment needs. These medical records 406 may include self-reported information from the patient, test results, and notes made by healthcare professionals that have been added to the patient's record. Treatment systems 404 may further monitor patient status to generate medical records 406 and may be designed to automatically administer and adjust treatments as needed.

[0034] Based on information obtained from real-time rate-controlled video surveillance (408), medical professionals (402) can then make medical decisions regarding patient care that are tailored to the patient's needs. For example, medical professionals (402) can diagnose the patient's health condition and prescribe specific medications, surgeries, and / or therapies.

[0035] The various elements of the healthcare facility 400 can communicate with each other via a network 410, for example, using any suitable wired or wireless communication protocol and medium. Thus, the real-time rate-controlled video surveillance 408 receives information about a patient and updates the medical records 406 with the relevant visual information. The real-time rate-controlled video surveillance 408 can also output video for review by a medical professional 402. In some cases, the video can be used to coordinate with treatment systems 404, and in some cases, to automatically administer or modify a treatment. For example, if the video indicates a dangerous condition, then the treatment systems 404 can automatically stop the administration of the treatment.

[0036] As in Fig. As shown in Figure 5, the computer device 500 includes, for illustrative purposes, the processor 510, an input / output subsystem 520, a memory 530, a data storage device 540, and a communication subsystem 550, and / or other components and devices commonly found in a server or similar computer device. In other embodiments, the computer device 500 may include other or additional components commonly found in a server computer (e.g., various input / output devices). Additionally, in some embodiments, one or more of the illustrative components may be integrated into another component or otherwise form part of it. For example, in some embodiments, the memory 530, or parts thereof, may be integrated into the processor 510.

[0037] The Processor 510 can be configured as any type of processor capable of performing the functions described herein. The Processor 510 can be configured as a single processor, multiple processors, a central processing unit (CPU), a graphics processing unit (GPU), a single- or multi-core processor, a digital signal processor, a microcontroller, or any other processor or processing / control circuit.

[0038] The Memory 530 can be configured as any type of volatile or non-volatile memory or data storage capable of performing the functions described herein. In operation, the Memory 530 can store various data and software used during the operation of the Computer Device 500, such as operating systems, applications, programs, libraries, and drivers. The Memory 530 is communicatively coupled to the Processor 510 via the I / O Subsystem 520, which can be configured as circuits and / or components to facilitate input / output operations with the Processor 510, the Memory 530, and other components of the Computer Device 500. For example, the I / O Subsystem 520 can be configured as memory controller hubs, input / output control hubs, platform controller hubs, integrated control circuits, firmware devices, communication links (e.g.,Point-to-point connections, bus connections, wires, cables, optical fibers, conductive traces on printed circuit boards, etc.) and / or other components and subsystems may be configured or otherwise include these to facilitate input / output operations. In some embodiments, the I / O subsystem 520 may form part of a system-on-a-chip (SoC) and be integrated together with the processor 510, the memory 530, and other components of the computer device 500 on a single integrated circuit chip.

[0039] The data storage device 540 can be configured as any type of device or devices configured for short-term or long-term data storage, such as storage devices and circuits, memory cards, hard disk drives, solid-state drives, or other data storage devices. The data storage device 540 can store program code 540A for training a model, 540B for selecting a QP value for encoding video, and / or 540C for encoding the video. Any or all of these blocks of program code can be included in a given computer system. The communication subsystem 550 of the computer device 500 can be configured as any network interface controller or other communication circuit, device, or collection thereof capable of enabling communication between the computer device 500 and other remote devices over a network.The Communications Subsystem 550 can be configured to use any one or more communication technologies (e.g., wired or wireless communications) and associated protocols (e.g., Ethernet, InfiniBand®, Bluetooth®, Wi-Fi®, WiMAX, etc.) to effect such communication.

[0040] As shown, the computer device 500 can also include one or more peripheral devices 560. The peripheral devices 560 can include any number of additional input / output devices, interface devices, and / or other peripheral equipment. For example, in some embodiments, the peripheral devices 560 can include a display, a touchscreen, graphics circuitry, a keyboard, a mouse, a speaker system, a microphone, a network interface, and / or other input / output devices, interface devices, and / or peripheral equipment.

[0041] Of course, the computer device 500 can also include other elements (not shown), as can be readily considered by a person skilled in the art, as well as omit certain elements. For example, various other sensors, input devices, and / or output devices can be incorporated into the computer device 500, depending on its specific implementation, as is readily understood by a person skilled in the art. For example, different types of wireless and / or wired input and / or output devices can be used. Furthermore, additional processors, controllers, memory, and so on can also be used in various configurations. These and other variations of the processing system 500 are readily conceivable by a person skilled in the art in light of the teachings of the present invention provided herein.

[0042] With reference to Fig. 6 and Fig.Section 7 presents exemplary neural network architectures that can be used to implement parts of the present models, such as the predictor head 204. A neural network is a generalized system that improves its performance and accuracy through exposure to additional empirical data. The neural network is trained by exposure to this empirical data. During training, the neural network stores and adjusts a variety of weights that are applied to the incoming empirical data. By applying the adjusted weights to the data, the network can identify the data as belonging to a specific predefined class from a set of classes, or it can output a probability that the input data belongs to each of the classes.

[0043] The empirical data, also known as training data, from a set of examples can be formatted as a chain of values ​​and fed into the input of the neural network. Each example can be associated with a known result or output. Each example can be represented as a pair, (x, y), where x represents the input data and y represents the known output. The input data can include a variety of different data types and can comprise multiple different values. The network can have an input node for each value that constitutes the example's input data, and a separate weight can be applied to each input value. The input data can be formatted, for example, as a vector, an array, or a string, depending on the architecture of the neural network being constructed and trained.

[0044] The neural network "learns" by comparing the output generated from the input data with known values ​​from the examples and adjusting the stored weights to minimize the differences between the output values ​​and the known values. These adjustments can be made by backpropagating the stored weights, where the effect of the weights on the output values ​​can be determined by calculating the mathematical gradient, and the weights are adjusted in a way that shifts the output towards a minimal difference. This optimization, known as a gradient descent approach, is a non-restrictive example of how training can be performed. A subset of examples with known values, not used for training, can be used to test and validate the accuracy of the neural network.

[0045] During operation, the trained neural network can be used on new data that was not previously used in training or validation through generalization. The adjusted weights of the neural network can be applied to the new data, with the weights estimating a function derived from the training examples. The parameters of the estimated function, captured by the weights, are based on statistical inference.

[0046] In layered neural networks, nodes are arranged in layers. An exemplary simple neural network has an input layer 620 consisting of source nodes 622 and a single computation layer 630, which has one or more computation nodes 632 that also function as output nodes, with a single computation node 632 for each possible category into which the input sample could be classified. An input layer 620 can have a number of source nodes 622 equal to the number of data values ​​612 in the input data 610. The data values ​​612 in the input data 610 can be represented as a column vector. Each computation node 632 in the computation layer 630 generates a linear combination of weighted values ​​from the input data 610 fed into the input nodes 620 and applies a differentiable, nonlinear activation function to the sum.The exemplary simple neural network can perform a classification on linearly separable examples (e.g., patterns).

[0047] A deep neural network, such as a multilayer perceptron, can have an input layer 620 consisting of source nodes 622, one or more computation layers 630 with one or more computation nodes 632, and an output layer 640, where there is a single output node 642 for each possible category into which the input sample could be classified. An input layer 620 can have a number of source nodes 622 equal to the number of data values ​​612 in the input data 610. The computation nodes 632 in the computation layer(s) 630 can also be called hidden layers, since they are located between the source nodes 622 and the output node(s) 642 and are not directly observed.Each node 632, 642 in a computation layer generates a linear combination of weighted values ​​from the values ​​output by the nodes in a previous layer and applies a nonlinear activation function that is differentiable over the range of the linear combination. The weights applied to the value from each previous node can be denoted, for example, by w1, w2, ... w. n-1, w n The output layer provides the network's overall response to the input data. A deep neural network can be fully connected, with each node in a computational layer connected to every other node in the previous layer, or it can have other configurations of connections between layers. If connections between nodes are missing, the network is said to be partially connected.

[0048] Training a deep neural network can involve two phases: a forward phase in which the weights of each node are set and the input is propagated through the network, and a backward phase in which an error value is propagated backward through the network and weight values ​​are updated.

[0049] The computation nodes 632 in the one or more computation (hidden) layer(s) 630 perform a nonlinear transformation on the input data 612, which generates a feature space. The classes or categories can be more easily separated in the feature space than in the original data space.

[0050] The embodiments described herein may be entirely hardware, entirely software, or comprise both hardware and software elements. In a preferred embodiment, the present invention is implemented in software, which includes, but is not limited to, firmware, resident software, microcode, etc.

[0051] Embodiments may include a computer program product accessible from a computer-usable or computer-readable medium that provides program code for use by or in conjunction with a computer or any command-execution system. A computer-usable or computer-readable medium may include any apparatus that stores, communicates, propagates, or transports the program for use by or in conjunction with the command-execution system, apparatus, or device. The medium may be a magnetic, optical, electronic, electromagnetic, infrared, or semiconductor system (or apparatus or device), or a propagation medium.The medium can include a computer-readable storage medium such as semiconductor or solid-state memory, magnetic tape, removable computer disk, random access memory (RAM), read-only memory (ROM), rigid magnetic disk, optical disk, etc.

[0052] Any computer program can be tangibly stored in machine-readable storage media or devices (e.g., program memory or magnetic disk) that are readable by a general or special programmable computer to configure and control the operation of a computer when the storage media or device is read by the computer to perform the procedures described herein. The system according to the invention can also be considered to be embodied in a computer-readable storage medium configured with a computer program, wherein the storage medium so configured causes a computer to operate in a specific and predefined manner to perform the functions described herein.

[0053] A data processing system capable of storing and / or executing program code may include at least one processor, which is directly or indirectly coupled to memory elements via a system bus. The memory elements may include local memory, used during the actual execution of the program code, mass storage, and cache memory, which provides temporary storage of at least some program code to reduce the number of times code is retrieved from mass storage during execution. Input / output or I / O devices (including, but not limited to, keyboards, displays, pointing devices, etc.) may be coupled to the system either directly or via intermediary I / O controllers.

[0054] Network adapters can also be connected to the system to allow the data processing system to connect to other data processing systems or remote printers or storage devices via intermediary private or public networks. Modems, cable modems, and Ethernet cards are just some of the types of network adapters currently available.

[0055] As used herein, the term “hardware processor subsystem” or “hardware processor” can refer to a processor, memory, software, or combinations thereof, working together to perform one or more specific tasks. In useful embodiments, the hardware processor subsystem may include one or more data processing elements (e.g., logic circuits, processing circuits, instruction execution devices, etc.). The one or more data processing elements may be contained within a central processing unit, a graphics processing unit, and / or a separate processor- or computationally-based controller (e.g., logic gates, etc.). The hardware processor subsystem may include one or more on-board memories (e.g., caches, dedicated memory arrays, read-only memories, etc.).In some embodiments, the hardware processor subsystem may include one or more memories, which may be located on-board or off-board, or which may be dedicated for use by the hardware processor subsystem (e.g., ROM, RAM, basic input / output system (BIOS), etc.).

[0056] In some embodiments, the hardware processor subsystem can include and execute one or more software elements. These software elements can include an operating system and / or one or more applications and / or specific code to achieve a specified result.

[0057] In other embodiments, the hardware processor subsystem may include dedicated, specialized circuits that perform one or more electronic processing functions to achieve a specified result. Such circuits may include one or more application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), and / or programmable logic arrays (PLAs).

[0058] These and other variations of a hardware processor subsystem are also considered in accordance with embodiments of the present invention.

[0059] References in the description to "one embodiment" or "an embodiment" of the present invention, as well as other variations thereof, mean that a particular feature, structure, property, and so forth, described in connection with the embodiment, is included in at least one embodiment of the present invention. Thus, the appearances of the phrase "in one embodiment" or "in an embodiment," as well as any other variations occurring at different points in the description, do not necessarily all refer to the same embodiment. However, it should be noted that features of one or more embodiments may be combined in light of the teachings of the present invention provided herein.

[0060] It should be noted that the use of any of the following " / ", "and / or", and "at least one of", for example, in the cases of "A / B", "A and / or B", and "at least one of A and B", is intended to include the selection of only the first listed option (A), or the selection of only the second listed option (B), or the selection of both options (A and B). As a further example, in the cases of "A, B and / or C" and "at least one of A, B and C", such wording is intended to include the selection of only the first listed option (A), or the selection of only the second listed option (B), or the selection of only the third listed option (C), or the selection of only the first and second listed options (A and B), or the selection of only the first and third listed options (A and C), or the selection of only the second and third listed options (B and C), or the selection of all three options (A and B and C).This can be extended to any number of listed elements.

[0061] The foregoing is to be understood in every respect as illustrative and exemplary, but not as limiting, and the scope of the invention disclosed herein is not to be determined from the detailed description, but rather from the claims as interpreted to the fullest extent permitted by patent law. It is understood that the embodiments shown and described herein are merely illustrative of the present invention and that those skilled in the field may implement various modifications without departing from the scope and spirit of the invention. Those skilled in the field could implement various other combinations of features without departing from the scope and spirit of the invention.Having thus described aspects of the invention, with the details and special features required by patent law, what is claimed and to be protected by a patent is set out in the attached claims. QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited patent literature

[0000] US 63 / 535,406

[0001] US 18 / 816,444

[0001]

Claims

[1] A computer-implemented rate control method comprising: Determining (322) an encoding parameter value for use with an input set of video frames based on a current channel capacity using a machine learning model that accepts the input set of video frames and the current channel capacity as inputs; Encoding (324) the input set of video frames using the encoding parameter to produce encoded video that has a bitrate below the current channel capacity; and Transmitting (326) the encoded video. [2] The method according to claim 1, further comprising determining the current channel capacity based on channel quality information received from a user device. [3] The method according to claim 1, wherein the machine learning model comprises a predictor head model trained to generate a parameter value which, when used to encode the input set of video frames, causes the encoded video to be at or below the current channel capacity. [4] The method according to claim 3, wherein the predictor head model is a deep neural network model comprising conditional group normalization using the current channel capacity as a condition. [5] The method according to claim 4, wherein the predictor head model comprises a plurality of convolution layers, each of which is followed by a respective conditional group normalization. [6] The method according to claim 1, wherein the coding parameter is a quantization parameter. [7] The method according to claim 1, further comprising changing the specified coding parameter value to reduce the video quality prior to encoding the video. [8] The method according to claim 1, wherein determining the coding parameter value comprises extracting features from the input set of video frames and processing the features with the current channel capacity in a predictor head model. [9] The method according to claim 1, wherein the coded video is transmitted to a medical professional to assist in medical decision-making. [10] The method according to claim 1, further comprising performing a treatment measure in response to the coded video, including automatically changing the treatment of a patient in response to patient activity shown in the coded video. [11] The method according to claim 1, wherein determining the encoding parameter value comprises maximizing the average video quality of a live video feed subject to the current channel capacity available for the set of video frames and minimizing the probability of packet loss and video artifacts during transmission of the encoded video. [12] A rate control system comprising: a hardware processor (510); and a memory (540) that stores a computer program which, when executed by the hardware processor, causes the hardware processor to: Determining (322) an encoding parameter value for use with an input set of video frames based on a current channel capacity using a machine learning model that accepts the input set of video frames and the current channel capacity as inputs; Encoding (324) the input set of video frames using the encoding parameter to produce encoded video that has a bitrate below the current channel capacity; and Transmitting (326) the encoded video. [13] The system according to claim 12, wherein the computer program further instructs the hardware processor to determine the current channel capacity based on channel quality information received from a user device. [14] The system according to claim 12, wherein the machine learning model comprises a predictor head model trained to generate a parameter value which, when used to encode the input set of video frames, causes the encoded video to be at or below the current channel capacity. [15] The system according to claim 14, wherein the predictor head model is a deep neural network model comprising conditional group normalization using the current channel capacity as a condition. [16] The system according to claim 15, wherein the predictor head model comprises a plurality of convolution layers, each of which is followed by a respective conditional group normalization. [17] The system according to claim 12, wherein the computer program further causes the hardware processor to change the specified encoding parameter value in order to reduce the video quality before encoding the video. [18] The system according to claim 12, wherein the computer program further instructs the hardware processor to extract features from the input set of video frames and to process the features with the current channel capacity in a predictive head model. [19] The system according to claim 12, wherein the encoded video is transmitted to a medical professional to assist in medical decision-making. [20] The system according to claim 12, wherein the computer program further causes the hardware processor to perform a treatment action in response to the encoded video, including automatically changing the treatment of a patient in response to patient activity shown in the encoded video.

Images