Machine learning models for video with real-time rate control

Abstract

A method and system for rate control includes using a machine learning model that accepts an input set of video frames and the current channel capacity as input to determine, based on the current channel capacity, the values ​​of encoding parameters to be used for the input set of video frames (322). Using the encoding parameters, the input set of video frames is encoded (324) to produce an encoded video having a bitrate less than the current channel capacity. The encoded video is transmitted (326).

Description

[Technical Field]

[0001] Related application information This application claims priority to U.S. Patent Application No. 63 / 535,406, filed on 30 August 2023, and U.S. Patent Application No. 18 / 816,444, filed on 27 August 2024, both of which are incorporated herein by reference in their entirety. [Background technology]

[0002] This invention relates to streaming video transmission, and more particularly to real-time rate control of video encoding. Explanation of related technologies

[0003] Especially when using wireless communication, channel quality for real-time information transmission can change in response to changes in environmental conditions. For example, users with mobile devices may move quickly from one set of conditions to another, causing the transmission path to change significantly from one moment to the next. Even between stationary transmitters and receivers, the channel state can change if objects such as people or vehicles move through the environment. [Overview of the Initiative]

[0004] A method for rate control includes using a machine learning model that accepts an input set of video frames and the current channel capacity as input to determine, based on the current channel capacity, the values ​​of encoding parameters to be used for the input set of video frames. Using the encoding parameters, the input set of video frames is encoded to produce an encoded video having a bitrate less than the current channel capacity. The encoded video is transmitted.

[0005] The rate control system includes a hardware processor and memory for storing a computer program. When the computer program is executed by the hardware processor, it causes the hardware processor to use a machine learning model that takes an input set of video frames and the current channel capacity as input to determine, based on the current channel capacity, the values ​​of encoding parameters to use for the input set of video frames, to encode the input set of video frames using the encoding parameters to produce an encoded video having a bitrate less than the current channel capacity, and to transmit the encoded video.

[0006] These and other features and advantages will become apparent from the following detailed description of the exemplary embodiment, which will be read in conjunction with the attached drawings. [Brief explanation of the drawing]

[0007] This disclosure provides further details in the following description of preferred embodiments with reference to the following figures.

[0008] [Figure 1] This is a diagram illustrating the configuration of a communication system according to one embodiment of the present invention, which transmits video over a channel according to the channel's maximum capacity, using encoding parameters that are dynamically adjusted according to real-time information about channel quality.

[0009] [Figure 2] This is a block diagram of a real-time rate controller according to one embodiment of the present invention, which dynamically adjusts coding parameters according to real-time information on channel quality using a machine learning model.

[0010] [Figure 3] This is a block / flow diagram of a method for training and using a machine learning model to transmit video using real-time rate control, according to one embodiment of the present invention.

[0011] [Figure 4] This is a block diagram of a healthcare facility using video surveillance with real-time rate control to support medical decision-making, according to one embodiment of the present invention.

[0012] [Figure 5] This is a block diagram of a computing device capable of performing video encoding using real-time rate control, according to one embodiment of the present invention.

[0013] [Figure 6] This is a diagram of an exemplary neural network architecture that can be used to implement a portion of a prediction head according to one embodiment of the present invention.

[0014] [Figure 7] This is a diagram of an exemplary deep neural network architecture that can be used to implement a portion of a prediction head according to one embodiment of the present invention. [Modes for carrying out the invention]

[0015] Channel quality information, such as the Channel Quality Index (CQI), is reported from the user equipment (UE) to the transmitter almost instantaneously. This channel quality information represents the upper limit of the potential bandwidth of the channel between the transmitter and the UE. The UE can use this to dynamically update the encoding quality of the transmitted video so that it can use as much of the available bandwidth as possible without exceeding the channel limit.

[0016] To achieve this, a real-time rate controller is used, which processes an input video clip consisting of a set of video frames to generate features that represent the video clip. These features are processed by a prediction head along with an upper limit on the current bandwidth of the channel to generate estimated quantization parameters (QP) for the channel. Using these estimated QPs, the set of video frames is encoded and sent to the UE.

[0017] The estimated QP is selected to be just below the maximum bitrate of the channel. When a new set of video frames is provided, the estimate may be updated to accommodate changing channel conditions. For rapidly changing channel conditions, the number of video frames in a set may be reduced so that the QP estimate is updated more frequently.

[0018] Referring now to FIG. 1, an exemplary video transmission system is shown. Video camera 102 captures frames of a scene within its field of view and outputs image data. Each captured frame is encoded by encoder 104 into a compressed image that consumes less bandwidth during transmission and is decodable at UE 108. Encoder 104 has the function of adapting the encoding according to the QP. The higher the QP value, the lower the bitrate and the lower the image quality of the decoded image at UE 108.

[0019] Transmitter 106 transmits the encoded image to UE 108, which receives and decodes it to a state suitable for viewing. Transmission is shown as occurring via a wireless medium, but it should be understood that the present principles may apply to any suitable wired or wireless communication medium and protocol. Transmission is susceptible to channel effects and is characterized, for example, by the signal-to-noise ratio indicating how environmental factors affect UE 108's ability to receive the transmission without errors.

[0020] Based on the received transmission, UE 108 can identify the CQI, or other suitable representation of the channel quality. Subsequently, UE 108 may return the CQI to real-time rate controller 110. In some cases, UE 108 can send back channel quality information along the same channel as the transmission, such as by using time division to schedule the use of the channel between transmitter 106 and UE 108. In some cases, UE 108 may transmit the channel quality information over a separate channel with its own respective transmission characteristics.

[0021] The real-time rate controller 110 accepts channel quality information and determines the QP. Therefore, after one or more frames are encoded and sent to the UE 108, the UE 108 provides the latest estimate of the channel state. This new estimate is used by the real-time rate controller 110 to update the QP for use when encoding the next set of one or more frames. In this way, the system can quickly adapt to changing channel conditions.

[0022] Next, referring to Figure 2, the details of the real-time rate controller 110 are shown. Frames from camera 102 are provided as input to the real-time rate controller 110, as well as channel quality information, to generate QP as output. The real-time rate controller 110 can accept groups of such frames together so that the QP output is used to encode all frames in the group.

[0023] When encoding a set of video frames with a given QP, it is difficult to predict the actual bitrate of the encoded video stream. Generally, a higher QP value results in a higher bitrate, but the visual and dynamic complexity of the group of video frames may affect the size of the resulting video. Therefore, analytical or formal approaches to QP selection may result in a bitrate exceeding the maximum capacity of the channel, potentially leading to dropped frames and other errors. Such errors are immediately apparent to the user, and slight quality degradation resulting from selecting a slightly higher QP value may go unnoticed. However, being too conservative in QP selection can lead to underutilization of channel capacity.

[0024] QP selection may instead be performed using machine learning, where video features are extracted from video frames by a feature extractor 202. These features are then processed by a predictive head 204, which is trained to generate coding parameters to provide a bitrate close to the channel capacity without exceeding the channel capacity. While it is specifically assumed that the predictive head 204 generates QP values, it should be understood that any suitable coding parameters may be used, according to the coding standard used by the encoder 104. The machine learning model of the predictive head 204 is trained to recognize the bitrate needs of the input video frames and balance them with the limitations imposed by the channels.

[0025] Thus, channel quality information is used to identify a bitrate cap 206, which represents the currently available bandwidth of the channel. If more information is transmitted than this bitrate cap 206, some of it may not be recoverable by the UE108.

[0026] Encoder 104 processes group V of video frames and encodes the video using a specific height H, width W, and frame rate fps.

number

number

number

number

number

[0027] Channel coherence time represents the time it takes for the channel state to change significantly. In most situations, channel coherence time is relatively long compared to the duration of a set of video frames, so a bitrate cap of 206 can be assumed to be appropriate for the entire period over which a set of video frames is being transmitted. In some situations, such as a rapidly changing environment or when the UE108 is moving rapidly, channel coherence time may be shorter than the duration of a set of video frames. In such situations, the number of video frames processed at once can be reduced to increase the frequency with which the QP estimate is updated.

[0028] If the encoded bitrate of a set of video frames is below the bitrate cap of 206, all container packets should be received without error. However, if the encoded bitrate exceeds the bitrate cap of 206, some packets (and therefore some video frames) may be dropped. Fluctuations in the channel bitrate r(h) are due to the fading characteristics of channel h, and fluctuations in the video bitrate of a set of video frames must be taken into account to avoid exceeding the upper limit.

[0029] The feature extractor 202 may include a convolutional neural network layer that recognizes video content having dimensions [B, T, W, H] and outputs features with exemplary dimensions [B, 196, T, W / 16, H / 16], where B is the batch size, T is the number of video frames in the segment, W is the pixel-level width of the video frame, and H is the pixel-level height of the video frame. Each batch has T frames, and a total of B batches are used for training.

[0030] The prediction head 204 may be implemented as a deep neural network containing multiple convolutional layers, each following a Conditional Group Normalization (CGN) block. The CGN block normalizes the output from the previous layer. The prediction head sets the bitrate cap 206 to log, with size [B,1]. 10 (BR max It can be taken as a tensor as a conditioning factor. Each element of the tensor represents the bitrate cap for each video in the batch.

[0031] The prediction head can process the conditioning factor through three linear layers, each having a Gaussian Error Linear Unit (GELU) activation function that transforms the tensor size to [2B,1]. The tensor is then split into two tensors of size [B,1], γ and β. The γ and β tensors are used to adjust the normalized output from the previous layer using the equation γ × output + β to train video features to identify QP values ​​across various input video examples with varying bitrate caps of 206.

[0032] The training data for training the prediction head 204 consists of groups of video frames V, with a specified target quantization parameter QP. target , and QP target The corresponding bitrate BR linked to target It may include a set of video frames. Video frame V is BR target This is used as a conditional factor for the CGN block and as input to the feature extractor 202. The result generated by the prediction head is the estimate.

number

number

[0033] During training, target bitrate BR target This is not strictly treated as the upper bitrate limit. Therefore, during testing, the estimated QP may be slightly lower than the QP value that guarantees a bitrate below the bitrate cap of 206. This rounding effect can be mitigated by using a QP value slightly higher than the estimated QP value (to accommodate lower quality encoding) when encoding the data. This adjustment effectively prevents unwanted and destructive artifacts in UE108 while resulting in only a slight decrease in video quality. QP adjustments can be set by running examples with various video feeds to identify typical adjustment amounts.

[0034] The exact structure of the prediction head 204 may depend on the specific dimensions and types of information within the video frame. For example, the prediction head 204 may consist of a 3D convolutional layer with dimensions (196,196, kernel=(3,3,3)), a CGN layer and a GELU layer with dimensions (number of groups=4, condition), a 3D convolutional layer with dimensions (196,196, kernel=(1,1,1)), a CGN layer and a GELU layer with dimensions (number of groups=4, condition), a 3D convolutional layer with dimensions (196,196, kernel=(3,3,3)), and a C An output QP value between 1 and 52 is generated using a GN layer, a GELU layer, a 3D convolutional layer with dimensions (196, 52, kernel = (1, 1, 1)), a CGN layer with dimensions (number of groups = 4, condition), a GELU layer, a fully connected layer with dimensions (12480, 3120) and GELU activation, a fully connected layer with dimensions (3120, 780) and GELU activation, a fully connected layer with dimensions (780, 195) and GELU activation, and a fully connected layer with dimensions (195, 52). The number 52 may be replaced with any appropriate number of QP partitions for a given encoding standard.

[0035] Each CGN layer of the prediction head 204 may process a condition (e.g., bitrate cap 206) in a fully connected layer with dimensions (1,196) and GELU activation, a fully connected layer with dimensions (196,392) and GELU activation, and a fully connected layer with dimensions (392,392) to generate a tensor with dimensions [2B,1]. This tensor may be split into a γ tensor and a β tensor, each with dimensions 196, as described above. Next, group normalization (e.g., the number of groups is 4) is performed on the output of the previous layer (e.g., the output of the 3D convolutional layer), this normalized output is multiplied by γ, and the product is added to β.

[0036] Next, referring to Figure 3, a method for training and using a real-time rate control model is shown. Block 300 trains the prediction head 204 as described above using a set of training data. The training data may be any suitable set of videos in which a set of video frames is encoded according to a variety of different QP values. To prepare the training dataset, each video may be divided into uniform chunks according to the number of frames (e.g., 8 consecutive frames). The chunked videos are encoded using, for example, 52 different QP values. The set of encoded video chunks is sampled to generate a training set and a test set, and the parameters of the prediction head 204 are optimized according to the loss function described above. Training 300 takes an input pair containing a video chunk from the training dataset and the corresponding encoded bitrate and generates a predicted QP value. The predicted QP value is compared to the QP value used to encode the video chunk of the input pair.

[0037] The trained model is deployed in block 310. This includes deploying the model as software to the camera device 102, for example, as part of an encoding software module that operates on raw camera data.

[0038] During operation, block 320 transmits video data encoded with real-time rate control. This involves an iterative process in which, for each consecutive set of video frames, block 322 determines the QP to use, block 324 encodes the video frame using the determined QP value, and block 326 sends the encoded video to UE108. Block 328 retrieves and processes channel quality information from UE108 used by block 322 to determine a new QP value for the next iteration.

[0039] Next, referring to Figure 4, a diagram of information extraction in the setting of medical facility 400 is shown. Video surveillance 408 with real-time rate control can be used to monitor patients within medical facility 400 to ensure appropriate treatment and avoid dangerous situations. For example, video surveillance can be used in combination with behavior detection to determine whether a patient is adhering to the treatment plan or engaging in dangerous behavior. Wireless transmission conditions within medical facility 400 can change rapidly, for example, due to the opening and closing of doors or the turning on and off of diagnostic equipment, which can cause electromagnetic interference.

[0040] A healthcare facility may include one or more healthcare professionals 402 who review information extracted from the patient's medical records 406 and determine the patient's medical and treatment needs. These medical records 406 may include self-reported information from the patient, test results, and notes made by healthcare professionals in the patient's file. The treatment system 404 may also be designed to monitor the patient's condition to generate the medical records 406 and to automatically manage and adjust treatment as needed.

[0041] Based on information extracted from video surveillance 408 using real-time rate control, a medical professional 402 can make medical decisions regarding patient healthcare that are appropriate to the patient's needs. For example, the medical professional 402 can diagnose the patient's health condition and prescribe specific medications, surgeries, and / or therapies.

[0042] Various elements of the medical facility 400 can communicate with each other via the network 410, for example, using any suitable wired or wireless communication protocol and medium. In this way, the video surveillance 408 with real-time rate control acquires information about the patient and updates the medical record 406 with relevant visual information. The video surveillance 408 with real-time rate control can further output video for review by medical professionals 402. In some cases, the video can be used to automatically administer or modify treatment, sometimes in conjunction with the treatment system 404. For example, if the video shows a dangerous condition, the treatment system 404 can automatically stop administering treatment.

[0043] As shown in Figure 5, the arithmetic unit 500 exemplary includes a processor 510, an input / output subsystem 520, memory 530, a data storage device 540, and a communication subsystem 550, and / or other components and devices commonly found in a server or similar arithmetic unit. In other embodiments, the arithmetic unit 500 may include other or additional components (e.g., various input / output devices) commonly found in a server computer. Furthermore, in some embodiments, one or more exemplary components may be incorporated into another component or form part of another component. For example, memory 530, or part thereof, may be incorporated into the processor 510 in some embodiments.

[0044] The processor 510 can be embodied as any type of processor capable of performing the functions described herein. The processor 510 may be embodied as a single processor, a multiprocessor, a central processing unit (CPU), a graphics processing unit (GPU), a single or multicore processor, a digital signal processor, a microcontroller, or other processor or processing / control circuit.

[0045] Memory 530 may be embodied as any type of volatile or non-volatile memory or data storage capable of performing the functions described herein. During operation, memory 530 may store various data and software used during the operation of the arithmetic unit 500, such as operating systems, applications, programs, libraries, and drivers. Memory 530 may be communicably coupled to the processor 510 via the I / O subsystem 520 and may be embodied as circuits and / or components to facilitate input / output operations with the processor 510, memory 530, and other components of the arithmetic unit 500. For example, the I / O subsystem 520 may be embodied as, or otherwise include, a memory controller hub, an input / output control hub, a platform controller hub, an integrated control circuit, a firmware device, communication links (e.g., point-to-point links, bus links, wires, cables, light guides, printed circuit board traces, etc.) and / or other components and subsystems to facilitate input / output operations. In some embodiments, the I / O subsystem 520 may form part of a system-on-a-chip (SOC) and be integrated into a single integrated circuit chip together with other components of the processor 510, memory 530, and arithmetic unit 500.

[0046] The data storage device 540 can be embodied as any type of device or apparatus configured for short-term or long-term storage of data, such as a memory device and circuit, a memory card, a hard disk drive, a solid-state drive, or other data storage device. The data storage device 540 can store program code 540A for training a model, program code 540B for selecting QP values ​​for encoding video, and / or program code 540C for encoding video. The communication subsystem 550 of the arithmetic unit 500 can be embodied as any network interface controller or other communication circuit, apparatus, or assembly thereof that can enable communication between the arithmetic unit 500 and other remote devices over a network. The communication subsystem 550 can be configured to achieve such communication using any one or more communication technologies (e.g., wired or wireless) and associated protocols (e.g., Ethernet, InfiniBand®, Bluetooth®, Wi-Fi®, WiMAX®, etc.).

[0047] As shown in the figure, the arithmetic unit 500 may also include one or more peripheral devices 560. Peripheral devices 560 may include any number of additional input / output devices, interface devices, and / or other peripheral devices. For example, in some embodiments, peripheral devices 560 may include a display, a touchscreen, a graphics circuit, a keyboard, a mouse, a speaker system, a microphone, a network interface, and / or other input / output devices, interface devices, and / or peripheral devices.

[0048] Of course, the arithmetic unit 500 may include other elements (not shown) as readily conceivable to those skilled in the art, and certain elements may be omitted. For example, various other sensors, input devices, and / or output devices may be included in the arithmetic unit 500, depending on specific implementations of the same, as readily understood to those skilled in the art. For example, various types of wireless and / or wired input and / or output devices may be used. Furthermore, processors, controllers, memory, etc., may be added to enable various configurations. These and other variations of the processing system 500 are readily conceivable to those skilled in the art, given the teachings of the present invention provided herein.

[0049] Referring to Figures 6 and 7, exemplary neural network architectures are shown, including the prediction head 204, which can be used to implement parts of this model. A neural network is a generalized system whose functionality and accuracy improve with exposure to additional empirical data. Neural networks are learned by being exposed to empirical data. During training, the neural network remembers and adjusts multiple weights that are applied to the input empirical data. By applying the adjusted weights to the data, it can identify whether the data belongs to a predefined class from a set of classes, or output the probability that the input data belongs to each class.

[0050] The empirical data obtained from a series of examples (also called training data) is formatted as a string of values ​​and fed into the neural network. Each example is associated with a known result or output. Each column is represented as a pair (x,y), where x is the input data and y is the known output. The input data can be of various data types and may contain multiple different values. The network can have one input node for each value that makes up the example's input data, and each input value can be assigned a separate weight. The input data can be formatted as a vector, array, or string, for example, depending on the architecture of the neural network being built and trained.

[0051] A neural network "learns" by comparing the neural network output generated from input data with known values ​​from examples, and adjusting the stored weights to minimize the difference between the output and the known values. This adjustment can be performed on the stored weights through backpropagation, and the effect of the weights on the output is determined by calculating a mathematical gradient and adjusting the weights in a way that shifts the output to the minimum difference. This optimization, called gradient descent, is a non-restrictive example of how training takes place. A subset of examples with known values ​​not used in training can be used to test and validate the accuracy of the neural network.

[0052] During operation, the trained neural network can be used on new data that has not been previously used for training or validation through generalization. The weights of the tuned neural network can be applied to the new data, and the weights estimate the function developed from the training examples. The parameters of the estimated function captured by the weights are based on statistical inference.

[0053] In a layered neural network, the nodes are arranged in layers. An exemplary simple neural network has an input layer 620 of source nodes 622 and a single computational layer 630 having one or more computational nodes 632 that also function as output nodes, with a single computational node 632 for each possible category into which an input example can be classified. The input layer 620 can have a number of source nodes 622 equal to the number of data values 612 of the input data 610. The data values 612 of the input data 610 can be represented as column vectors. Each computational node 632 of the computational layer 630 generates a weighted linear combination of the values from the input data 610 supplied to the input nodes 620 and applies a non - linear activation function that is differentiable with respect to the sum. An exemplary simple neural network can perform classification on linearly separable examples (e.g., patterns).

[0054] Deep neural networks, such as multi - layer perceptrons, can have an input layer 620 of source nodes 622, one or more computational layers 630 having one or more computational nodes 632, and an output layer 640 having one output node 642 for each category into which an input example might be classified. The input layer 620 can have a number of source nodes 622 equal to the number of data values 612 of the input data 610. The computational nodes 632 of the computational layer 630 are between the source nodes 622 and the output nodes 642 and are not directly observable, so they are also called hidden layers. Each node 632, 642 of the computational layer generates a weighted linear combination of the values output from the nodes of the previous layer and applies a non - linear activation function that is differentiable over the range of the linear combination. The weights applied to the values from each previous node can be represented, for example, as w1, w2,... w n-1 , w n and so on. The output layer provides the overall response of the network to the input data. Deep neural networks can be fully connected, where each node of a computational layer is connected to all nodes of the previous layer, or the connections between layers can be in other configurations. If links between nodes are missing, the network is called partially connected.

[0055] Training a deep neural network involves two phases: a forward phase in which the weights of each node are fixed and the input is propagated through the network, and a backward phase in which error values ​​are propagated back through the network and the weight values ​​are updated.

[0056] One or more computational (hidden) layers 630 compute nodes 632 perform a nonlinear transformation on the input data 612 that generates the feature space. Classes and categories may be easier to separate in the feature space than in the original data space.

[0057] The embodiments described herein may be entirely hardware, entirely software, or may include both hardware and software elements. In preferred embodiments, the present invention is implemented in software including, but not limited to, firmware, resident software, and microcode.

[0058] Embodiments may include computer program products accessible from computer-enabled or computer-readable media that provide program code for use by or in connection with a computer or any instruction execution system. Computer-enabled or computer-readable media may include any device that stores, communicates, propagates, or transports programs for use by or in connection with an instruction execution system, apparatus, or device. The medium may be magnetic, optical, electronic, electromagnetic, infrared, or semiconductor systems (or apparatus or devices), or propagation media. The medium may include computer-readable storage media such as semiconductor or solid-state memory, magnetic tape, removable computer diskettes, random-access memory (RAM), read-only memory (ROM), rigid magnetic disks, and optical disks.

[0059] Each computer program can be substantially stored in a machine-readable storage medium or device (e.g., program memory or magnetic disk) that is readable by a general-purpose or special-purpose programmable computer, in order to configure and control the operation of the computer when the storage medium or device is read by the computer in order to perform the procedures described herein. The system of the present invention can also be considered to be implemented on a computer-readable storage medium configured with a computer program, in which case the configured storage medium causes the computer to operate in a specific predetermined manner to perform the functions described herein.

[0060] A data processing system suitable for storing and / or executing program code may include at least one processor directly or indirectly coupled to a memory element via a system bus. The memory element may include local memory, bulk storage, and cache memory that provides at least some temporary storage for the program code to reduce the number of times the code is retrieved from bulk storage during execution. Input / output or I / O devices (including, but not limited to, keyboards, displays, pointing devices, etc.) may be coupled to the system directly or via an intermediary I / O controller.

[0061] Network adapters can also be integrated into a system to enable a data processing system to connect to other data processing systems or remote printers or storage devices via an intervening private or public network. Modems, cable modems, and Ethernet cards are just a few of the types of network adapters currently available.

[0062] As used herein, the terms “hardware processor subsystem” or “hardware processor” may refer to a processor, memory, software, or combination thereof that works together to perform one or more specific tasks. In useful embodiments, a hardware processor subsystem may include one or more data processing elements (e.g., logic circuits, processing circuits, instruction execution devices, etc.). One or more data processing elements may be included in a central processing unit, a graphics processing unit, and / or a separate processor- or arithmetic element-based controller (e.g., logic gates, etc.). A hardware processor subsystem may include one or more onboard memories (e.g., caches, dedicated memory arrays, read-only memory, etc.). In some embodiments, a hardware processor subsystem may include one or more memories (e.g., ROM, RAM, Basic Input / Output System (BIOS), etc.) that may be onboard or offboard, or that may be dedicated for use by the hardware processor subsystem.

[0063] In some embodiments, a hardware processor subsystem may include and execute one or more software elements. These software elements may include an operating system and / or one or more applications and / or specific code to achieve a specified result.

[0064] In other embodiments, the hardware processor subsystem may include dedicated circuits that perform one or more electronic processing functions to achieve a specified result. Such circuits may include one or more application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), and / or programmable logic arrays (PLAs).

[0065] These and other variations of the hardware processor subsystem are also intended in accordance with embodiments of the present invention.

[0066] In this specification, any reference to “one embodiment” or “one embodiment” of the present invention, and to other modifications, means that certain features, structures, properties, etc., described in relation to the embodiments are included in at least one embodiment of the present invention. Therefore, expressions such as “in one embodiment” or “in one embodiment” appearing elsewhere in this specification, and any other modifications, do not necessarily all refer to the same embodiment. However, it should be understood that, considering the teachings of the present invention provided herein, features of one or more embodiments can be combined.

[0067] For example, in the case of "A / B," the use of any of the following " / ," "and / or," or "at least one," such as "A and / or B" or "at least one of A and B," will be understood as intended to include the selection of only the first listed option (A), only the second listed option (B), or both options (A and B). As further examples, in the case of "A, B, and / or C" and "at least one of A, B, and C," such expressions are intended to include the selection of only the first listed option (A), only the second listed option (B), only the third listed option (C), only the first and second listed options (A and B), only the first and third listed options (A and C), only the second and third listed options (B and C), or all three options (A, B, and C). This can be extended as many times as there are listed items.

[0068] The foregoing is to be understood in all respects to be illustrative and not restrictive, and the scope of the invention disclosed herein is to be determined not from the detailed description but from the claims as interpreted in accordance with the full width permitted by patent law. The embodiments shown and described herein are merely illustrative of the invention, and those skilled in the art should understand that various modifications can be implemented without departing from the scope and spirit of the invention. Those skilled in the art can implement various other combinations of features without departing from the scope and spirit of the invention. Thus, while aspects of the invention have been described with the detail and specificity required by patent law, what is claimed and intended to be protected by the patent is as stated in the appended claims.

Claims

1. A method for rate control implemented in a computer, Using a machine learning model that accepts a set of input video frames and the current channel capacity as input, the values ​​of the encoding parameters to be used for the input set of video frames are determined based on the current channel capacity (322). Using the aforementioned encoding parameters, the input set of video frames is encoded (324) to produce an encoded video having a bitrate less than the current channel capacity. A method comprising transmitting the encoded video (326).

2. In the method according to claim 1, A method further comprising determining the current channel capacity based on channel quality information received from user equipment.

3. In the method according to claim 1, A method comprising a predictive head model trained to generate parameter values ​​such that, when used to encode the input set of video frames, the encoded video is less than or equal to the current channel capacity.

4. In the method according to claim 3, The method wherein the predictive head model is a deep neural network model that includes conditional group normalization using the current channel capacity as a condition.

5. In the method according to claim 4, The aforementioned predictive head model includes multiple convolutional layers, and each convolutional layer is followed by a conditional group normalization.

6. In the method according to claim 1, A method in which the coding parameter is a quantization parameter.

7. In the method according to claim 1, A method further comprising changing the values ​​of the determined encoding parameters in order to degrade the video quality before encoding the video.

8. In the method according to claim 1, A method for determining the values ​​of the encoding parameters includes extracting features from the input set of video frames and processing the features in a predictive head model using the current channel capacity.

9. In the method according to claim 1, A method by which the encoded video is transmitted to a medical professional to assist in medical decision-making.

10. In the method according to claim 1, A method further comprising performing a therapeutic action in response to the encoded video, and automatically modifying the patient's treatment in response to the patient's actions shown in the encoded video.

11. In the method according to claim 1, A method for determining the values ​​of the encoding parameters, which includes maximizing the average video quality of a live video feed and minimizing the probability of packet drops and video artifacts in the transmission of the encoded video, subject to the current channel capacity available for the set of video frames.

12. A system for rate control, Hardware processor (510), The system has a memory (540) for storing a computer program, and when the computer program is executed by the hardware processor, the hardware processor has a memory (540) for storing a computer program, A machine learning model that accepts a set of input video frames and the current channel capacity as input is used to determine the values ​​of the encoding parameters to be used for the set of input video frames based on the current channel capacity (322). Using the encoding parameters, the input set of video frames is encoded (324) to produce an encoded video having a bitrate less than the current channel capacity. A system (326) for transmitting the encoded video.

13. In the system according to claim 12, The computer program is a system that causes the hardware processor to determine the current channel capacity based on channel quality information received from user equipment.

14. In the system according to claim 12, The machine learning model is a system that includes a predictive head model trained to generate parameter values ​​such that, when used to encode the input set of video frames, the encoded video is less than or equal to the current channel capacity.

15. In the system described in claim 14, The predictive head model is a deep neural network model that includes conditional group normalization using the current channel capacity as a condition.

16. In the system described in claim 15, The aforementioned predictive head model is a system that includes multiple convolutional layers, with each convolutional layer followed by its respective conditional group normalization.

17. In the system according to claim 12, The computer program is a system that causes the hardware processor to further modify the values ​​of the determined encoding parameters in order to degrade the video quality before encoding the video.

18. In the system according to claim 12, The computer program is a system that causes the hardware processor to further extract features from the input set of video frames and to process the features in a predictive head model using the current channel capacity.

19. In the system according to claim 12, The encoded video is transmitted to a medical professional to assist in medical decision-making.

20. In the system according to claim 12, The computer program is a system that further includes causing the hardware processor to perform therapeutic actions in response to the encoded video and automatically modifying the patient's treatment in response to the patient's actions shown in the encoded video.

Images