Anchor enhancer based on neural acceleration with selective super-resolution video encoding acceleration function and anchor enhancement method using the same
The neural accelerator-based anchor enhancer addresses bandwidth limitations and computational costs in live streaming by preprocessing video streams, applying neural inference to selected frames, and using a lightweight codec to maintain high-quality video delivery efficiently.
Patent Information
- Application Number
- KR1020230095456
- Authority / Receiving Office
- KR · KR
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2022-07-25
- Filing Date
- 2023-07-21
- Publication Date
- 2026-11-11
- Estimated Expiration
- 2043-07-21
AI Technical Summary
Current live streaming technologies struggle to consistently deliver high-quality video due to bandwidth limitations, leading to significant viewer attrition and high computational costs for neural reinforcement, which is too expensive for commercial-scale streaming.
A neural accelerator-based anchor enhancer with selective super-resolution video encoding acceleration function that preprocesses video streams, applies neural network inference only to selected anchor frames, and uses a lightweight codec to compress high-resolution frames while offloading non-anchor frame reconstruction to clients, optimizing GPU context switching and reducing computational overhead.
Enables efficient and scalable neural enhancement for live streams by processing end-to-end neural enhancement at selective super-resolution speeds, reducing computational costs and maintaining video quality without bottlenecks.
Smart Images

Figure R1020230095456_ABST
Abstract
Description
Technology Field
[0001] The present invention relates to a neural accelerator-based anchor enhancer having a selective super-resolution video encoding acceleration function and an anchor enhancing method using the same, and more specifically, to a neural accelerator-based anchor enhancer having a selective super-resolution video encoding acceleration function that provides efficient and scalable neural enhancement for live streams and an anchor enhancing method using the same. Background Technology
[0002] The demand for live streaming has increased rapidly. Live video traffic is expected to account for 17% of internet traffic by 2022. Current live streaming infrastructure relies on two key elements. Video is transmitted to media servers using low-latency streaming protocols. In terms of distribution, clients run an Adaptive Bitrate (ABR) algorithm to select the highest quality video capable of real-time streaming.
[0003] Unfortunately, because ingest video quality depends heavily on the streamer's uplink bandwidth, traditional live streaming is insufficient to consistently deliver high-quality video (e.g., 4K / 8K). When the ingest path becomes congested, the overall downstream video quality degrades directly. However, video quality is the most critical factor affecting user engagement in live streaming. For example, more than 50% of live viewers abandon a video stream if the video quality drops for more than 90 seconds. This viewer attrition can deal a significant blow to the revenue of live streaming providers.
[0004] Recent advancements in neural reinforcement streaming demonstrate significant potential for improving ingested video quality by leveraging computations on media servers. When ingested video quality degrades, the media server recovers high-quality video by executing end-to-end neural reinforcement consisting of low-quality stream decoding, the application of ultra-high-resolution Deep Neural Networks (DNNs), and super-resolved output encoding. This provides dramatic quality improvements in downstream video.
[0005] However, neural reinforcement is too expensive to support commercial-scale live streaming. For example, Twitch supports over 100,000 concurrent live streams. Applying end-to-end neural reinforcement in this setup requires tens of thousands of GPUs (graphics processing units) and costs over $169,000 per hour on a public cloud. The cost breakdown shows that both video super-resolution and encoding are expensive. Neural super-resolution requires 100 to 1,000 times more computation than DNNs used for discrimination tasks, and video encoding is up to 3.3 times slower than super-resolution. Prior art literature
[0006] Korean Registered Patent No. 10-2313136 (October 15, 2021) The problem to be solved
[0007] Accordingly, the technical problem of the present invention is based on this point, and the objective of the present invention is to provide a neural accelerator-based anchor enhancer having an optional super-resolution video encoding acceleration function capable of processing end-to-end neural enhancement at an optional super-resolution speed by utilizing a lightweight codec capable of compressing high-resolution streams as fast as optional inference to provide efficient and scalable neural enhancement for live streams.
[0008] Another objective of the present invention is to provide an anchor enhancing method using the anchor enhancer described above. means of solving the problem
[0009] According to one embodiment, to realize the objective of the present invention described above, a neural accelerator-based anchor enhancer having a selective super-resolution video encoding acceleration function comprises: a preprocessing processor that selects the most advantageous anchor frame from a video stream and preprocesses a Deep Neural Network (DNN) for each video stream and a DNN provided by an anchor scheduler that outputs the selected anchor frame; an inference engine that converts a decoded image into a high-resolution image through neural network inference when the image is input to a Graphics Processing Unit (GPU) or Neural Processing Unit (NPU); and an encoder that reuses the input video stream, compresses only the super-resolution anchor frames using an image codec, simultaneously offloads the reconstruction of non-anchor frames to a client, and packages the encoded video and the super-resolution anchor frames into a single file.
[0010] In one embodiment, the preprocessor can optimize GPU context switching by running a super-high resolution DNN in a framework optimized for inference.
[0011] In one embodiment, the preprocessing processor may take a mock DNN that is randomly initialized before live streaming starts, pre-optimize it on a target accelerator, and then, if optimization of the DNN is required during live streaming, generate an optimized version using the optimized mock DNN.
[0012] In one embodiment, the encoder can compress only the super-high resolution anchor frames while offloading the reconstruction of non-anchor frames to the client, and package the encoded video and the super-high resolution anchor frames into a single file.
[0013] In one embodiment, the size of each of the anchor frames can be set equally to satisfy the bit rate constraints of live streaming.
[0014] In one embodiment, a new header attached to each frame may include a frame type and a super-resolution frame of the anchor frame.
[0015] To realize another objective of the present invention described above, an anchor enhancing method according to one embodiment comprises: (i) selecting the most beneficial anchor frame from a video stream and preprocessing a deep neural network (DNN) for each video stream; (ii) converting a decoded image into a high-resolution image through neural network inference when the image is input to a Graphics Processing Unit (GPU) or Neural Processing Unit (NPU); and (iii) reusing an input video stream and using an image codec to compress only the super-high-resolution anchor frame while offloading the reconstruction of non-anchor frames to a client, and packaging the encoded video and the super-resolution anchor frame into a single file.
[0016] In one embodiment, the step (i) may include: (i-1) taking a mock DNN that is randomly initialized before live streaming starts and pre-optimizing it on a target accelerator; and (i-2) generating an optimized version using the mock DNN optimized offline when optimization of the DNN is required during live streaming.
[0017] In one embodiment, the step (iii) may include: (iii-1) reusing the input video stream and using an image codec to compress only the super-high resolution anchor frames while offloading the reconstruction of non-anchor frames to the client; and (iii-2) packaging the encoded video and the super-high resolution anchor frames into a single file. Effects of the invention
[0018] According to the neural accelerator-based anchor enhancer with such selective super-resolution video encoding acceleration capabilities and the anchor enhancing method using the same, end-to-end neural enhancement requires re-encoding, but since existing video encoders are computationally expensive and cause bottlenecks, end-to-end neural enhancement can be processed at selective super-resolution speeds by utilizing a lightweight codec that can compress high-resolution streams as fast as selective inference to prevent bottlenecks while providing quality similar to existing codecs. Brief explanation of the drawing
[0019] Figure 1 is a conceptual diagram illustrating conventional streaming and neural network-enhanced streaming (an example of adaptive streaming). Figure 2 is a conceptual diagram for schematically explaining selective super-resolution. FIG. 3 is a conceptual diagram for schematically explaining a neuroscaler according to one embodiment of the present invention. Figure 4 is a conceptual diagram illustrating the distribution model of a neural scaler for adaptive streaming. Figure 5 is a conceptual diagram for schematically explaining the anchor scheduler illustrated in Figure 3. FIG. 6 is a conceptual diagram for schematically explaining the anchor selector illustrated in FIG. 3. FIG. 7 is a conceptual diagram for explaining the overall operation of a hybrid video codec unit including an encoder illustrated in FIG. 3. Specific details for implementing the invention
[0020] Hereinafter, with reference to the attached drawings, a neural scaler according to the present invention and a neural scaling method using the same will be described in more detail. Since the present invention is susceptible to various modifications and may take various forms, specific embodiments are illustrated in the drawings and described in detail in the text. However, this is not intended to limit the present invention to the specific disclosed forms, and it should be understood that it includes all modifications, equivalents, and substitutions that fall within the spirit and scope of the present invention.
[0021] In describing each drawing, similar reference numerals have been used for similar components. In the attached drawings, the dimensions of the structures are depicted enlarged compared to their actual size to ensure clarity of the invention.
[0022] Terms such as "first," "second," etc., may be used to describe various components, but said components should not be limited by said terms. These terms are used solely for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be named the second component, and similarly, the second component may be named the first component. A singular expression includes a plural expression unless the context clearly indicates otherwise.
[0023] In this application, terms such as "comprising" or "having" are intended to specify the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.
[0024] Furthermore, unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as generally understood by those skilled in the art to which the present invention pertains. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and should not be interpreted in an ideal or overly formal sense unless explicitly defined in this application.
[0025] Hereinafter, the present invention will be described in detail with reference to the attached drawings.
[0026] Live streaming consists of collection and distribution aspects as shown in Fig. 1.
[0027] Figure 1 is a conceptual diagram illustrating conventional streaming and neural network-enhanced streaming (an example of adaptive streaming).
[0028] Referring to Fig. 1, in terms of collection, the streamer captures live video and uploads it to a media server. In adaptive streaming, the streamer uploads a single video stream, and the media server transcodes it into multiple quality versions.
[0029] In a multi-party video conference, the broadcaster uploads video streams of various qualities using simultaneous broadcasting or a Scalable Video Codec (SVC), and the media server delivers these video streams to viewers. Next, on the distribution side, the client executes an Adaptive Bitrate (ABR) algorithm to regulate video streaming quality, selecting and downloading the highest quality video within a given network bandwidth. The ABR algorithm can be classified into heuristic-based and machine learning-based methods. Heuristic-based methods are further classified based on network bandwidth, client buffer occupancy, and hybrid methods that combine both.
[0030] A common limitation of traditional live streaming is that video quality is restricted by the streamer's uplink bandwidth. Even if viewers have sufficient network bandwidth, this limitation may deprive them of the opportunity to enjoy high-quality video.
[0031] Super-resolution is a type of technique that generates high-resolution images from low-resolution images. Recent research uses deep neural networks (DNNs) to learn mapping from low resolution to high resolution and demonstrates dramatic quality improvements.
[0032] Neural reinforcement live streaming uses a super-resolution DNN to enhance the collected video stream as shown in Fig. 1. Unlike conventional live streaming, the media server performs end-to-end neural reinforcement. When the video arrives, it is first decoded into raw frames. Then, the DNN is applied to all or some frames according to a super-resolution method. The output is re-encoded into video before being delivered to the client.
[0033] To provide stable quality improvement, content-aware DNNs are typically used for each video stream and can be updated even during live streaming. Context switching between content-aware DNNs causes two types of overhead.
[0034] First, modern DNN compilers provide optimized inference for target accelerators, but this involves upfront costs due to model optimization that can take up to several minutes. Both graph-level and operator-level optimizations are applied to generate a DNN tailored to a specific target.
[0035] Second, the DNN and frames must be loaded into the device memory of the target accelerator before executing inference. Since neural accelerators generally have limited memory sizes (e.g., 16GB for the NVIDIA T4), multiple content recognition DNNs, each requiring several GB of memory, must be frequently swapped.
[0036] Selective super-resolution (SR) reduces computing overhead by utilizing temporal redundancy across entire video frames. Neural inference is applied only to selected frames called anchor frames.
[0037] Figure 2 is a conceptual diagram for schematically explaining selective super-resolution.
[0038] As can be seen in Fig. 2, non-anchor frames are reconstructed by reusing previous super-resolution frames guided by codec-level information (e.g., reference index, motion vector, residual). This includes lightweight bilinear interpolation and decoding, so it can be processed in real time even on mobile devices.
[0039] Upscaling non-anchor frames by reusing previous super-resolution frames inevitably results in quality loss due to time lag (compared to per-frame inference). This loss accumulates in consecutive non-anchor frames but is reset in anchor frames. Therefore, to maximize quality, it is important to select a beneficial set of anchor frames. To this end, the NEMO algorithm relies on costly per-frame inference. Consequently, it cannot be used for live streaming where anchor frame selection must be performed online in real time.
[0040] FIG. 3 is a conceptual diagram for schematically explaining a neuroscaler according to one embodiment of the present invention.
[0041] Referring to FIG. 3, a NeuroScaler according to one embodiment of the present invention includes an Anchor scheduler (100) and an Anchor enhancer (200) to take a pair of collected video streams and a DNN and output a high-resolution video stream. The DNN may be dynamically updated through online learning, such as in LiveNAS.
[0042] The anchor scheduler (100) decodes the collected video stream and selects the most favorable anchor frame from the video stream. Here, the anchor frame refers to a frame to which the super-resolution is applied using codec-level information in real time, while expanding the optional super-resolution to accelerate the video super-resolution. Then, for each video stream, the anchor scheduler (100) transmits the DNN and the selected anchor frame to an anchor enhancer (200) equipped with a neural accelerator.
[0043] The anchor enhancer (200) has a selective super-resolution video encoding acceleration function based on a neural accelerator, preprocesses the DNN, applies the preprocessed DNN to the anchor frame, and re-encodes the super-resolution output.
[0044] In this embodiment, all processes of the anchor scheduler (100) and anchor enhancer (200) are pipelined and parallelized to maximize overall throughput.
[0045] According to the neuroscaler of the present invention, by addressing the limitations of neural network augmentation live streaming, the end-to-end neural network augmentation cost can be reduced and the overall quality improvement in a computing cluster can be maximized.
[0046] Figure 4 is a conceptual diagram illustrating the deployment model of a neuroscaler for adaptive streaming.
[0047] Referring to Fig. 4, when a low-quality video stream (e.g., 360p or 720p) is uploaded to a media server, the neuroscaler generates a high-quality video stream using real-time super-resolution.
[0048] Low-resolution versions (e.g., 240p to 720p) are also generated from video streams collected using existing transcoding pipelines. By using a neuroscaler, clients can view high-resolution video even when the collection path is congested. Deploying a neuroscaler for video conferencing is similar to the process above, but multi-resolution transcoding is not required.
[0049] FIG. 5 is a conceptual diagram for schematically explaining the anchor scheduler (100) illustrated in FIG. 3.
[0050] Referring to FIGS. 3 and FIGS. 5, the anchor scheduler (100) includes a decoder (110) and a resource management module (120).
[0051] The decoder (110) decodes a video stream provided from the outside and provides it to the resource management module (120). The decoded video stream may be provided directly to the resource management module (120) or may be provided to the resource management module (120) via memory.
[0052] The resource management module (120) includes an anchor selector (122) and an anchor-level load balancer (124) to distribute the load while optimally allocating computing resources to each video stream. In this embodiment, the resource management module (120) distributes the load while optimally allocating computing resources to each video stream to maximize quality improvement when processing a large number of video streams. In particular, the resource management module (120) executes global anchor frame selection on all video streams to select an optimal set of anchor frames and distributes the load to instances in anchor frame units, which can accurately predict the overhead of the anchor frames. In this embodiment, the anchor scheduler (100) has been described as including an anchor selector (122) and an anchor-level load balancer (124), but this is a logical distinction made for convenience of explanation only and is not a hardware distinction.
[0053] The anchor selector (122) runs on a centralized server and selects the most favorable anchor frame from all video streams. In particular, the anchor selector (122) periodically runs a zero-inference algorithm by selecting the maximum number of anchor frames that can be processed in real time on a computing cluster, as shown in the following formula (1).
[0054] [Formula 1]
[0055]
[0056] Here, is an anchor frame aligned with the anchor gain, is the DNN latency, is the anchor frame selection interval which is a configurable parameter, and is the number of compute instances in the cluster.
[0057] Unless otherwise noted, this specification uses = 666ms (40 frames for 60fps video). This global anchor frame selection can mitigate the imbalance of anchor frames per video stream.
[0058] In general, using longer scheduling intervals allows for the selection of more influential anchor frames, thereby improving quality. Therefore, the interval must be adjusted according to application requirements.
[0059] An anchor-level load balancer (124) selects an anchor frame and then dynamically balances the load between computing instances in anchor frame granularity. To do this, an anchor scheduler (100) divides the selected anchor frame into instance-specific groups, and the frames of each group are delivered to the corresponding computing instances for processing.
[0060] To meet real-time constraints, the anchor scheduler (100) allocates a maximum number of anchor frames for each group where the total waiting time is less than the anchor frame selection interval (). This anchor-level load balancing can alleviate the imbalance of load per instance.
[0061] FIG. 6 is a conceptual diagram for schematically explaining the anchor selector (122) illustrated in FIG. 5.
[0062] Referring to FIGS. 5 and 6, the anchor selector (122) includes a frame splitting unit (122a), an anchor gain estimation unit (122b), a group merging and alignment unit (122c), and an anchor frame selection unit (122d), and selects a beneficial anchor frame by utilizing codec-level information (e.g., frame type, residual). In this embodiment, the anchor selector (122) is described as including a frame splitting unit (122a), an anchor gain estimation unit (122b), a group merging and alignment unit (122c), and an anchor frame selection unit (122d), but this is a logical distinction made for convenience of explanation and is not a hardware distinction.
[0063] The frame splitting unit (122a) divides the frames into video stream-specific groups according to the type of each video stream. That is, the frame splitting unit (122a) divides the frames into key frame groups ( ), alternate reference frame group( ) and other frame groups( Divided by ). Here, the group , and There is a priority order for selecting anchor frames.
[0064] The anchor gain estimator (122b) is an alternative reference frame group ( ) and other frame groups( For the frames of ), the benefit of using anchor frames, called anchor gain, is estimated. Anchor gain is calculated based on accumulated residuals using Algorithm 1 described below. Key frame group ( In the case of ), it is assumed that there are only a few key frames and generally all are selected as anchor frames, so it has the same anchor gain as the overall performance.
[0065] The anchor gain estimation unit (122b) includes a residual calculation module (not shown), a remaining amount calculation module (not shown), an anchor gain setting module (not shown), and an update module (not shown). In this embodiment, the anchor gain estimation unit (122b) is described as including a residual calculation module, a remaining amount calculation module, an anchor gain setting module, and an update module; however, this is a logical distinction made for convenience of explanation and is not a hardware distinction. The residual calculation module calculates the residual accumulated over the entire frame. The remaining amount calculation module calculates the remaining amount that decreases for each frame for each iteration. The anchor gain setting module selects the frame that reduces the residual the most and sets the anchor gain to the amount of the reduced residual. The update module updates the accumulated residual of each frame.
[0066] The group merging and alignment unit (122c) merges video stream-specific groups into a global group. That is, a key frame global group ( ), Alternative reference frame global group( ), and other Frame Global Group( Each contains a key frame, an alternate reference frame, and a normal frame of the entire video stream. Frames in the same group are aligned according to anchor gain.
[0067] The anchor frame selection unit (122d) is a key frame global group ( Alternate reference frame global group in ) ) and other Frame Global Groups( Iteratively selects anchor frames from global groups aligned up to ).
[0068] In this embodiment, the neuroscaler measures the latency of the DNN once to satisfy real-time constraints and selects the maximum number of anchor frames where the total latency is shorter than the available computing time.
[0069] Below, we will explain how the neuroscaler estimates the anchor gain using residuals through Algorithm 1.
[0070] Algorithm 1 shows how the neuroscaler estimates the gain of the anchor frame.
[0071]
[0072] Algorithm 1 first calculates the accumulated residuals over the entire frame (Algorithm 1, line #2).
[0073] Next, the most favorable frame is repeatedly selected and the anchor gain is estimated. That is, for each iteration, the remaining amount of each frame that is reduced is calculated as in the following formula (2) (lines #6-8).
[0074] [Equation 2]
[0075]
[0076] Here, Res(F[j]) is the accumulated residual of the j-th frame, and k is the index of the nearest frame where the residual is reset. The residual is cleared from the key frame or the frame where the anchor gain was estimated in the previous iteration. If such a frame does not exist within the given frame, the algorithm predicts that the residual is reset from the key frame of the next video chunk.
[0077] Next, select the frame that reduces the residual the most and set the anchor gain to the amount of the reduced residual (lines 12, 13).
[0078] Finally, the accumulated residuals of each frame are updated to reflect the influence of the selected frame (Line #14). To quickly estimate the anchor gain, the total residual pixel value is approximated by the size of the encoded residual frame. Since both values are highly correlated, this has minimal impact on quality (= in PSNR). 0.05dB).
[0079] Referring again to FIG. 3, the anchor enhancer (200) includes a preprocessing processor (210), an inference engine (220), and an encoder (230).
[0080] The preprocessing processor (210) preprocesses the DNN provided by the anchor scheduler (100).
[0081] The preprocessor (210) optimizes GPU (graphics processing unit) context switching to efficiently run a super-high-resolution DNN in an inference-optimized framework (e.g., TensorRT, TVM). If the framework is used naively in neural reinforcement live streaming, the speed is severely degraded due to the following two types of GPU context switching overhead.
[0082] The first issue is the DNN update problem. While model optimization can speed up inference, it typically takes from seconds to minutes depending on the DNN size. Therefore, it is difficult to translate the benefits of model optimization into a live streaming context. Since DNNs can be updated online, optimization must be performed in real time. In this embodiment, to reduce optimization latency, attention is paid to the fact that the results of model optimization have low correlation with actual weight values. Based on this, a model pre-optimization method is proposed in which optimization is performed offline and occurs only once.
[0083] The preprocessing processor (210) takes a randomly initialized mock DNN before live streaming starts and pre-optimizes it on the target accelerator.
[0084] Next, the preprocessing processor (210) generates an optimized version using a mock DNN optimized offline when optimization of the DNN is required during live streaming. This involves replacing the parameters of the mock DNN with the parameters of the target DNN. This scheme reduces the waiting time from tens of seconds to several milliseconds.
[0085] The second issue is the DNN loading problem. Since multiple DNNs / frames arrive at each anchor enhancer (200) per scheduling interval, device / host memory must be frequently allocated and deallocated. This overhead is similar to the overhead of neural inference for high resolution (4K / 8K), as each DNN requires several MB / GB of host and device memory. To avoid frequent memory allocation overhead, the preprocessor (210) pre-allocates host / device memory and configures the memory pool when the program starts.
[0086] The inference engine (220) converts a decoded image into a high-resolution image through neural network inference when the image is input into the GPU or NPU.
[0087] The encoder (230) reuses the input video stream and uses an image codec to compress only the super-high resolution anchor frames while offloading the reconstruction of non-anchor frames to the client, and packages the encoded video and the super-high resolution anchor frames into a single file.
[0088] End-to-end neural reinforcement requires re-encoding, but existing video encoders (230) (e.g., VPx, H. 26x) are computationally expensive and become a major computational bottleneck. Therefore, a lightweight codec unit is proposed that can compress high-resolution video streams as fast as selective inference without bottlenecks while providing quality similar to existing codecs. Using a lightweight codec unit allows end-to-end neural reinforcement to be processed at selective super-resolution speeds. This is because non-anchor frames can be easily reconstructed on the client side, even on commercial mobile devices, without significant overhead.
[0089] This workaround allows the target video to be encoded very quickly. Additionally, since non-anchor frames are transmitted directly at lower resolution without upscaling, there is more room in terms of bandwidth to encode anchor frames at high quality.
[0090] Based on this, a hybrid video codec unit specialized in encoding selective ultra-high resolution output was devised.
[0091] FIG. 7 is a conceptual diagram for explaining the overall operation of a hybrid video codec unit (300) including an encoder (230) illustrated in FIG. 3.
[0092] Referring to FIGS. 3 and FIGS. 7, the hybrid video codec unit (300) includes a hybrid encoder (310) placed on the server side and a hybrid decoder (320) placed on the client side. In this embodiment, the hybrid encoder (310) corresponds to the encoder (230) placed on the anchor enhancer (200) shown in FIG. 3, and the hybrid decoder (320) may be placed on a client-side terminal that plays live streaming.
[0093] Unlike conventional encoders, the hybrid encoder (310) utilizes both video and image codec sections in a synergistic manner. The hybrid encoder (310) reuses the input video stream and uses the image codec section to compress only the super-resolution anchor frames while offloading the reconstruction of non-anchor frames to the client. Each anchor frame size is set equally to satisfy the bitrate constraints of live streaming. Next, the hybrid encoder (310) packages the encoded video and super-resolution anchor frames into a single file to be delivered to the client later. A new header attached to each frame includes the frame type (e.g., anchor or non-anchor frame) and the super-resolution frame of the anchor frame.
[0094] The hybrid encoder (310) has two main advantages.
[0095] First, computing overhead can be reduced by 78.6 to 235.8 times compared to a video encoder. This is because the hybrid encoder (310) applies a lightweight image codec that is 6.25 times cheaper than a video encoder to only a few anchor frames (5 to 10% of the total frames).
[0096] Second, because anchor frames are rarely distributed within the video chunk, the impact on compression efficiency is minimized even when an image codec is applied to a few frames. The Bjontegaard rate difference (BD-rate) is improved by 6.69% when using the hybrid encoder (310) compared to re-encoding the video using a conventional encoder.
[0097] A hybrid decoder (320) placed on the client side decodes a video stream encoded by a hybrid encoder (310). When a compressed frame arrives, the hybrid decoder (320) first checks if the current frame is an anchor frame. If it is confirmed that the current frame is an anchor frame, the hybrid decoder (320) decodes the super-resolution frame and caches it in memory. Meanwhile, if it is confirmed that the current frame is not an anchor frame, the hybrid decoder (320) reuses a previous super-resolution frame to enlarge the current frame using codec-level information. Decoding a hybrid encoded video stream results in minimal overhead.
[0098] As described above, according to the present invention, end-to-end neural reinforcement requires re-encoding, but since conventional video encoders are computationally expensive and cause bottlenecks, end-to-end neural reinforcement can be processed at selective super-resolution speeds by using a lightweight codec that can compress high-resolution streams as quickly as selective inference without causing bottlenecks while providing quality similar to existing codecs.
[0099] Components according to embodiments of the present invention may be implemented in the form of software or hardware such as an FPGA (Field Programmable Gate Array) or ASIC (Application Specific Integrated Circuit) and may perform certain roles. However, the term 'components' is not limited to software or hardware, and each component may be configured to reside in an addressable storage medium or configured to run one or more processors.
[0100] Accordingly, as an example, components include components such as software components, object-oriented software components, class components and task components, and processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays and variables.
[0101] The components and the functions provided within those components may be combined into a smaller number of components or further separated into additional components. In this case, it will be understood that each block of the drawings may be executed by computer program instructions. Since these computer program instructions may be loaded into the processor of a general-purpose computer, a specialized computer, or other programmable data processing equipment, the instructions executed through the processor of the computer or other programmable data processing equipment create the means to perform the functions described in the block(s). Since these computer program instructions may be used with a computer or other programmable data processing equipment capable of implementing the functions in a specific manner, or may be stored in computer-readable memory, the instructions used with the computer or stored in computer-readable memory may also produce a manufactured item containing the means of instruction to perform the functions described in the block(s). Since computer program instructions can be loaded onto a computer or other programmable data processing equipment, instructions that perform a series of operation steps on the computer or other programmable data processing equipment to create a process executed by the computer can also provide steps for executing the functions described in the block(s).
[0102] Additionally, each block may represent a module, segment, or part of code containing one or more executable instructions for executing a specified logical function(s). It should also be noted that in some alternative execution examples, the functions mentioned in the blocks may occur out of order. For instance, two blocks described in succession may actually be executed substantially simultaneously, or the blocks may be executed in reverse order according to their corresponding functions.
[0103] In this embodiment, the term "part" refers to a software or hardware component, such as an FPGA or ASIC, and the "part" performs certain roles. However, the meaning of "part" is not limited to software or hardware. The "part" may be configured to reside in an addressable storage medium or configured to operate one or more processors. Thus, as an example, the "part" includes components such as software components, object-oriented software components, class components, and task components, as well as processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables. The functions provided within the components and "parts" may be combined into a smaller number of components and "parts" or further separated into additional components and "parts." Furthermore, the components and "parts" may be implemented to operate one or more CPUs within a device or secure multimedia card.
[0104] Although the invention has been described above with reference to embodiments, those skilled in the art will understand that various modifications and changes can be made to the invention without departing from the spirit and scope of the invention as described in the following claims. Explanation of the symbols
[0105] 120 : Resource Management Module 122 : Anchor Selector 124: Anchor-level load balancer 122a: Frame splitter 122b: Anchor gain estimation section 122c: Group merging and alignment section 122d : Anchor frame selection unit 210 : Preprocessor 220: Inference Engine 230: Encoder 300: Hybrid Video Codec Section 310: Hybrid Encoder 320: Hybrid decoder
Claims
Claim 1 A neural accelerator-based anchor enhancer having an optional super-resolution video encoding acceleration function, comprising: a preprocessing processor that selects the most beneficial anchor frame from a video stream and preprocesses the DNN provided by a Deep Neural Network (DNN) and an anchor scheduler that outputs the selected anchor frame for each video stream; an inference engine that converts a decoded image into a high-resolution image through neural network inference when the image is input to a Graphics Processing Unit (GPU) or Neural Processing Unit (NPU); and an encoder that reuses the input video stream, compresses only the super-resolution anchor frame using an image codec while offloading the reconstruction of non-anchor frames to a client, and packages the encoded video and the super-resolution anchor frame into a single file, wherein the preprocessing processor is characterized by taking a mock DNN that is randomly initialized before live streaming starts, pre-optimizing it on a target accelerator, and then generating an optimized version using the optimized mock DNN if optimization of the DNN is required during live streaming. Claim 2 A neural accelerator-based anchor enhancer having an optional super-resolution video encoding acceleration function, wherein the preprocessing processor optimizes GPU context switching by executing a super-resolution DNN in a framework optimized for inference. Claim 3 delete Claim 4 A neural accelerator-based anchor enhancer having an optional super-resolution video encoding acceleration function, wherein the encoder compresses only super-resolution anchor frames while offloading non-anchor frame reconstruction to a client and packaging the encoded video and super-resolution anchor frames into a single file. Claim 5 A neural accelerator-based anchor enhancer having an optional super-resolution video encoding acceleration function, characterized in that, in claim 1, the size of each of the anchor frames is set equally to satisfy the bit rate constraint of live streaming. Claim 6 A neural accelerator-based anchor enhancer having an optional super-resolution video encoding acceleration function, characterized in that, in claim 1, a new header attached to each frame includes a frame type and a super-resolution frame of the anchor frame. Claim 7 (i) a step of selecting the most beneficial anchor frames from a video stream and preprocessing a Deep Neural Network (DNN) for each video stream; (ii) a step of converting a decoded image into a high-resolution image through neural network inference when the image is input to a Graphics Processing Unit (GPU) or Neural Processing Unit (NPU); and (iii) a step of reusing an input video stream and using an image codec to compress only the super-high-resolution anchor frames while offloading the reconstruction of non-anchor frames to a client, and packaging the encoded video and the super-resolution anchor frames into a single file, wherein step (i) comprises: (i-1) a step of taking a mock DNN that is randomly initialized before live streaming starts and pre-optimizing it on a target accelerator; and (i-2) a step of generating an optimized version using a mock DNN optimized offline if optimization of the DNN is required during live streaming. Claim 8 delete Claim 9 An anchor enhancing method according to claim 7, wherein step (iii) comprises: (iii-1) a step of reusing an input video stream and using an image codec to compress only super-high resolution anchor frames while offloading non-anchor frame reconstruction to a client; and (iii-2) a step of packaging the encoded video and super-resolution anchor frames into a single file.
