Anchor scheduler with super-resolution acceleration function and anchor scheduling method using the same

The anchor scheduler optimally allocates resources and selects anchor frames to enhance video quality in live streaming, addressing the inefficiencies of neural network-enhanced streaming and reducing costs, ensuring high-quality video delivery.

KR103043347B1Active Publication Date: 2026-11-06KOREA ADVANCED INST OF SCI & TECH
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
KR1020230095284
Authority / Receiving Office
KR · KR
Patent Type
Patents
Current Assignee / Owner
Priority Date
2022-07-25
Filing Date
2023-07-21
Publication Date
2026-11-06
Estimated Expiration
2043-07-21

AI Technical Summary

Technical Problem

Current live streaming infrastructure struggles to consistently deliver high-quality video due to limitations in neural network-enhanced streaming, which is too expensive and inefficient for commercial-scale applications, leading to significant viewer attrition and revenue loss.

Method used

An anchor scheduler with a super-resolution acceleration function and an anchor scheduling method that optimally allocates computing resources and selects anchor frames for neural inference, reducing end-to-end neural network costs and enhancing video quality.

Benefits of technology

The solution provides high-quality video streams efficiently by selecting anchor frames without neural inference, reducing computational overhead and costs, while maintaining quality comparable to frame-by-frame inference.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure R1020230095284_ABST
    Figure R1020230095284_ABST
Patent Text Reader

Abstract

An anchor scheduler having a super-resolution acceleration function and an anchor scheduling method using the same are disclosed to motivate the limitations of neural network-enhanced live streaming, thereby reducing end-to-end neural network enhancement costs and maximizing overall quality improvement in a computing cluster. The anchor scheduler having a super-resolution acceleration function includes: a decoder that decodes a video stream provided from an external source; and a resource management module that distributes the load while optimally allocating computing resources to each video stream decoded by the decoder.
Need to check novelty before this filing date? Find Prior Art

Description

Technology Field

[0001] The present invention relates to an anchor scheduler having a super-resolution acceleration function and an anchor scheduling method using the same, and more specifically, to an anchor scheduler having a super-resolution acceleration function and an anchor scheduling method using the same for reducing end-to-end neural network reinforcement costs and maximizing overall quality improvement in a computing cluster. Background Technology

[0002] The demand for live streaming has increased rapidly. Live video traffic is expected to account for 17% of internet traffic by 2022. Current live streaming infrastructure relies on two key elements. Video is transmitted to media servers using low-latency streaming protocols. In terms of distribution, clients run an Adaptive Bitrate (ABR) algorithm to select the highest quality video capable of real-time streaming.

[0003] Unfortunately, because ingest video quality depends heavily on the streamer's uplink bandwidth, traditional live streaming is insufficient to consistently deliver high-quality video (e.g., 4K / 8K). When the ingest path becomes congested, the overall downstream video quality degrades directly. However, video quality is the most critical factor affecting user engagement in live streaming. For example, more than 50% of live viewers abandon a video stream if the video quality drops for more than 90 seconds. This viewer attrition can deal a significant blow to the revenue of live streaming providers.

[0004] Recent advancements in neural reinforcement streaming demonstrate significant potential for improving ingested video quality by leveraging computations on media servers. When ingested video quality degrades, the media server recovers high-quality video by executing end-to-end neural reinforcement consisting of low-quality stream decoding, the application of ultra-high-resolution Deep Neural Networks (DNNs), and super-resolved output encoding. This provides dramatic quality improvements in downstream video.

[0005] However, neural reinforcement is too expensive to support commercial-scale live streaming. For example, Twitch supports over 100,000 concurrent live streams. Applying end-to-end neural reinforcement in this setup requires tens of thousands of GPUs (graphics processing units) and costs over $169,000 per hour on a public cloud. The cost breakdown shows that both video super-resolution and encoding are expensive. Neural super-resolution requires 100 to 1,000 times more computation than DNNs used for discrimination tasks, and video encoding is up to 3.3 times slower than super-resolution. Prior art literature

[0006] Korean Registered Patent No. 10-2313136 (October 15, 2021) The problem to be solved

[0007] Accordingly, the technical problem of the present invention is based on this point, and the objective of the present invention is to provide an anchor scheduler having a super-resolution acceleration function to maximize overall quality improvement in a computing cluster by reducing end-to-end neural network enhancement costs by addressing the limitations of neural network-enhanced live streaming, and by taking pairs of collected video streams and DNNs to output high-resolution video streams.

[0008] Another objective of the present invention is to provide an anchor scheduling method using an anchor scheduler having the above-described super-resolution acceleration function. means of solving the problem

[0009] According to one embodiment, to realize the above-described objective of the present invention, an anchor scheduler having a super-resolution acceleration function comprises: a decoder that decodes a video stream provided from the outside; and a resource management module that distributes the load while optimally allocating computing resources to each video stream decoded by the decoder.

[0010] In one embodiment, the resource management module may include a zero-inference anchor frame selector that runs on a server and selects the most favorable anchor frame from all video streams; and an anchor-level load balancer that dynamically balances the load between computing instances in anchor frame granularity after selecting the anchor frame.

[0011] In one embodiment, the zero-inference anchor frame selector may include: a frame splitting unit that divides frames into a key frame group, an alternate reference frame group, and other frame groups according to the type of each video stream; an anchor gain estimation unit that repeatedly selects the most favorable frame for the frames of the alternate reference frame group and the other frame group and estimates the anchor gain; a group merging and alignment unit that merges each of the key frame group, the alternate reference frame group, and other frame groups to merge and align them into a key frame global group, an alternate reference frame global group, and other frame global groups; and an anchor frame selection unit that repeatedly selects anchor frames from the global groups aligned from the key frame global group to the alternate reference frame global group and the other frame global group.

[0012] In one embodiment, the anchor gain estimation unit may include: a residual calculation module that calculates a residual accumulated over the entire frame; a residual amount calculation module that calculates a residual amount that is reduced for each frame for each iteration; an anchor gain setting module that selects the frame that reduces the residual the most and sets the anchor gain to the amount of the reduced residual; and an update module that updates the accumulated residual of each frame.

[0013] In one embodiment, the remaining amount calculation module is,

[0014]

[0015] The residual amount can be calculated using (where Res(F[j]) is the accumulated residual of the j-th frame and k is the index of the nearest frame where the residual is reset).

[0016] In one embodiment, the zero-inference anchor frame selector can periodically execute a zero-inference algorithm by selecting the maximum number of anchor frames that can be processed in real time in a computing cluster.

[0017] In one embodiment, the zero-inference anchor frame selector is,

[0018]

[0019] (Here, AF is an anchor frame aligned with anchor gain, TDNN is the DNN latency, Tintv is the anchor frame selection interval which is a configurable parameter, and M is the number of computing instances in the cluster) can be used.

[0020] To realize another objective of the present invention as described above, an anchor scheduling method according to one embodiment comprises: a step of decoding a video stream provided from an external source; and a step of distributing the load while optimally allocating computing resources to each of the decoded video streams.

[0021] In one embodiment, the step of distributing the load may include: (i) a step of executing on a server to select the most favorable anchor frame from all video streams; and (ii) a step of dynamically adjusting the load balance between computing instances in anchor frame granularity after selecting the anchor frame.

[0022] In one embodiment, the step of selecting the anchor frame may include: (i-1) dividing the frames into a key frame group, an alternate reference frame group, and other frame groups according to the type of each video stream; (i-2) estimating the anchor gain for the frames of the alternate reference frame group and the other frame group; (i-3) merging each of the key frame group, the alternate reference frame group, and the other frame group to merge and align them into a key frame global group, an alternate reference frame global group, and other frame global groups; and (i-4) repeatedly selecting the anchor frame from the global groups aligned from the key frame global group to the alternate reference frame global group and the other frame global group.

[0023] In one embodiment, the anchor frame is selected in the order of the key frame group, the replacement frame group, and the other frame group, characterized by having a priority order.

[0024] In one embodiment, the anchor gain can be calculated based on the accumulated residual.

[0025] In one embodiment, the anchor gain can be estimated using residuals.

[0026] In one embodiment, the step of estimating the anchor gain may include: calculating residuals accumulated over the entire frame; repeatedly selecting the most favorable frame and estimating the anchor gain; selecting the frame that reduces the residual the most and setting the anchor gain to the amount of the reduced residual; and updating the accumulated residual of each frame to reflect the influence of the selected frame. Effects of the invention

[0027] According to the anchor scheduler with such super-resolution acceleration capabilities and the anchor scheduling method utilizing it, it is possible to provide quality similar to algorithms that execute frame-by-frame inference while accelerating the encoding of selective super-resolution video by selecting anchor frames without neural inference through a zero-inference anchor frame selection algorithm. Additionally, through a resource management module, global anchor frame selection is executed across all video streams to select the optimal set of anchor frames, and the load can be distributed to instances on an anchor frame basis, allowing for accurate prediction of anchor frame overhead. Brief explanation of the drawing

[0028] Figure 1 is a conceptual diagram illustrating conventional streaming and neural network-enhanced streaming (an example of adaptive streaming). Figure 2 is a conceptual diagram for schematically explaining selective super-resolution. FIG. 3 is a conceptual diagram for schematically explaining a neuroscaler according to one embodiment of the present invention. Figure 4 is a conceptual diagram illustrating the deployment model of a neuroscaler for adaptive streaming. Figure 5 is a conceptual diagram for schematically explaining the anchor scheduler illustrated in Figure 3. FIG. 6 is a conceptual diagram for schematically explaining the anchor frame selector illustrated in FIG. 3. Specific details for implementing the invention

[0029] Hereinafter, with reference to the attached drawings, an anchor scheduler having a super-resolution acceleration function according to the present invention and an anchor scheduling method using the same will be described in more detail. Since the present invention is susceptible to various modifications and may take various forms, specific embodiments are illustrated in the drawings and described in detail in the text. However, this is not intended to limit the present invention to the specific disclosed forms, and it should be understood that it includes all modifications, equivalents, and substitutions that fall within the spirit and scope of the present invention.

[0030] In describing each drawing, similar reference numerals have been used for similar components. In the attached drawings, the dimensions of the structures are depicted enlarged compared to their actual size to ensure clarity of the invention.

[0031] Terms such as "first," "second," etc., may be used to describe various components, but said components should not be limited by said terms. These terms are used solely for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be named the second component, and similarly, the second component may be named the first component. A singular expression includes a plural expression unless the context clearly indicates otherwise.

[0032] In this application, terms such as "comprising" or "having" are intended to specify the existence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0033] Furthermore, unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as generally understood by those skilled in the art to which the present invention pertains. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and should not be interpreted in an ideal or overly formal sense unless explicitly defined in this application.

[0034] Hereinafter, the present invention will be described in detail with reference to the attached drawings.

[0035] Live streaming consists of collection and distribution aspects as shown in Fig. 1.

[0036] Figure 1 is a conceptual diagram illustrating conventional streaming and neural network-enhanced streaming (an example of adaptive streaming).

[0037] Referring to Fig. 1, in terms of collection, the streamer captures live video and uploads it to a media server. In adaptive streaming, the streamer uploads a single video stream, and the media server transcodes it into multiple quality versions.

[0038] In a multi-party video conference, the broadcaster uploads video streams of various qualities using simultaneous broadcasting or a Scalable Video Codec (SVC), and the media server delivers these video streams to viewers. Next, on the distribution side, the client executes an Adaptive Bitrate (ABR) algorithm to regulate video streaming quality, selecting and downloading the highest quality video within a given network bandwidth. The ABR algorithm can be classified into heuristic-based and machine learning-based methods. Heuristic-based methods are further classified based on network bandwidth, client buffer occupancy, and hybrid methods that combine both.

[0039] A common limitation of traditional live streaming is that video quality is restricted by the streamer's uplink bandwidth. Even if viewers have sufficient network bandwidth, this limitation may deprive them of the opportunity to enjoy high-quality video.

[0040] Super-resolution is a type of technique that generates high-resolution images from low-resolution images. Recent research uses Deep Neural Networks (DNNs) to learn mapping from low resolution to high resolution and demonstrates dramatic quality improvements.

[0041] Neural reinforcement live streaming uses a super-resolution DNN to enhance the collected video stream as shown in Fig. 1. Unlike conventional live streaming, the media server performs end-to-end neural reinforcement. When the video arrives, it is first decoded into raw frames. Then, the DNN is applied to all or some frames according to a super-resolution method. The output is re-encoded into video before being delivered to the client.

[0042] To provide stable quality improvement, content-aware DNNs are typically used for each video stream and can be updated even during live streaming. Context switching between content-aware DNNs causes two types of overhead.

[0043] First, modern DNN compilers provide optimized inference for target accelerators, but this involves upfront costs due to model optimization that can take up to several minutes. Both graph-level and operator-level optimizations are applied to generate a DNN tailored to a specific target.

[0044] Second, the DNN and frames must be loaded into the device memory of the target accelerator before executing inference. Since neural accelerators generally have limited memory sizes (e.g., 16GB for the NVIDIA T4), multiple content recognition DNNs, each requiring several GB of memory, must be frequently swapped.

[0045] Selective super-resolution (SR) reduces computing overhead by utilizing temporal redundancy across entire video frames. Neural inference is applied only to selected frames called anchor frames.

[0046] Figure 2 is a conceptual diagram for schematically explaining selective super-resolution.

[0047] As can be seen in Fig. 2, non-anchor frames are reconstructed by reusing previous super-resolution frames guided by codec-level information (e.g., reference index, motion vector, residual). This includes lightweight bilinear interpolation and decoding, so it can be processed in real time even on mobile devices.

[0048] Upscaling non-anchor frames by reusing previous super-resolution frames inevitably results in quality loss due to time lag (compared to per-frame inference). This loss accumulates in consecutive non-anchor frames but is reset in anchor frames. Therefore, to maximize quality, it is important to select a beneficial set of anchor frames. To this end, the NEMO algorithm relies on costly per-frame inference. Consequently, it cannot be used for live streaming where anchor frame selection must be performed online in real time.

[0049] FIG. 3 is a conceptual diagram for schematically explaining a neuroscaler according to one embodiment of the present invention.

[0050] Referring to FIG. 3, a NeuroScaler according to one embodiment of the present invention includes an Anchor scheduler (100) and an Anchor enhancer (200) to take a pair of collected video streams and a Deep Neural Network (DNN) and output a high-resolution video stream. The DNN may be dynamically updated through online learning, such as in LiveNAS.

[0051] The anchor scheduler (100) has a super-resolution acceleration function, decodes the collected video stream, and selects the most advantageous anchor frame from the video stream. Here, an anchor frame refers to a frame to which the super-resolution is applied using codec-level information in real time, while expanding the optional super-resolution to accelerate the video super-resolution. Then, for each video stream, the anchor scheduler (100) transmits the DNN and the selected anchor frame to an anchor enhancer (200) equipped with a neural accelerator.

[0052] The anchor enhancer (200) preprocesses the DNN, applies the preprocessed DNN to the anchor frame, and re-encodes the super-resolution output.

[0053] In this embodiment, all processes of the anchor scheduler (100) and anchor enhancer (200) are pipelined and parallelized to maximize overall throughput.

[0054] According to the neuroscaler of the present invention, by addressing the limitations of neural network augmentation live streaming, the end-to-end neural network augmentation cost can be reduced and the overall quality improvement in a computing cluster can be maximized.

[0055] Figure 4 is a conceptual diagram illustrating the deployment model of a neuroscaler for adaptive streaming.

[0056] Referring to Fig. 4, when a low-quality video stream (e.g., 360p or 720p) is uploaded to a media server, the neuroscaler generates a high-quality video stream using real-time super-resolution.

[0057] Low-resolution versions (e.g., 240p to 720p) are also generated from video streams collected using existing transcoding pipelines. By using a neuroscaler, clients can view high-resolution video even when the collection path is congested. Deploying a neuroscaler for video conferencing is similar to the process above, but multi-resolution transcoding is not required.

[0058] FIG. 5 is a conceptual diagram for schematically explaining the anchor scheduler (100) illustrated in FIG. 3.

[0059] Referring to FIGS. 3 and FIGS. 5, the anchor scheduler (100) includes a decoder (110) and a resource management module (120).

[0060] The decoder (110) decodes a video stream provided from the outside and provides it to the resource management module (120). The decoded video stream may be provided directly to the resource management module (120) or may be provided to the resource management module (120) via memory.

[0061] The resource management module (120) includes an anchor selector (122) and an anchor-level load balancer (124) to distribute the load while optimally allocating computing resources to each video stream. In this embodiment, the resource management module (120) distributes the load while optimally allocating computing resources to each video stream to maximize quality improvement when processing a large number of video streams. In particular, the resource management module (120) executes global anchor frame selection on all video streams to select an optimal set of anchor frames and distributes the load to instances in anchor frame units, which can accurately predict the overhead of the anchor frames. In this embodiment, the anchor scheduler (100) has been described as including an anchor selector (122) and an anchor-level load balancer (124), but this is a logical distinction made for convenience of explanation only and is not a hardware distinction.

[0062] The anchor selector (122) runs on a centralized server and selects the most favorable anchor frame from all video streams. In particular, the anchor selector (122) periodically runs a zero-inference algorithm by selecting the maximum number of anchor frames that can be processed in real time on a computing cluster, as shown in the following formula (1).

[0063] [Formula 1]

[0064]

[0065] Here, is an anchor frame aligned with anchor gain, and is the DNN latency, and is the anchor frame selection interval, which is a configurable parameter, and is the number of computing instances in the cluster.

[0066] Unless otherwise noted in this specification = 666ms (40 frames for 60fps video) is used. This global anchor frame selection can alleviate the imbalance of anchor frames per video stream.

[0067] In general, using longer scheduling intervals allows for the selection of more influential anchor frames, thereby improving quality. Therefore, the interval must be adjusted according to application requirements.

[0068] An anchor-level load balancer (124) selects an anchor frame and then dynamically balances the load between computing instances in anchor frame granularity. To do this, an anchor scheduler (100) divides the selected anchor frame into instance-specific groups, and the frames of each group are delivered to the corresponding computing instances for processing.

[0069] To meet real-time constraints, the anchor scheduler (100) has a total waiting time between anchor frame selection intervals ( Allocate a maximum number of anchor frames for each group that is less than ). This anchor-level load balancing can alleviate load imbalances per instance.

[0070] FIG. 6 is a conceptual diagram for schematically explaining the anchor selector (122) illustrated in FIG. 5.

[0071] Referring to FIGS. 5 and 6, the anchor selector (122) includes a frame splitting unit (122a), an anchor gain estimation unit (122b), a group merging and alignment unit (122c), and an anchor frame selection unit (122d), and selects a beneficial anchor frame by utilizing codec-level information (e.g., frame type, residual). In this embodiment, the anchor selector (122) is described as including a frame splitting unit (122a), an anchor gain estimation unit (122b), a group merging and alignment unit (122c), and an anchor frame selection unit (122d), but this is a logical distinction made for convenience of explanation and is not a hardware distinction.

[0072] The frame splitting unit (122a) divides the frames into video stream-specific groups according to the type of each video stream. That is, the frame splitting unit (122a) divides the frames into key frame groups ( ), alternate reference frame group( ) and other frame groups( Divided by ). Here, the group , and There is a priority order for selecting anchor frames.

[0073] The anchor gain estimator (122b) is an alternative reference frame group ( ) and other frame groups( For the frames of ), the benefit of using anchor frames, called anchor gain, is estimated. Anchor gain is calculated based on accumulated residuals using Algorithm 1 described below. Key frame group ( In the case of ), it is assumed that there are only a few key frames and generally all are selected as anchor frames, so it has the same anchor gain as the overall performance.

[0074] The anchor gain estimation unit (122b) includes a residual calculation module (not shown), a remaining amount calculation module (not shown), an anchor gain setting module (not shown), and an update module (not shown). In this embodiment, the anchor gain estimation unit (122b) is described as including a residual calculation module, a remaining amount calculation module, an anchor gain setting module, and an update module; however, this is a logical distinction made for convenience of explanation and is not a hardware distinction. The residual calculation module calculates the residual accumulated over the entire frame. The remaining amount calculation module calculates the remaining amount that decreases for each frame for each iteration. The anchor gain setting module selects the frame that reduces the residual the most and sets the anchor gain to the amount of the reduced residual. The update module updates the accumulated residual of each frame.

[0075] The group merging and alignment unit (122c) merges video stream-specific groups into a global group. That is, a key frame global group ( ), Alternative reference frame global group( ), and other Frame Global Group( Each contains a key frame, an alternate reference frame, and a normal frame of the entire video stream. Frames in the same group are aligned according to anchor gain.

[0076] The anchor frame selection unit (122d) is a key frame global group ( In the alternate reference frame global group ( ) and other Frame Global Groups( Iteratively selects anchor frames from global groups aligned up to ).

[0077] In this embodiment, the neuroscaler measures the latency of the DNN once to satisfy real-time constraints and selects the maximum number of anchor frames where the total latency is shorter than the available computing time.

[0078] Below, we will explain how the neuroscaler estimates the anchor gain using residuals through Algorithm 1.

[0079] Algorithm 1 shows how the neuroscaler estimates the gain of the anchor frame.

[0080]

[0081] Algorithm 1 first calculates the accumulated residuals over the entire frame (Algorithm 1, line #2).

[0082] Next, the most favorable frame is repeatedly selected and the anchor gain is estimated. That is, for each iteration, the remaining amount of each frame that is reduced is calculated as in the following formula (2) (lines #6-8).

[0083] [Equation 2]

[0084]

[0085] Here, Res(F[j]) is the accumulated residual of the j-th frame, and k is the index of the nearest frame where the residual is reset. The residual is cleared from the key frame or the frame where the anchor gain was estimated in the previous iteration. If such a frame does not exist within the given frame, the algorithm predicts that the residual is reset from the key frame of the next video chunk.

[0086] Next, select the frame that reduces the residual the most and set the anchor gain to the amount of the reduced residual (lines 12, 13).

[0087] Finally, the accumulated residuals of each frame are updated to reflect the influence of the selected frame (Line #14). To quickly estimate the anchor gain, the total residual pixel value is approximated by the size of the encoded residual frame. Since both values ​​are highly correlated, this has minimal impact on quality (△0.05 dB in PSNR).

[0088] As described above, according to the present invention, a zero-inference anchor frame selection algorithm can accelerate the encoding of selective super-resolution video by selecting anchor frames without neural inference, while providing quality similar to that of an algorithm that performs frame-by-frame inference. Additionally, through a resource management module, global anchor frame selection can be performed on all video streams to select an optimal set of anchor frames, and the load can be distributed to instances on an anchor frame basis, allowing for accurate prediction of anchor frame overhead.

[0089] Components according to embodiments of the present invention may be implemented in the form of software or hardware such as an FPGA (Field Programmable Gate Array) or ASIC (Application Specific Integrated Circuit) and may perform certain roles. However, the term 'components' is not limited to software or hardware, and each component may be configured to reside in an addressable storage medium or configured to run one or more processors.

[0090] Accordingly, as an example, components include components such as software components, object-oriented software components, class components and task components, and processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays and variables.

[0091] The components and the functions provided within those components may be combined into a smaller number of components or further separated into additional components. In this case, it will be understood that each block of the drawings may be executed by computer program instructions. Since these computer program instructions may be loaded into the processor of a general-purpose computer, a specialized computer, or other programmable data processing equipment, the instructions executed through the processor of the computer or other programmable data processing equipment create the means to perform the functions described in the block(s). Since these computer program instructions may be used with a computer or other programmable data processing equipment capable of implementing the functions in a specific manner, or may be stored in computer-readable memory, the instructions used with the computer or stored in computer-readable memory may also produce a manufactured item containing the means of instruction to perform the functions described in the block(s). Since computer program instructions can be loaded onto a computer or other programmable data processing equipment, instructions that perform a series of operation steps on the computer or other programmable data processing equipment to create a process executed by the computer can also provide steps for executing the functions described in the block(s).

[0092] Additionally, each block may represent a module, segment, or part of code containing one or more executable instructions for executing a specified logical function(s). It should also be noted that in some alternative execution examples, the functions mentioned in the blocks may occur out of order. For instance, two blocks described in succession may actually be executed substantially simultaneously, or the blocks may be executed in reverse order according to their corresponding functions.

[0093] In this embodiment, the term "part" refers to a software or hardware component, such as an FPGA or ASIC, and the "part" performs certain roles. However, the meaning of "part" is not limited to software or hardware. The "part" may be configured to reside in an addressable storage medium or configured to operate one or more processors. Thus, as an example, the "part" includes components such as software components, object-oriented software components, class components, and task components, as well as processes, functions, attributes, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables. The functions provided within the components and "parts" may be combined into a smaller number of components and "parts" or further separated into additional components and "parts." Furthermore, the components and "parts" may be implemented to operate one or more CPUs within a device or secure multimedia card.

[0094] Although the invention has been described above with reference to embodiments, those skilled in the art will understand that various modifications and changes can be made to the invention without departing from the spirit and scope of the invention as described in the following claims. Explanation of the symbols

[0095] 120 : Resource Management Module 122 : Anchor Selector 124: Anchor-level load balancer 122a: Frame splitter 122b: Anchor gain estimation section 122c: Group merging and alignment section 122d : Anchor frame selection unit 210 : Preprocessor 220: Inference Engine 230: Encoder

Claims

Claim 1 A decoder that decodes an externally provided video stream; and a resource management module that distributes the load while optimally allocating computing resources to each video stream decoded by the decoder, comprising a zero-inference anchor frame selector that runs on a server and selects the most favorable anchor frame from all video streams, and an anchor-level load balancer that dynamically adjusts the load balance between computing instances in anchor frame granularity after selecting the anchor frame, wherein the zero-inference anchor frame selector includes a frame splitting unit that divides frames into a key frame group, an alternate reference frame group, and other frame groups according to the type of each video stream; and an anchor gain estimation unit that iteratively selects the most favorable frame for the frames of the alternate reference frame group and the other frame group and estimates the anchor gain, wherein the anchor gain estimation unit includes a residual calculation module that calculates residuals accumulated over the entire frame; a residual amount calculation module that calculates the residual amount reduced by each frame for each iteration; and an anchor gain setting module that selects the frame that reduces the residual the most and sets the anchor gain to the amount of the reduced residual. An anchor scheduler having a super-resolution acceleration function characterized by including an update module that updates the accumulated residual of each frame. Claim 2 delete Claim 3 An anchor scheduler having a super-resolution acceleration function according to claim 1, wherein the zero-inference anchor frame selector further comprises: a group merging and alignment unit that merges each of the key frame group, alternate reference frame group, and other frame group to merge and align into a key frame global group, an alternate reference frame global group, and other frame global group; and an anchor frame selection unit that repeatedly selects anchor frames from the global groups aligned from the key frame global group to the alternate reference frame global group and the other frame global group. Claim 4 delete Claim 5 In paragraph 1, the above-mentioned remaining amount calculation module is, An anchor scheduler having a super-resolution acceleration function characterized by calculating the residual amount using (where Res(F[j]) is the accumulated residual of the j-th frame and k is the index of the nearest frame where the residual is reset). Claim 6 An anchor scheduler having a super-resolution acceleration function, wherein, in claim 1, the zero-inference anchor frame selector selects the maximum number of anchor frames that can be processed in real time in a computing cluster and periodically executes a zero-inference algorithm. Claim 7 In paragraph 6, the above zero-inference anchor frame selector is, (Here, AF is an anchor frame aligned with anchor gain, and T DNN An anchor scheduler with a super-resolution acceleration function characterized by using (where is the DNN latency, Tintv is the anchor frame selection interval which is a configurable parameter, and M is the number of computing instances in the cluster). Claim 8 An anchor scheduling method comprising: a step of decoding a video stream provided from an external source; and a step of distributing the load while optimally allocating computing resources to each decoded video stream, wherein the step of selecting the anchor frame includes: (i-1) dividing the frames into a key frame group, an alternate reference frame group, and other frame groups according to the type of each video stream; and (i-2) estimating the anchor gain for the frames of the alternate reference frame group and the other frame group, wherein the step of estimating the anchor gain includes: a step of calculating residuals accumulated over the entire frame; a step of repeatedly selecting the most advantageous frame and estimating the anchor gain; a step of selecting the frame that reduces the residuals the most and setting the anchor gain to the amount of the reduced residual; and a step of updating the accumulated residuals of each frame to reflect the influence of the selected frame. Claim 9 delete Claim 10 An anchor scheduling method according to claim 8, wherein the step of selecting the anchor frame further comprises: (i-3) merging each of the key frame group, alternate reference frame group, and other frame group to merge and align them into a key frame global group, an alternate reference frame global group, and other frame global group; and (i-4) repeatedly selecting an anchor frame from the global group aligned from the key frame global group to the alternate reference frame global group and the other frame global group. Claim 11 An anchor scheduling method according to claim 10, characterized in that there is a priority for selecting the anchor frame in the order of the key frame group, the alternative reference frame group, and the other frame group. Claim 12 An anchor scheduling method according to claim 10, characterized in that the anchor gain is calculated based on accumulated residuals. Claim 13 An anchor scheduling method according to claim 10, characterized in that the anchor gain is estimated using residuals. Claim 14 delete

Citation Information

Patent Citations

  • Method and Apparatus for Accelerating Super-Resolution in Real-time Video Streaming

    KR102313136B1