Method for upscaling videos based on artificial intelligence, and apparatus implementing the same method

Abstract

An AI-based video upscaling method performed by a computing device according to one embodiment of the present disclosure comprises: a step of obtaining an upscaled frame for a first frame by inputting the first frame into a first AI model if the type of the first frame of a video transmitted from a server in a streaming manner is a compressed intra-frame; a step of determining whether to input the second frame into the first AI model based on the bit amount of the second frame if the type of the second frame of the transmitted video is a compressed inter-frame; and a step of obtaining an upscaled frame for the second frame by decoding the second frame and inputting it into a second AI model different from the first AI model if it is determined not to input the second frame into the first AI model.

Description

Technology Field

[0001] The present disclosure relates to an AI-based video upscaling method and an apparatus for implementing the same, and more specifically, to an AI-based video upscaling method for providing a video upscaled based on an AI model when providing a real-time video streaming service, and an apparatus for implementing the same. Background Technology

[0002] With the recent advancement of science and technology, anyone can easily use camera devices embedded in portable devices such as smartphones, tablets, and laptops, and the digitization, storage, sharing, and transmission of multimedia content generated by cameras are taking place rapidly and globally.

[0003] In particular, with the rise of OTT (Over-The-Top) media services, consumers of multimedia content have come to prefer higher-resolution video to watch vivid and realistic images.

[0004] The standard video resolution on streaming services is FullHD (1920x1080) or 4K (3840x2160), and the rate of increase in video resolution has been very rapid, reaching 8K (7680x4320) recently, and it is expected that the current maximum resolution will soon be replaced by a higher resolution format.

[0005] However, since high-resolution video not only requires a large amount of storage space but also takes a long time to transmit to the client, low-resolution video may be transmitted depending on network conditions.

[0006] In addition, while existing low-resolution video content must also be provided to OTT service consumers, there is a problem in that the image quality of such low-resolution videos is unsatisfactory if they are not played on UHD (Ultra-High-Definition) display devices.

[0007] Accordingly, interest is growing in Video Super-Resolution (VSR) technology, which sophisticatedly converts low-resolution video into high-resolution video clips so that consumers can watch higher-quality video.

[0008] VSR technology reconstructs low-resolution video into high-resolution video using various frame features; it originated from SISR (Single Image Super-Resolution) or MISR (Multiple-Image Super Resolution) and utilizes spatial or temporal elements of the image.

[0009] Recently, it has been confirmed that using deep learning-based VSR algorithms can produce clearer, high-resolution output. Recent research on VSR models has shown significant progress in terms of high-resolution video quality and conversion speed, and among the major VSR models presented are (1) Recurrent Frame Based VSR Network (RBPN, FRVSR, RRN), (2) Spatio-Temporal VSR Network (STVSR, TDAN, SOF-VSR, TOFlow, TDVSR-L), (3) GAN (Generative Adversarial Networks) based VSR Network, (4) Video Compression informed VSR Network (FAST, COMISR), and (5) Transformer-Based VSR (VRT, TTVSR, FTVSR, MIA-VSR).

[0010] Existing VSR models as described above have strengths and weaknesses in various aspects. For example, (1) the Recurrent Frame Based VSR Network contributes to integrating SISR and MISR into a unified VSR framework and helps to understand large gaps between frames by independently calculating each context, but it requires more parameters and computations (FLOPs: Floating point operations per second) for training compared to other existing VSR networks, and implicitly has a problem of gradient vanishing during the training process.

[0011] In addition, (2) the Spatio-Temporal VSR Network is a well-integrated network and shows good performance in terms of PSNR (Peak Signal-to-Noise Ratio) and Structural Similarity Index (SSIM), but this model has the disadvantage of not being able to properly generate high resolution when the movement between frames is very fast and not taking into account when a new object enters a consecutive frame.

[0012] Meanwhile, (3) GAN-based VSR Networks have made significant improvements in generating and constraining high-frequency details of reconstructed frames, but they have the disadvantage that the parameters of GAN-based VSR models are difficult to use in real-time environments compared to other VSR models because they include generator and discriminator modules that require much more time and resources for training and evaluation.

[0013] Finally, the (4) Video Compression informed VSR Network, which uses compressed video information, reduces the complexity of the VSR model architecture and reduces the training burden and inference tasks in real-time environments of existing VSR models, but it has the disadvantage that it requires periodic frame resets to obtain appropriate results when there is a sudden change in the video, and is difficult to apply in real-time due to the complexity of the model, just like the (5) Transformer-Based VSR model.

[0014] Therefore, there is a need for technology that can solve the problem of existing VSR models being difficult to apply to real-time streaming services due to model complexity and high computational load.

[0015] Furthermore, instead of complex VSR models applied to videos with high motion in service environments such as OTT and VCS (Video Communication System), there is a need for a method to provide services by utilizing various information contained in compressed video. Prior art literature

[0016] Republic of Korea Registered Patent Publication No. 10-2770525 (Registered on Feb. 17, 2025) China Published Patent Publication No. 117041603 (Published on Nov. 10, 2023) The problem to be solved

[0017] The technical problem that the present disclosure aims to solve is to provide an AI-based video upscaling method capable of providing an AI-based upscaling function for video frames in a real-time streaming environment using various information contained in a compressed video, and an apparatus for implementing the same.

[0018] Another technical problem that the present disclosure aims to solve is to provide an AI-based video upscaling method capable of providing a high-resolution frame generation function even when there are many changes in motion between frames of a video, and an apparatus for implementing the same.

[0019] Another technical problem that the present disclosure aims to solve is to provide an AI-based video upscaling method capable of generating high-resolution frames even when a new object enters a continuous frame of a video or when a sudden scene change occurs, and to provide an apparatus for implementing the same.

[0020] Another technical problem that the present disclosure aims to solve is to provide an AI-based video upscaling method capable of reducing resources during video frame upscaling by applying a VSR model in which the number of parameters and computational load during the learning process are minimized compared to existing VSR models, and to provide an apparatus for implementing the same.

[0021] The technical problems of the present disclosure are not limited to those mentioned above, and other unmentioned technical problems will be clearly understood by a person skilled in the art from the description below. means of solving the problem

[0022] An AI-based video upscaling method performed by a computing device according to an embodiment of the present disclosure to solve the above technical problem comprises: a step of obtaining an upscaled frame for a first frame by inputting the first frame into a first AI model if the type of a first frame of a video transmitted from a server in a streaming manner is a compressed intra-frame; a step of determining whether to input the second frame into the first AI model based on the bit amount of the second frame if the type of a second frame of the transmitted video is a compressed inter-frame; and a step of obtaining an upscaled frame for the second frame by decoding the second frame and inputting it into a second AI model different from the first AI model if it is determined not to input the second frame into the first AI model.

[0023] In one embodiment, the step of obtaining an upscaled frame for the first frame by inputting the first frame into a first AI model may include the step of inputting the first frame into the first AI model composed of an encoder and a decoder, and the step of obtaining an upscaled frame for the first frame by performing pixel shuffling on the result output from the decoder of the first AI model.

[0024] In one embodiment, the first frame and the second frame are frames compressed to a resolution lower than a preset reference value, and the upscaled frame for the first frame and the upscaled frame for the second frame may be frames decompressed to a resolution higher than a preset reference value.

[0025] In one embodiment, the step of determining whether to input the second frame into the first AI model based on the bit amount of the second frame may include identifying whether the bit amount of the second frame exceeds a preset threshold, and if it is identified that the bit amount exceeds the preset threshold, determining to input the second frame into the first AI model.

[0026] In one embodiment, the step of determining whether to input the second frame into the first AI model based on the bit amount of the second frame may include the step of inputting the second frame into an AI-based discrimination model, and the step of determining whether to input the second frame into the first AI model from the result output from the AI ​​discrimination model.

[0027] In one embodiment, the step of obtaining an upscaled frame for the second frame by decoding the second frame and then inputting it into a second AI model different from the first AI model may include the step of inputting the second frame into a decoder, and the step of obtaining an upscaled frame for the second frame by inputting the result output from the decoder into a lightweight VSR (Video Super-Resolution) model based on spatio-temporal information.

[0028] In one embodiment, the lightweight VSR (Video Super-Resolution) model includes a plurality of convolution layers and a SFT (Spatial Feature Transform) layer, and the step of obtaining an upscaled frame for the second frame may include the step of sequentially inputting the result output from the decoder to the plurality of convolution layers, the step of inputting the result output through the plurality of convolution layers to the SFT layer, and the step of obtaining an upscaled frame for the second frame from the result output from the SFT layer.

[0029] In one embodiment, the step of inputting the result output through the plurality of convolution layers into the SFT (Spatial Feature Transform) layer may include the step of calculating the output result of the SFT layer using an intermediate result obtained from the result of performing a convolution operation with the feature map obtained through the plurality of convolution layers and a patch condition using motion vectors as inputs.

[0030] In one embodiment, the first AI model applied to the first frame may be a model trained using training data consisting of a bitstream of a plurality of intra-frames encoded at a resolution lower than a preset threshold and a plurality of raw images of intra-frames at a resolution higher than the preset threshold.

[0031] In one embodiment, the first AI model applied to the second frame may be a model trained using training data consisting of a bitstream of a plurality of inter-frames encoded at a resolution lower than a preset threshold and a raw image of a plurality of inter-frames at a resolution higher than the preset threshold.

[0032] In one embodiment, the second AI model applied to the second frame may be a model trained using training data consisting of a plurality of raw images of inter-frames with a resolution lower than a preset threshold and a plurality of raw images of inter-frames with a resolution higher than the preset threshold.

[0033] In one embodiment, the first AI model applied to the first frame and the second frame may use an MSE loss function during the training process, and the second AI model applied to the second frame may use a Huber loss function during the training process.

[0034] An AI-based video upscaling method performed by a computing device according to an embodiment of the present disclosure to solve the above technical problem comprises: a step of obtaining an upscaled frame for a first frame by inputting the first frame into an AI model composed of an encoder and a decoder if the type of the first frame of a video transmitted from a server in a streaming manner is a compressed intra-frame; a step of identifying whether the bit amount of the second frame is greater than or equal to a preset threshold if the type of the second frame of the transmitted video is a compressed inter-frame; and, if the bit amount of the second frame is greater than or equal to the preset threshold, obtaining an upscaled frame for the second frame by inputting the second frame into the AI ​​model, and if the bit amount of the second frame is less than the preset threshold, decoding the second frame and inputting it into a lightweight VSR model based on spatio-temporal information, thereby obtaining an upscaled frame for the second frame It includes the step of acquiring an upscaled frame.

[0035] In one embodiment, the spatiotemporal information-based lightweight VSR model includes a Spatial Feature Transform (SFT) module, wherein the SFT module generates different scale and shift parameters for each position using a patch condition utilizing motion vectors obtained through a plurality of convolution layers, and performs a position-adjusted transformation on a feature map obtained through the plurality of convolution layers using the generated scale and shift parameters.

[0036] An AI-based video upscaling method performed by a computing device according to an embodiment of the present disclosure to solve the above technical problem comprises: a step of identifying the type of each of a plurality of frames included in a group of pictures (GOP) of a video transmitted from a server in a streaming manner and the bit amount of each frame; and a step of performing upscaling using a different AI model for each frame group classified by the identified type and bit amount of each frame.

[0037] In one embodiment, the Group of Pictures (GOP) comprises one compressed intra-frame and two or more compressed inter-frames, and the step of performing upscaling using different AI learning models for each frame group classified by the type and bit amount of each identified frame may include: classifying the frame into a first frame group when the type of the frame is the compressed intra-frame; classifying the frame into a second frame group when the type of the frame is the compressed inter-frame and the bit amount of the frame exceeds a preset threshold; and classifying the frame into a third frame group when the type of the frame is the compressed inter-frame and the bit amount of the frame is less than or equal to a preset threshold.

[0038] In one embodiment, the step of performing upscaling using different AI learning models for each frame group classified by the type and bit amount of each identified frame may include the step of obtaining an upscaled frame for each frame by inputting each frame belonging to the first frame group and each frame belonging to the second frame group into an AI model composed of an encoder and a decoder, and the step of obtaining an upscaled frame for each frame by decoding each frame belonging to the third frame group and inputting it into a lightweight VSR model based on spatio-temporal information.

[0039] A video upscaling device according to one embodiment of the present disclosure for solving the above technical problem comprises instructions for performing operations to obtain an upscaled frame for a first frame by inputting the first frame into a first AI model when the type of the first frame of a video transmitted from a server in a streaming manner is a compressed intra-frame; an operation to determine whether to input the second frame into the first AI model based on the bit amount of the second frame when the type of the second frame of the transmitted video is a compressed inter-frame; and an operation to obtain an upscaled frame for the second frame by decoding the second frame and inputting it into a second AI model different from the first AI model when it is determined not to input the second frame into the first AI model.

[0040] In one embodiment, the first AI model is an AI model composed of an encoder and a decoder, and the second AI model may be a lightweight VSR (Video Super-Resolution) model based on spatio-temporal information.

[0041] A video upscaling device according to one embodiment of the present disclosure for solving the above technical problem comprises one or more processors, a memory for loading a computer program executed by said processors, and a storage for storing said computer program, wherein the computer program includes instructions for performing an operation to identify the type of each of a plurality of frames included in a group of pictures (GOP) of a video transmitted from a server in a streaming manner, the bit amount of each frame, and an operation to perform upscaling using a different AI model for each frame group classified by the type and bit amount of each identified frame. Brief explanation of the drawing

[0042] FIG. 1 illustrates the configuration of an overall system including a video upscaling device according to an embodiment of the present disclosure. FIG. 2 is a block diagram illustrating the detailed configuration of a video upscaling device according to an embodiment of the present disclosure. FIG. 3 is a flowchart illustrating an AI-based video upscaling method according to one embodiment of the present disclosure. FIG. 4 is a flowchart illustrating an AI-based video upscaling method according to another embodiment of the present disclosure. FIG. 5 is a flowchart illustrating an AI-based video upscaling method according to another embodiment of the present disclosure. FIG. 6 is an example showing the types of frames included in the GOP of a video according to some embodiments of the present disclosure. FIG. 7 is an example showing the entire process of generating an AI-based upscaled frame based on the type and bit amount of a frame according to some embodiments of the present disclosure. FIG. 8 is an example illustrating the overall structure of a spatiotemporal information-based lightweight VSR model used for upscaling inter-frames according to some embodiments of the present disclosure. FIG. 9 is an example of a formula expressing upscaling for intra-frame, inter-frame, and GOP according to some embodiments of the present disclosure. FIG. 10 is an example of training input data for an AI model of an encoder-decoder structure applied to an intra-frame according to some embodiments of the present disclosure. FIG. 11 is an example of training input data for an AI model of an encoder-decoder structure applied to an inter-frame according to some embodiments of the present disclosure. FIG. 12 is an example of training input data for a spatiotemporal information-based lightweight VSR model applied to an inter-frame according to some embodiments of the present disclosure. FIG. 13 is an example of a formula expressing a loss function used in the learning process of an AI model according to some embodiments of the present disclosure. FIG. 14 is a hardware configuration diagram of an exemplary computing device capable of implementing methods according to one embodiment of the present disclosure. Specific details for implementing the invention

[0043] Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the attached drawings. The advantages and features of the present disclosure and the methods for achieving them will become clear by referring to the embodiments described below in detail together with the attached drawings. However, the technical concept of the present disclosure is not limited to the following embodiments but can be implemented in various different forms. The following embodiments are provided merely to complete the technical concept of the present disclosure and to fully inform those skilled in the art of the scope of the present disclosure, and the technical concept of the present disclosure is defined only by the scope of the claims.

[0044] It should be noted that when assigning reference numerals to the components of each drawing, the same components are given the same reference numeral whenever possible, even if they are shown in different drawings. Furthermore, in describing the present disclosure, if it is determined that a detailed description of related known components or functions could obscure the essence of the present disclosure, such detailed description is omitted.

[0045] Unless otherwise defined, all terms used herein (including technical and scientific terms) may be used in a meaning commonly understood by those skilled in the art to which this disclosure pertains. Additionally, terms defined in commonly used dictionaries are not to be interpreted ideally or excessively unless explicitly and specifically defined otherwise. The terms used herein are for describing the embodiments and are not intended to limit this disclosure. In this specification, the singular form includes the plural form unless specifically stated otherwise in the text.

[0046] Additionally, terms such as first, second, A, B, (a), (b), etc., may be used to describe the components of the present disclosure. These terms are intended only to distinguish the components from other components and do not limit the nature, order, or sequence of the components. Where it is stated that a component is "connected," "coupled," or "joined" to another component, it should be understood that the component may be directly connected or joined to the other component, but that another component may also be "connected," "coupled," or "joined" between each component.

[0047] As used in the specification, "comprises" and / or "comprising" do not exclude the presence or addition of one or more other components, steps, actions, and / or elements to the mentioned components, steps, actions, and / or elements.

[0048] Hereinafter, several embodiments of the present disclosure will be described in detail with reference to the attached drawings.

[0049] FIG. 1 illustrates the configuration of an overall system including a video upscaling device according to an embodiment of the present disclosure.

[0050] Referring to FIG. 1, the entire system according to an embodiment of the present disclosure is composed of a video upscaling device (1) and a server (2), and the video upscaling device (1) is connected to the server (2) via a network. Here, the server (2) may be, for example, an application server, a cloud server, or a virtual server. In addition, the video upscaling device (1) may be connected to other system configurations not shown in FIG. 1 in performing upscaling of a video.

[0051] The video upscaling device (1) is a client terminal that renders and displays a video transmitted from a server (2), such as a smartphone, tablet, laptop, PC monitor, TV, etc. Additionally, the video upscaling device (1) may be a device that performs only upscaling on frames of a video, and in this case, the upscaled frames may be transmitted to a display device connected to the video upscaling device (1).

[0052] The video upscaling device (1) automatically performs the process of upscaling video frames transmitted from the server (2) in a streaming manner based on an AI model (10).

[0053] The AI ​​model (10) may be a model trained using training data stored in a training data DB (30). The AI ​​model (10) can generate an upscaled frame for each frame in response to the input of one intra-frame (see reference numeral 61 in FIG. 6) and two or more inter-frames (see reference numeral 62 in FIG. 6) included in a Group of Pictures (GOP) (see reference numeral 60 in FIG. 6) that groups the frames of a video into one. At this time, the upscaled frame generated by the AI ​​model (10) may have a resolution higher than a preset reference value.

[0054] When multiple frames of a video are transmitted from a server (2) in a streaming manner, the video upscaling device (1) identifies what type each of the multiple frames is. For example, the video upscaling device (1) can identify whether the type of each frame is an intra-frame (61) located at the very first position in the structure of the Group of Pictures (GOP) (60) shown in FIG. 6, or an inter-frame (62) located at the second position or later.

[0055] A video upscaling device (1) can perform upscaling to increase the resolution of each frame by applying different AI models (10) depending on whether the type of each identified frame is an intra-frame or an inter-frame. At this time, the intra-frame or inter-frame input to the different AI models (10) may be in the form of a compressed frame that has not been decoded.

[0056] FIG. 2 is a block diagram illustrating the detailed configuration of a video upscaling device according to an embodiment of the present disclosure.

[0057] Referring to FIG. 2, the video upscaling device (1) includes a plurality of modules that interface with each other so that a series of processes for generating an upscaled frame for each frame of a video streamed from a server (2) are automatically performed.

[0058] Specifically, the video upscaling device (1) includes a video receiving module (11), an intra-frame processing module (12), an inter-frame processing module (13), and an upscaling result providing module (14), and may further include a storage unit (not shown) for storing data generated during the execution process of each module.

[0059] The video receiving module (11) receives a compressed video from the server (2) in a streaming manner. The compressed video may be a GOP unit that combines one intra-frame and multiple inter-frames.

[0060] The video receiving module (11) identifies whether the type of a frame of a compressed video periodically transmitted from the server (2) is an intra-frame or an inter-frame. At this time, if the identified frame type is an intra-frame, the frame is transmitted to the intra-frame processing module (12), and if the identified frame type is an inter-frame, the frame is transmitted to the inter-frame processing module (13).

[0061] The intra-frame processing module (12) inputs the compressed intra-frame transmitted from the video receiving module (11) into the first AI model (1010) of the encoder-decoder structure to obtain an upscaled frame for the compressed intra-frame.

[0062] The inter-frame processing module (13) may optionally apply a first AI model (1010) or a second AI model (1020) for upscaling a compressed inter-frame transmitted from a video receiving module (11). At this time, the criteria for selecting the first AI model (1010) or the second AI model (1020) may be, for example, information regarding the bit amount of the compressed inter-frame.

[0063] In one embodiment, the inter-frame processing module (13) can obtain an upscaled frame for the compressed inter-frame by inputting it into a second AI model (1020) using a Spatial Feature Transform (SFT) module when the bit amount of the compressed inter-frame transmitted from the video receiving module (11) is less than or equal to a preset threshold.

[0064] At this time, the second AI model (1020) can use multiple inter-frames as training data in the learning process to utilize the temporal relationship between inter-frames.

[0065] In addition, the second AI model (1020) can generate motion-compensated frames based on reference frames to perform a motion compensation process during the learning process, and can perform learning and inference of the model using the generated frames and reference frames.

[0066] In particular, the second AI model (1020) can perform effective upscaling for videos with a lot of motion by applying a Spatial Feature Transform (SFT) module that utilizes video codec information to the frame to which motion-compensated frames are to be applied.

[0067] The upscaling result providing module (14) can receive the upscaling result generated by the intra-frame processing module (12) or the inter-frame processing module (13), render it to display it on a screen, or transmit it to a connected display device.

[0068] In one embodiment, the video upscaling device (1) can perform upscaling on a video using only one of the intra-frame processing module (12) and the inter-frame processing module (13).

[0069] For example, if the network condition is poor, the video upscaling device (1) may receive only compressed intra-frames from the server (2). In this case, the video upscaling device (1) can ensure at least minimum quality in video quality by generating upscaled frames for the compressed intra-frames using only the intra-frame processing module (12).

[0070] As another example, when receiving video from a CCTV installed in a location with little movement from a server (2), the video upscaling device (1) may receive only compressed inter-frames from the server (2). In this case, the video upscaling device (1) may generate upscaled frames for the compressed inter-frames by utilizing only the inter-frame processing module (13).

[0071] In this way, according to the above embodiment, the video upscaling device (1) can generate an upscaled frame for a compressed frame by selectively using at least one of an intra-frame processing module (12) and an inter-frame processing module (13). That is, the combination of the two modules (12, 13) that utilize various information contained in the compressed video can be used as a simple yet adaptable model for upscaling the video.

[0072] As described above, according to the configuration of the video upscaling device (1) of the present disclosure, an AI-based upscaling function for video frames can be provided in a real-time streaming environment of compressed video. In addition, high-quality upscaling results can be provided by applying an optimal AI model according to the type of video frame.

[0073] FIG. 3 is a flowchart illustrating an AI-based video upscaling method according to one embodiment of the present disclosure.

[0074] Referring to FIG. 3, the AI-based video upscaling method according to an embodiment of the present disclosure may be a video upscaling device (1) shown in FIG. 1 and FIG. 2 or a computing device (100) shown in FIG. 14. The computing device (100) may be, for example, a smartphone, tablet, laptop, PC monitor, TV, etc., or a decoder device that performs only upscaling.

[0075] It should be noted that the subject performing some of the operations included in the method according to the embodiment of the present disclosure may be omitted, and in such case, the subject is the computing device (100).

[0076] According to the embodiments of the present disclosure described below, an AI-based upscaling function for video frames can be provided in a real-time streaming environment by utilizing various information contained in a compressed video. In addition, a high-resolution frame generation function can be provided even in cases where there are many changes in movement between video frames or sudden scene changes occur.

[0077] Referring to FIG. 3, first in operation S10, the computing device (100) identifies the type of frame of a video transmitted via streaming from the server (2).

[0078] At this time, if the type of the identified video frame is an intra-frame, in operation S30, the computing device (100) inputs the frame into the first AI model (1010) to obtain an upscaled frame. At this time, the first AI model (1010) may be an AI model with an encoder-decoder structure. At this time, the first AI model (1010) may additionally include a pixel upscaling module, such as pixel shuffling, after the decoder to generate an upscaled frame from the result output from the decoder.

[0079] If the type of the identified video frame is an inter-frame, in operation S20, the computing device (100) determines whether to input the frame into the first AI model (1010) based on the bit amount of the frame.

[0080] When the computing device (100) determines to input a frame into the first AI model (1010) based on the bit amount of the frame, in operation S30, the frame is input into the first AI model (1010) to obtain an upscaled frame. For example, the computing device (100) may determine to input the frame into the first AI model (1010) if the bit amount of the frame exceeds a preset threshold.

[0081] If it is decided not to input the frame into the first AI model (1010), in operation S40, the computing device (100) decodes the frame and then inputs it into the second AI model (1020) to obtain an upscaled frame. For example, the computing device (100) may decide to input the frame into the second AI model (1020) rather than the first AI model (1010) if the bit amount of the frame is less than or equal to a preset threshold.

[0082] At this time, the second AI model (1020) may be a lightweight VSR (Video Super-Resolution) model based on spatio-temporal information. At this time, the lightweight VSR model may include a SFT (Spatial Feature Transform) module that performs spatial feature transformation on frames for upscaling of videos with a lot of motion.

[0083] Meanwhile, when performing operation S20, if the type of video frame is inter-frame, the computing device (100) may select a model based on deep learning instead of selecting a first AI model (1010) or a second AI model (1020) based on a rule based on bit amount.

[0084] For example, when the type of the frame is an inter-frame, the computing device (100) inputs the frame into an AI-based discrimination model and can determine whether to input the frame into a first AI model (1010) or a second AI model (1020) based on the result output from the AI ​​discrimination model. Accordingly, when the type of the frame is an inter-frame, the optimal model for upscaling that reflects various information other than the bit amount of the frame can be automatically applied using the trained AI discrimination model.

[0085] As described above, according to the method of the embodiment of the present disclosure, an AI-based upscaling function for video frames can be provided in a real-time streaming environment by utilizing information such as the type and bit rate of frames transmitted via a streaming method. In addition, resources can be saved during video frame upscaling by applying a VSR model in which the number of parameters and computational load during the training process are minimized compared to existing VSR models.

[0086] FIG. 4 is a flowchart illustrating an AI-based video upscaling method according to another embodiment of the present disclosure.

[0087] Referring to FIG. 4, the AI-based video upscaling method according to an embodiment of the present disclosure may be a video upscaling device (1) shown in FIG. 1 and FIG. 2 or a computing device (100) shown in FIG. 14. The computing device (100) may be, for example, a smartphone, tablet, laptop, PC monitor, TV, etc., or a decoder device that performs only upscaling.

[0088] It should be noted that the subject performing some of the operations included in the method according to the embodiment of the present disclosure may be omitted, and in such case, the subject is the computing device (100).

[0089] Referring to FIG. 4, first, in operation S100, the computing device (100) identifies the type of frame of a video transmitted from the server (2) in a streaming manner.

[0090] At this time, if the type of the identified frame is a compressed intra-frame, in operation S300, the computing device (100) inputs the frame into an AI model of an encoder-decoder structure to obtain an upscaled frame. At this time, the frame input into the AI ​​model is an intra-frame compressed to a resolution lower than a preset threshold, and the upscaled frame may be a frame decompressed to a resolution higher than a preset threshold.

[0091] If the type of the identified frame is a compressed inter-frame, in operation S200, the computing device (100) determines whether the bit amount of the frame is greater than or equal to a preset threshold.

[0092] At this time, if the bit amount of the frame is greater than or equal to a preset threshold, in operation 300, the computing device (100) inputs the frame into an AI model of an encoder-decoder structure to obtain an upscaled frame.

[0093] However, if the bit amount of the frame is smaller than a preset threshold, in operation S400, the computing device (100) inputs the frame into a lightweight VSR model based on spatiotemporal information to obtain an upscaled frame.

[0094] Regarding the video upscaling method of Fig. 4 described above, we will explain it using the examples of Figs. 7 and 8.

[0095] Referring to FIG. 7, a computing device (100) can receive video from a server (2) in a streaming manner. At this time, each frame of the video received from the server (2) may be an intra-frame (701) compressed to a low resolution or an inter-frame (702) compressed to a low resolution. At this time, low resolution means a resolution lower than a preset reference value.

[0096] In one embodiment, when the type of video frame received from the server (2) is an intra-frame (701) compressed to a low resolution, the computing device (100) can obtain an upscaled frame (76) that has been decompressed for the frame by inputting the frame into an AI model (71) having the structure of an encoder (711) and a decoder (712) (Flow 1 (771)). At this time, the frame input to the AI ​​model (71) may be the compressed bitstream itself that has not been decoded.

[0097] Meanwhile, pixel upscaling (72) may be additionally performed to generate a finally decompressed upscaled frame (76) from the result output from the decoder (712) of the AI ​​model (71). For example, pixel upscaling (72) may be pixel shuffling that converts to high resolution by rearranging the features output from the decoder (712) in an alternating manner.

[0098] In one embodiment, when the type of video frame received from the server (2) is an inter-frame (702) compressed to a low resolution, the computing device (100) determines whether the bit amount of the frame exceeds a preset reference bit amount through an Adaptive Model Selector (73), and selectively applies an AI model according to the determined result.

[0099] At this time, the adaptive model selector (73) can set the average bit amount per frame of inter-frames excluding intra-frames within the GOP (Group Of Pictures) as the reference bit amount. In addition, the reference bit amount used by the adaptive model selector (73) for model selection may be set using various information instead of the average bit amount, such as the highest bit amount, lowest bit amount, and median bit amount per frame of inter-frames.

[0100] For example, when the type of frame is an inter-frame (702) compressed to low resolution and the bit amount of the frame determined by the adaptive model selector (73) exceeds a preset reference bit amount, the computing device (100) can obtain an upscaled frame (76) that has been decompressed for the frame by inputting the frame into an AI model (71) having the structure of an encoder (711) and a decoder (712) (Flow 2 (772)).

[0101] As such, since inter-frames occupying a relatively large bitstream contain many intra-mode macroblocks internally, they can be processed in a manner similar to that applied to intra-frames.

[0102] As another example, when the type of frame is an inter-frame (702) compressed to low resolution and the bit amount of the frame determined by the adaptive model selector (73) is less than or equal to a preset reference bit amount, the computing device (100) can obtain an upscaled frame (76) with decompression for the frame by decoding the frame through a decoder (74) and inputting it into a lightweight VSR model (75) based on spatio-temporal information (Flow 3 (773)).

[0103] At this time, the lightweight VSR model (75) based on spatio-temporal information may have the structure shown in FIG. 8.

[0104] Figure 8 illustrates an example of the overall structure of a spatiotemporal information-based lightweight VSR model used for upscaling compressed inter-frames.

[0105] Referring to FIG. 8, when the type of frame transmitted from the server (2) is an inter-frame compressed to low resolution and the bit amount of the frame is less than or equal to a preset reference bit amount, the computing device (100) decodes the frame (81) and inputs it into a lightweight VSR model (75 in FIG. 7) based on spatiotemporal information.

[0106] At this time, the spatiotemporal information-based lightweight VSR model (75) includes a plurality of convolution layers (82), an SFT (Spatial Feature Transform) layer (83), an additional convolution layer (84), and a Laplacian filter (85), and an upscaled frame (86) can be generated from the result of the corresponding frame (81) input to the model (75) passing through each layer (82, 83, 84, 85) sequentially. Here, the SFT layer (83) can be implemented as a separate SFT module within the computing device (100).

[0107] Specifically, the computing device (100) decodes the frame (81) through a decoder (74 in FIG. 7) and then sequentially inputs it into a plurality of convolution layers (82) of a spatiotemporal information-based lightweight VSR model (75). At this time, the plurality of convolution layers (82) may receive an inter-frame that has been motion-compensated and an inter-frame that is currently to be upscaled.

[0108] From this, the output from the multiple convolution layers (82) is input to the SFT layer (83), and the output from the SFT layer (83) passes through an additional convolution layer (84) and a Laplacian filter (85) to finally generate an upscaled frame (86) for the corresponding frame (81). The depth and connectivity of the layers included in the spatiotemporal information-based lightweight VSR model (75) are easy to add and expand, and the upscaling performance of the model can be improved by adding more layers. In addition, this sequential layer structure has the advantage of being easy to expand for GPU-based H / W structures.

[0109] Meanwhile, the SFT layer (83) can perform a convolution operation using a feature map (831) obtained from the output result of a plurality of convolution layers (82) and a patch condition (832) using motion vectors as inputs, and can produce an output result (835) of the SFT layer (83) using intermediate results (833, 834) obtained from the result.

[0110] At this time, the intermediate results (833, 834) include different scale (γ) (8330) and shift (β) (8340) parameters depending on the location generated using the patch condition (832).

[0111] From this, a positionally adjusted transformation can be performed on the feature map (831) input to the SFT layer (83) using the scale (γ) (8330) and shift (β) (8340) parameters included in the intermediate results (833, 834). This transformation is called an affine transformation, and in the affine transformation, all information obtained using both the patch condition (832) and the feature map (831) is utilized, and in the training process of a CNN-based model utilizing this information, a pair of affine transformation parameters such as the scale (γ) (8330) and shift (β) (8340) parameters can be derived.

[0112] That is, the affine transformation performed in the SFT layer (83) can be expressed as [Equation 1] below. Here, F represents a feature map of dimensions corresponding to the dimensions of γ and β, and the symbol ⊙ means element-wise multiplication.

[0114] [Mathematical Equation 1] SFT (F|γ,β) = γ⊙ F + β

[0116] In the same way as above, the SFT layer (83) can preserve spatial dimensions and simultaneously perform feature-specific manipulation and spatial transformation.

[0117] In this way, for upscaling inter-frames with a low bit rate, a spatiotemporal information-based lightweight VSR model (75) with an SFT layer (83) can be applied. This is because inter-frames with a low bit rate are mostly implemented based on intra-mode macroblocks, and these intra-mode macroblocks are determined based on pixels with a lot of movement during the video compression process.

[0118] That is, by using a lightweight VSR model (75) based on spatiotemporal information with an SFT layer (83), upscaled frames can be generated even when there is a lot of change in movement between frames.

[0119] According to the embodiments of the present disclosure as described above, a high-resolution frame generation function can be provided even when there are many changes in movement between frames of a video. In addition, a high-resolution frame generation function can be provided even when a new object enters a continuous frame of a video or when a sudden scene change occurs.

[0120] In addition, by applying a lightweight VSR model with minimized parameters and computational load during the training process compared to existing VSR models, resources can be saved during video frame upscaling.

[0121] FIG. 5 is a flowchart illustrating an AI-based video upscaling method according to another embodiment of the present disclosure.

[0122] Referring to FIG. 5, the AI-based video upscaling method according to an embodiment of the present disclosure may be a video upscaling device (1) shown in FIG. 1 and FIG. 2 or a computing device (100) shown in FIG. 14. The computing device (100) may be, for example, a smartphone, tablet, laptop, PC monitor, TV, etc., or a decoder device that performs only upscaling.

[0123] It should be noted that the subject performing some of the operations included in the method according to the embodiment of the present disclosure may be omitted, and in such case, the subject is the computing device (100).

[0124] Referring to FIG. 5, first, in operation S1000, the computing device (100) identifies the type of each of the plurality of frames included in the group of pictures (GOP) of the video transmitted from the server (2) in a streaming manner, and the bit amount of each frame. Here, the GOP may include one compressed intra-frame and two or more compressed inter-frames.

[0125] Next, in operation S2000, the computing device (100) performs upscaling using different AI models for each frame group classified by the type and bit amount of each identified frame.

[0126] At this time, the computing device (100) may classify the frame into a first frame group if the type of the frame is the compressed intra-frame. Additionally, the computing device (100) may classify the frame into a second frame group if the type of the frame is the compressed inter-frame and the bit amount of the frame exceeds a preset threshold. Additionally, the computing device (100) may classify the frame into a third frame group if the type of the frame is the compressed inter-frame and the bit amount of the frame is less than or equal to a preset threshold.

[0127] From this, the computing device (100) can obtain an upscaled frame for each frame by inputting each frame belonging to the first frame group and each frame belonging to the second frame group into an AI model composed of an encoder and a decoder.

[0128] Additionally, the computing device (100) can obtain an upscaled frame for each frame by decoding each frame belonging to the third frame group and inputting it into a lightweight VSR model based on spatio-temporal information.

[0129] According to the method according to the embodiment of the present disclosure as described above, an optimal AI model for upscaling can be applied depending on the frame type and bit amount of the video.

[0130] FIG. 9 is an example of a formula expressing upscaling for intra-frame, inter-frame, and GOP according to some embodiments of the present disclosure.

[0131] Referring to FIG. 9, Equation (91) is an expression representing an upscaled frame generated by inputting a compressed intra-frame or a compressed inter-frame with a bit amount exceeding a threshold into an AI model (71) of an encoder-decoder structure.

[0132] Additionally, Equation (92) is an expression representing an upscaled frame generated by inputting a compressed inter-frame with a bit amount below a threshold into a lightweight VSR model (75) having an SFT layer (83). Here, L represents Laplacian enhancement, and refers to multi-frame ESPCN-based upscaling, and represents the weight parameter values ​​of the ESPCN and SFT layer (83).

[0133] Additionally, formula (93) is an upscaled result for one GOP (Group Of Pictures) ( ) upscaled result for intra-frame ( ) and, upscaled results for inter-frames( It is an expression using ).

[0134] Hereinafter, examples of training data for training an AI model used in an embodiment of the present disclosure will be described through FIGS. 10 to 12.

[0135] FIG. 10 shows an example of input data for training an AI model of an encoder-decoder structure applied to an intra-frame in relation to the training of the AI ​​model used in Flow 1 (771) of FIG. 7 described earlier.

[0136] Referring to FIG. 10, an AI model (53) of an encoder-decoder structure used for upscaling low-resolution compressed intra-frames uses a bitstream (51) of multiple intra-frames encoded at low resolution and a raw image (52) of multiple intra-frames at high resolution as training input data (50) during the training process. Here, low resolution means a resolution below a preset threshold, and high resolution means a resolution exceeding a preset threshold.

[0137] Figure 11 shows an example of input data for training an AI model of an encoder-decoder structure applied to an inter-frame in relation to the training of the AI ​​model used in Flow 2 (772) of Figure 7 described earlier.

[0138] Referring to FIG. 11, an AI model (113) of an encoder-decoder structure used for upscaling inter-frames compressed to a low resolution uses a bitstream (111) of multiple inter-frames encoded to a low resolution and a raw image (112) of multiple inter-frames at a high resolution as training input data (110) during the training process. Here, low resolution means a resolution below a preset threshold, and high resolution means a resolution exceeding a preset threshold.

[0139] Figure 12 shows an example of input data for training a lightweight VSR model based on spatiotemporal information applied to an inter-frame, in relation to the training of the AI ​​model used in Flow 3 (773) of Figure 7 described earlier.

[0140] Referring to FIG. 12, a spatiotemporal information-based lightweight VSR model (123) used for upscaling inter-frames compressed to low resolution uses raw images (121) of multiple inter-frames encoded to low resolution and raw images (122) of multiple inter-frames at high resolution as learning input data (120) during the learning process. Here, low resolution means a resolution below a preset threshold, and high resolution means a resolution exceeding a preset threshold.

[0141] In constructing the training input data described in FIGS. 10 to 12, for example, the CDVL (Consumer Digital Video Library) dataset and the REDS (Realistic and Dynamic Scenes) dataset may be utilized.

[0142] For example, the REDS datasets consist of a total of 300 video sequences, each with a resolution of 720 x 1280, and each video may consist of 100 frames. To train a consistent model using two datasets with different resolutions, frames obtained from the CDVL and REDS datasets can be reduced to high-quality frames of 960 x 540 using bicubic downsampling techniques, and then converted into low-quality frames through an interpolation process based on these frames. During this conversion process, a resize function applying an OpenCV-based inter-area algorithm can be applied. Subsequently, the acquired low-quality training frames are compressed using a standard video encoder, and encoding settings such as the compression standard profile, GOP, target bitrate, quantization value, maximum search range for motion estimation, and the number of reference frames may be used.

[0143] FIG. 13 is an example of a formula expressing a loss function used in the learning process of an AI model according to some embodiments of the present disclosure.

[0144] Referring to FIG. 13, in the training process of an AI model applied to intra-frames and high-bit inter-frames, a high-resolution frame obtained by performing Laplacian enhancement on an upscaled image obtained from an AI model with an encoder-decoder structure ( and the original high-definition frame ( The MSE loss function (131) between them can be used.

[0145] In addition, during the training process of an AI model applied to low-bit-rate inter-frames, the high-resolution inter-frame obtained by performing Laplacian enhancement on the output frame through ESPCN and SFT layers utilizing multiple frames ( and the original high-definition inter-frame ( In addition to the MSE loss function, the Huber loss function ( A loss function (132) that adds ) can be used. In this case, the Huber loss function can be used to constrain the values ​​of motion vectors during the learning process.

[0146] As described above, according to the embodiments of the present disclosure, optimal upscaling results can be obtained by differentially applying an AI model for upscaling based on the frame type and bit rate of the compressed video. Furthermore, as an alternative to existing complex VSR models applied to videos with a lot of motion, a lightweight VSR model utilizing not only codec information but also an SFT module can be applied.

[0147] Accordingly, selective upscaling is possible depending on network conditions and video frame types, and can be applied to various solutions requiring video upscaling.

[0148] FIG. 14 is a hardware configuration diagram of an exemplary computing device capable of implementing methods according to some embodiments of the present invention. As illustrated in FIG. 14, the computing device (100) may include one or more processors (101), a bus (107), a network interface (102), a memory (103) for loading a computer program (105) executed by the processor (101), and a storage (104) for storing the computer program (105). However, FIG. 14 illustrates only the components related to the embodiments of the present invention. Therefore, a person skilled in the art to which the present invention pertains will understand that other general-purpose components may be included in addition to the components illustrated in FIG. 14.

[0149] The processor (101) controls the overall operation of each component of the computing device (100). The processor (101) may be configured to include at least one of a CPU (Central Processing Unit), MPU (Micro Processor Unit), MCU (Micro Controller Unit), GPU (Graphic Processing Unit), or any form of processor well known in the art of the present invention. Additionally, the processor (101) may perform operations for at least one application or program for executing a method / operation according to various embodiments of the present invention. The computing device (100) may have one or more processors.

[0150] Memory (103) stores various data, commands and / or information. Memory (103) may load one or more programs (105) from storage (104) to execute methods / operations according to various embodiments of the present invention. For example, when a computer program (105) is loaded into memory (103), logic (or a module) may be implemented on memory (103). Examples of memory (103) may be RAM, but are not limited thereto.

[0151] The bus (107) provides communication functions between components of the computing device (100). The bus (107) can be implemented as various types of buses, such as an address bus, a data bus, and a control bus.

[0152] The network interface (102) supports wired and wireless internet communication of the computing device (100). The network interface (102) may also support various communication methods other than internet communication. To this end, the network interface (102) may be configured to include a communication module well known in the technical field of the present invention.

[0153] Storage (104) may store one or more computer programs (105) non-temporarily. Storage (104) may be configured to include non-volatile memory such as flash memory, a hard disk, a removable disk, or any form of computer-readable recording medium well known in the art to which the present invention belongs.

[0154] A computer program (105) may include one or more instructions in which methods / operations according to various embodiments of the present invention are implemented. When the computer program (105) is loaded into memory (103), a processor (101) may perform methods / operations according to various embodiments of the present invention by executing the one or more instructions.

[0155] In one embodiment, a computer program (105) may include instructions for performing operations to obtain an upscaled frame for a first frame by inputting the first frame into a first AI model if the type of the first frame of a video transmitted from a server in a streaming manner is a compressed intra-frame, an operation to determine whether to input the second frame into the first AI model based on the bit amount of the second frame if the type of the second frame of the transmitted video is a compressed inter-frame, and an operation to obtain an upscaled frame for the second frame by decoding the second frame and inputting it into a second AI model different from the first AI model if it is determined not to input the second frame into the first AI model.

[0156] In another embodiment, a computer program (105) performs the following operations: if the type of a first frame of a video transmitted from a server in a streaming manner is a compressed intra-frame, the first frame is input into an AI model composed of an encoder and a decoder to obtain an upscaled frame for the first frame; if the type of a second frame of the transmitted video is a compressed inter-frame, the bit amount of the second frame is identified as being greater than or equal to a preset threshold; and if the bit amount of the second frame is greater than or equal to a preset threshold, the second frame is input into the AI ​​model to obtain an upscaled frame for the second frame, and if the bit amount of the second frame is less than a preset threshold, the second frame is decoded and input into a spatio-temporal information-based lightweight VSR model to obtain an upscaled frame for the second frame. It may include instructions.

[0157] In another embodiment, the computer program (105) may include instructions for performing operations to identify the type of each of a plurality of frames included in a group of pictures (GOP) of a video transmitted from a server in a streaming manner, the bit amount of each frame, and to perform upscaling using different AI models for each frame group classified by the type and bit amount of each identified frame.

[0158] Various embodiments of the present invention and effects according to those embodiments have been described with reference to FIGS. 1 to 14. The effects according to the technical concept of the present invention are not limited to those mentioned above, and other unmentioned effects will be clearly understood by a person skilled in the art from the description below.

[0159] The technical concept of the present invention described so far may be implemented as computer-readable code on a computer-readable medium. The computer-readable recording medium may be, for example, a removable recording medium (CD, DVD, Blu-ray disc, USB storage device, removable hard disk) or a fixed recording medium (ROM, RAM, computer-equipped hard disk). The computer program recorded on the computer-readable recording medium may be transmitted to another computing device via a network such as the Internet and installed on the other computing device, thereby allowing it to be used on the other computing device.

[0160] Although it has been described above that all components constituting the embodiments of the present invention are combined as one or operate in combination, the technical concept of the present invention is not necessarily limited to such embodiments. That is, within the scope of the objectives of the present invention, all components may be selectively combined in one or more ways to operate.

[0161] Although operations are depicted in a specific order in the drawings, it should not be understood that the operations must be executed in the specific order depicted or in a sequential order, or that all depicted operations must be executed to obtain the desired result. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of the various configurations in the embodiments described above should not be understood as a necessary separation, and it should be understood that the described program components and systems can generally be integrated together into a single software product or packaged into multiple software products.

[0162] Although embodiments of the present invention have been described above with reference to the attached drawings, those skilled in the art will understand that the present invention may be implemented in other specific forms without altering the technical concept or essential features thereof. Therefore, the embodiments described above should be understood as illustrative in all respects and not restrictive. The scope of protection of the present invention shall be interpreted by the claims below, and all technical concepts within the equivalent scope shall be interpreted as being included within the scope of rights of the technical concept defined by the present invention.

Claims

Claim 1 A method for performing an AI-based video upscaling method by a computing device, comprising: a step of obtaining an upscaled frame for a first frame by inputting the first frame into a first AI model if the type of the first frame of the video transmitted from a server in a streaming manner is a compressed intra-frame; a step of determining whether to input the second frame into the first AI model based on the bit amount of the second frame if the type of the second frame of the transmitted video is a compressed inter-frame; and a step of obtaining an upscaled frame for the second frame by decoding the second frame and inputting it into a second AI model different from the first AI model if it is determined not to input the second frame into the first AI model. Claim 2 In claim 1, the step of obtaining an upscaled frame for the first frame by inputting the first frame into a first AI model comprises: the step of inputting the first frame into the first AI model composed of an encoder and a decoder; and the step of obtaining an upscaled frame for the first frame by performing pixel shuffling on the result output from the decoder of the first AI model. Claim 3 In claim 1, the first frame and the second frame are frames compressed to a resolution lower than a preset reference value, and the upscaled frame for the first frame and the upscaled frame for the second frame are frames decompressed to a resolution higher than a preset reference value, an AI-based video upscaling method. Claim 4 In claim 1, the step of determining whether to input the second frame into the first AI model based on the bit amount of the second frame comprises: a step of identifying whether the bit amount of the second frame exceeds a preset threshold; and a step of determining to input the second frame into the first AI model if it is identified that the bit amount exceeds the preset threshold. Claim 5 In claim 1, the step of determining whether to input the second frame into the first AI model based on the bit amount of the second frame comprises: the step of inputting the second frame into an AI-based discrimination model; and the step of determining whether to input the second frame into the first AI model based on the result output from the AI ​​discrimination model, wherein the AI-based video upscaling method comprises Claim 6 In claim 1, the step of obtaining an upscaled frame for the second frame by decoding the second frame and inputting it into a second AI model different from the first AI model comprises: the step of inputting the second frame into a decoder; and the step of obtaining an upscaled frame for the second frame by inputting the result output from the decoder into a lightweight VSR (Video Super-Resolution) model based on spatio-temporal information. Claim 7 In claim 6, the lightweight VSR (Video Super-Resolution) model comprises a plurality of convolution layers and a SFT (Spatial Feature Transform) layer, and the step of obtaining an upscaled frame for the second frame comprises: a step of sequentially inputting a result output from the decoder to the plurality of convolution layers; a step of inputting a result output through the plurality of convolution layers to the SFT layer; and a step of obtaining an upscaled frame for the second frame from a result output from the SFT layer, an AI-based video upscaling method. Claim 8 In claim 7, the step of inputting the result output through the plurality of convolution layers into the SFT layer comprises the step of calculating the output result of the SFT layer using an intermediate result obtained from the result of performing a convolution operation with the feature map output through the plurality of convolution layers and a patch condition using motion vectors as inputs. Claim 9 In claim 1, the first AI model applied to the first frame is a model trained using training data consisting of a bitstream of a plurality of intra-frames encoded at a resolution lower than a preset threshold and a raw image of a plurality of intra-frames at a resolution higher than the preset threshold, an AI-based video upscaling method. Claim 10 In claim 1, the first AI model applied to the second frame is a model trained using training data consisting of a bitstream of a plurality of inter-frames encoded at a resolution lower than a preset threshold and a raw image of a plurality of inter-frames at a resolution higher than the preset threshold, an AI-based video upscaling method. Claim 11 In claim 1, the second AI model applied to the second frame is a model trained using training data composed of raw images of a plurality of inter-frames with a resolution lower than a preset threshold and raw images of a plurality of inter-frames with a resolution higher than the preset threshold, an AI-based video upscaling method. Claim 12 In claim 1, the first AI model applied to the first frame and the second frame uses an MSE loss function during the learning process, and the second AI model applied to the second frame uses a Huber loss function during the learning process, an AI-based video upscaling method. Claim 13 A method performed by a computing device comprises: a step of obtaining an upscaled frame for the first frame by inputting the first frame into an AI model composed of an encoder and a decoder, if the type of the first frame of the video transmitted from a server in a streaming manner is a compressed intra-frame; and a step of identifying whether the bit amount of the second frame is greater than or equal to a preset threshold, if the type of the second frame of the transmitted video is a compressed inter-frame. An AI-based video upscaling method comprising the step of, if the bit amount of the second frame is greater than or equal to a preset threshold, inputting the second frame into the AI ​​model to obtain an upscaled frame for the second frame, and if the bit amount of the second frame is less than a preset threshold, decoding the second frame and inputting it into a spatio-temporal information-based lightweight VSR model to obtain an upscaled frame for the second frame. Claim 14 In claim 13, the spatiotemporal information-based lightweight VSR model includes a Spatial Feature Transform (SFT) module, wherein the SFT module generates different scale and shift parameters for each position using a patch condition using motion vectors obtained through a plurality of convolution layers, and performs a positionally adjusted transformation on a feature map obtained through the plurality of convolution layers using the generated scale and shift parameters. Claim 15 An AI-based video upscaling method, comprising: a step of identifying the type of each of a plurality of frames included in a group of pictures (GOP) of a video transmitted from a server in a streaming manner, and the bit amount of each frame; and a step of performing upscaling using different AI models for each frame group classified by the identified type and bit amount of each frame. Claim 16 In claim 15, the Group of Pictures (GOP) comprises one compressed intra-frame and two or more compressed inter-frames, and the step of performing upscaling using different AI models for each frame group classified by the type and bit amount of each identified frame comprises: a step of classifying the frame into a first frame group when the type of the frame is the compressed intra-frame; a step of classifying the frame into a second frame group when the type of the frame is the compressed inter-frame and the bit amount of the frame exceeds a preset threshold; and a step of classifying the frame into a third frame group when the type of the frame is the compressed inter-frame and the bit amount of the frame is less than or equal to a preset threshold. Claim 17 In claim 16, the step of performing upscaling using different AI models for each frame group classified by the type and bit amount of each identified frame comprises: a step of obtaining an upscaled frame for each frame by inputting each frame belonging to the first frame group and each frame belonging to the second frame group into an AI model composed of an encoder and a decoder; and a step of obtaining an upscaled frame for each frame by decoding each frame belonging to the third frame group and inputting it into a lightweight VSR model based on spatio-temporal information. Claim 18 A video upscaling device comprising: one or more processors; memory for loading a computer program executed by said processors; and storage for storing said computer program, wherein the computer program comprises instructions for performing operations such as: acquiring an upscaled frame for said first frame by inputting said first frame into a first AI model if the type of a first frame of a video transmitted from a server in a streaming manner is a compressed intra-frame; determining whether to input said second frame into the first AI model based on the bit amount of said second frame if the type of a second frame of said transmitted video is a compressed inter-frame; and, if it is determined not to input said second frame into the first AI model, acquiring an upscaled frame for said second frame by decoding said second frame and then inputting said second AI model different from said first AI model. Claim 19 A video upscaling device according to claim 18, wherein the first AI model is an AI model composed of an encoder and a decoder, and the second AI model is a lightweight VSR (Video Super-Resolution) model based on spatio-temporal information. Claim 20 A video upscaling device comprising: one or more processors; memory for loading a computer program executed by said processors; and storage for storing said computer program, wherein the computer program comprises instructions for performing operations to identify the type of each of a plurality of frames included in a group of pictures (GOP) of a video transmitted from a server in a streaming manner, the bit amount of each frame, and the operation of performing upscaling using different AI models for each frame group classified by the identified type and bit amount of each frame.

Images