High-speed real-time global encoding method capable of consistent image quality of given size when generating video having similar information distribution to original video

Abstract

The present specification discloses an encoding method capable of efficient resource distribution. A video encoding method according to the present specification may be executed in a state of being stored in a main memory in a computing device by segmenting a portion of a video into block data and inputting the segmented data to an image processing model. Through this, image processing and encoding are possible at a faster speed than the prior art.

Description

A high-speed, real-time global encoding method that enables consistent image quality of a given size when generating a video with similar information distribution to the original video.

[0001] The present invention relates to video encoding, and more particularly, to an encoding method capable of efficient resource allocation.

[0002] This application claims priority to Korean Patent Application No. 10-2024-0048820, filed on April 11, 2024, the entire disclosure of which is incorporated herein by reference.

[0003] The material described in this section merely provides background information on the embodiments described herein and does not necessarily constitute prior art.

[0004] The fundamental principle of video is an optical illusion: when multiple still images are shown in rapid succession, they appear to be moving. When still images are shown in rapid succession, the brain experiences an afterimage effect, unable to properly recognize each image and instead sees it as a continuation of the previous image. This creates the illusion of movement.

[0005] Video must be still images at 24 frames per second or higher to convey natural motion. However, video files typically use compression techniques to compress large amounts of data into smaller files. This process is called video encoding.

[0006] The main bitrate distribution methods used in video encoding are as follows: CBR (Constant Bit Rate) maintains a constant bitrate. This method makes it easy to predict the encoded video size, making it primarily used in live streaming and broadcasting. However, it can result in inconsistent image quality. VBR (Variable Bit Rate) dynamically adjusts the bitrate based on the video's complexity. This method offers better image quality, but can result in unpredictable file sizes and longer encoding times. ABR (Average Bit Rate) is a hybrid between CBR and VBR, setting an average bitrate target and adjusting the bitrate across various parts of the video within a consistent range. This method offers more predictable file sizes than VBR while maintaining a consistent level of image quality. CRF (Constant Rate Factor) is a variant of VBR that adjusts the bitrate to maintain a specific "quality level." Lower CRF values ​​improve image quality but increase file size. It is primarily used when file storage space is not limited.

[0007] The purpose of this specification is to provide an encoding method that enables efficient resource allocation.

[0008] This specification is not limited to the above-mentioned tasks, and other tasks not mentioned will be clearly understood by those skilled in the art from the description below.

[0009] A video encoding method according to one embodiment of the present specification for solving the above-described problem may include: (a) a step in which a processor stores video data as input data; (b) a step in which the processor extracts block data of a preset size from the video data; (c) a step in which the processor inputs the block data into an image processing model; and (d) a step in which the processor calculates a characteristic value of the image data output from the image processing model and encodes the image data according to the calculated characteristic value. In this case, steps (b) and (d) may be repeatedly executed until block data extraction is performed for all of the video data.

[0010] According to one embodiment of the present specification, step (d) may be a step in which the processor encodes within bit resources pre-allocated to the corresponding block data.

[0011] A video encoding method according to another embodiment of the present specification for solving the above-described problem may include: (a) a step in which a processor stores video data as input data; (b) a step in which the processor calculates a feature value for the entire video data; (c) a step in which the processor extracts block data of a preset size from the video data; (d) a step in which the processor inputs the block data into an image processing model; and (e) a step in which the processor encodes image data output from the image processing model according to the feature value calculated in step (b). In this case, steps (c) and (e) may be repeatedly executed until block data extraction is performed for the entire video data.

[0012] According to another embodiment of the present specification, step (e) may be a step in which the processor encodes the entire input video within pre-allocated bit resources.

[0013] According to this specification, the characteristic value may be the complexity of an image (static or dynamic). In this case, the processor may calculate the complexity using at least one of motion information analysis, variation, prediction error, quality requirements, screen division, and screen switching.

[0014] According to this specification, the processor can generate the corresponding block data into at least one frame data and input the generated frame data into an image processing model. In this case, the image processing model may be a denoising or super-resolution model.

[0015] According to this specification, the processor can encode by adjusting the bit rate according to the calculated characteristic value.

[0016] According to this specification, the processor can merge encoded data and output it as a single video data.

[0017] According to this specification, the processor can output image data output from the image processing model as one video data through one encoding.

[0018] The video encoding method according to the present specification is characterized in that it can be executed while block data is stored in the main memory of a computing device.

[0019] The video encoding method according to the present specification can be implemented in the form of a computer program written to perform each step of the video encoding method on a computer and recorded on a computer-readable recording medium.

[0020] Other specific details of the present invention are included in the detailed description and drawings.

[0021] According to one aspect of this specification, bitrate resources can be efficiently allocated when encoding video. Therefore, image quality can be improved even within limited resources.

[0022] According to another aspect of the present specification, fast encoding is possible by processing on main memory.

[0023] According to another aspect of the present specification, encoding of the entire image can be completed simultaneously with the completion of processing the last frame of image data sequentially input to the image processing model. Therefore, encoding is possible in real time without waiting for an additional encoding process for the entire processed image data.

[0024] The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

[0025] Figure 1 is a reference diagram for an image processing and encoding process using an image processing model according to a prior art.

[0026] FIG. 2 is a flowchart of a video encoding method according to one embodiment of the present specification.

[0027] FIG. 3 is a reference diagram for block-by-block 2-pass encoding according to one embodiment of the present specification.

[0028] FIG. 4 is a flowchart of a video encoding method according to another embodiment of the present specification.

[0029] FIG. 5 is a reference diagram for a separate 2-pass encoding according to another embodiment of the present specification.

[0030] The advantages and features of the invention disclosed in this specification, and the methods for achieving them, will become clearer with reference to the embodiments described in detail below together with the accompanying drawings. However, this specification is not limited to the embodiments disclosed below and may be implemented in various different forms. These embodiments are provided only to ensure that the disclosure of this specification is complete and to fully inform those of ordinary skill in the art (hereinafter referred to as "skilled workers") of the scope of this specification, and the scope of rights of this specification is defined only by the scope of the claims.

[0031] The terminology used herein is for the purpose of describing embodiments and is not intended to limit the scope of the present disclosure. In this specification, singular forms also include plural forms, unless specifically stated otherwise. As used herein, the terms "comprises" and / or "comprising" do not exclude the presence or addition of one or more other components in addition to the components mentioned.

[0032] Throughout the specification, the same reference numerals refer to the same elements, and the term "and / or" includes each and every combination of the elements mentioned. Although terms such as "first," "second," etc. are used to describe various elements, these elements are not limited by these terms. These terms are used only to distinguish one element from another. Therefore, it should be understood that a first element mentioned below may also be a second element within the technical scope of the present invention.

[0033] Unless otherwise defined, all terms (including technical and scientific terms) used herein may be used in their common sense to those skilled in the art to which this specification pertains. Furthermore, terms defined in commonly used dictionaries are not to be interpreted ideally or excessively unless explicitly and specifically defined otherwise. Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0034] There are two types of encoding: single-pass encoding and two-pass encoding. Single-pass encoding encodes video files in a single pass. During this process, the encoder dynamically adjusts the bitrate and processes the file. Its advantage is its fast encoding speed. Therefore, it is used for real-time streaming or other situations requiring fast encoding. Its disadvantage is its inefficient bitrate distribution, which can lead to inconsistent image quality. In particular, single-pass encoding is more likely to result in image quality degradation in complex video scenes. Two-pass encoding encodes video files in two passes. The first pass analyzes the complexity of the video data, analyzing its complexity and bitrate distribution. The encoder traverses the entire video, calculating the visual complexity and required data volume for each frame. The second pass optimizes bitrate distribution based on this information. Compared to single-pass encoding, two-pass encoding allows for more precise bitrate adjustment, resulting in improved overall image quality. In particular, two-pass encoding can provide higher bitrates and better image quality for complex or dynamic scenes. The downside is that encoding times are longer than one-pass encoding, and the entire process requires more system resources.

[0035] Meanwhile, recent tasks like denoising and super-resolution utilize image processing models. Image processing models, which are artificial neural network models, remove noise from input images or increase their resolution before outputting them. The problem with these image processing models is that encoded image data is not input directly, but rather, it must be divided into frames, with each frame input.

[0036] Figure 1 is a reference diagram for an image processing and encoding process using an image processing model according to a prior art.

[0037] Referring to Fig. 1, the conventional technology separates the entire input video into individual frames. The separated individual frames are input to an image processing model, and the processed frame images are all collected and encoded to generate an output video. In this process, if the length of the input video is long, the entire separated frame data is very large and is stored and processed in secondary storage rather than main memory, which significantly slows down the processing speed. In this specification, main memory refers to a memory device directly connected to a processor, such as RAM (Random Access Memory), and secondary memory refers to a non-volatile storage device, such as an HDD (hard disk drive) or SSD (solid-state drive).

[0038] A typical two-pass encoding scheme, applicable when access to the entire video is available, is illustrated. In Pass 1, the video's complexity is analyzed. In Pass 2, bits are allocated and encoded based on the Pass 1 records. This has the advantage of optimally distributing limited bit resources, but it also has the disadvantage of taking excessively long encoding times.

[0039] In order to improve the shortcomings of these conventional technologies, a video encoding method according to the present specification was conceived.

[0040] FIG. 2 is a flowchart of a video encoding method according to one embodiment of the present specification.

[0041] FIG. 3 is a reference diagram for block-by-block 2-pass encoding according to one embodiment of the present specification.

[0042] Referring to FIGS. 2 and 3, in a video encoding method according to one embodiment of the present disclosure, first, in step S100, the processor can store video data as input data. The video data is input video and is not limited by the size and format of the data.

[0043] In the next step S110, the processor can extract block data of a preset size from the video data. The block data may have various sizes and may preferably be extracted sequentially within the video data.

[0044] In the next step S120, the processor can input the corresponding block data into the image processing model. More specifically, the processor can generate the corresponding block data into at least one frame data and input the generated frame data into the image processing model. At this time, the size of the generated frame data can be executed while stored in the main memory of the computing device. That is, unlike the conventional technology where the entire frame data of a video is too large to be stored and processed in auxiliary memory, some frame data of the video may be relatively small in size and may be stored and processed in the main memory. For this purpose, it is obvious that a person skilled in the art can appropriately set the size of the block data. Meanwhile, the image processing model may be various image processing models such as a denoising or super-resolution model.

[0045] In the next step S130, the processor can calculate a characteristic value of the output image data. The characteristic value is reference data that can improve encoding efficiency during a subsequent encoding process. According to one embodiment, the characteristic value is the complexity of the image (static or dynamic).

[0046] The processor can calculate complexity using at least one of motion information analysis (Motion Estimation), variability, prediction error, quality constraints, macroblock partitioning, and cutaway. Each of the above methods is known to those skilled in the art, and a detailed description thereof will be omitted. In addition to the complexity calculation methods listed above, various methods that can be easily used by those skilled in the art after the present application may also be included.

[0047] And the processor can encode according to the calculated characteristic value (complexity). For example, a frame with relatively high complexity can be encoded by increasing the bit rate, and a frame with relatively low complexity can be encoded by decreasing the bit rate. In other words, the bit rate can be adjusted according to the characteristic value (complexity). In this process, the processor can encode within the bit resources pre-allocated to the corresponding block data. At this time, the size of the frame data output from the image processing model can also be a size that can be stored and processed in the main memory. Therefore, the encoding process of step S130 can also be executed while being stored in the main memory of the computing device.

[0048] In step S140, the processor can determine whether block data extraction has been performed for the entire video. If block data extraction has not yet been performed for the entire video ("NO" in step S140), the process can proceed to step S110. Steps S110 to S140 can then be repeatedly performed. On the other hand, if block data extraction has been performed for the entire video ("YES" in step S140), the encoding process is completed.

[0049] Afterwards, the processor can merge the encoded data and output it as a single video data.

[0050] A video encoding method according to one embodiment of the present specification analyzes and encodes characteristics only with frames included in block data, and will be referred to as "block-by-block 2-pass encoding." The block-by-block 2-pass encoding according to the present specification can achieve a fast processing speed by storing and processing in the main memory without using an auxiliary storage device compared to the prior art through block size adjustment. In addition, encoding can be performed by most appropriately distributing limited bit resources within a single block. However, since bit resources are distributed only within a single block, even if any one block in the entire video is simple, excess bit resources are not distributed to other blocks.

[0051] FIG. 4 is a flowchart of a video encoding method according to another embodiment of the present specification.

[0052] FIG. 5 is a reference diagram for a separate 2-pass encoding according to another embodiment of the present specification.

[0053] Referring to FIGS. 4 and 5, a video encoding method according to another embodiment of the present disclosure may first, in step S200, a processor may store video data as input data. The video data is input video and is not limited by the size and format of the data.

[0054] In the next step S210, the processor can calculate a feature value for the entire video data. This is the difference from the block-by-block 2-pass encoding described above. The feature value can be the complexity of the video. The processor can calculate the complexity using at least one of motion information analysis, variation, prediction error, quality requirements, and screen division. Each of the above methods is known to those skilled in the art, and a detailed description thereof will be omitted. In addition to the complexity calculation methods listed above, various methods that can be easily used by those skilled in the art after the present application may also be included.

[0055] In the next step S220, the processor can extract block data of a preset size from the video data. The block data may have various sizes and may preferably be extracted sequentially within the video data.

[0056] In the next step S230, the processor may input the corresponding block data into the image processing model. More specifically, the processor may generate the corresponding block data into at least one frame data and input the generated frame data into the image processing model. At this time, the size of the generated frame data may be executed while stored in the main memory of the computing device. That is, unlike the conventional technology where the entire frame data of a video is too large to be stored and processed in auxiliary memory, some frame data of the video may be relatively small in size and may be stored and processed in the main memory. For this purpose, it is obvious that a person skilled in the art can appropriately set the size of the block data. Meanwhile, the image processing model may be various image processing models, such as a denoising or super-resolution model.

[0057] In the next step S240, the processor can encode the image data output from the image processing model according to the characteristic value (complexity) calculated in step S210. For example, frames with relatively high complexity can be encoded by increasing the bit rate, and frames with relatively low complexity can be encoded by decreasing the bit rate, that is, by adjusting the bit rate according to the characteristic value (complexity). In this process, the processor can encode within the bit resources pre-allocated to the entire input video. This is the difference from the block-type 2-pass encoding described above. At this time, the size of the frame data output from the image processing model can also be a size that can be stored and processed in the main memory. Therefore, the encoding process of step S240 can also be executed while being stored in the main memory of the computing device.

[0058] In step S250, the processor can determine whether block data extraction has been performed for the entire video. If block data extraction has not yet been performed for the entire video ("NO" in step S250), the process can proceed to step S220. Steps S220 to S250 can then be repeatedly performed. On the other hand, if block data extraction has been performed for the entire video ("YES" in step S250), the encoding process is completed.

[0059] Thereafter, the processor can merge the encoded data and output it as a single video data. According to another embodiment, the processor can output the video data output from the image processing model as a single video data through a single encoding. Since the encoding operation can be paused and restarted, the video data output from the image processing model can be added to the encoding queue. In this case, a single video can be output through a single encoding without generating block-unit clips. In other words, this means a method in which image processing and encoding are performed simultaneously.

[0060] A video encoding method according to another embodiment of the present specification analyzes the characteristics of frames included in the entire video data, divides them into block data, and encodes them, which will be called "separate 2-pass encoding." This is because the step of calculating the characteristic value (pass 1) and the step of encoding (pass 2) are separated in terms of location. The separate 2-pass encoding according to the present specification can achieve a fast processing speed by storing and processing in the main memory without using an auxiliary storage device compared to the prior art through block size adjustment. In addition, unlike the block-type 2-pass encoding, it can encode by most appropriately distributing bit resources in the entire video. Therefore, if any one block in the entire video is simple, the extra bit resources can be distributed to other blocks. On the other hand, block division and encoding can be performed after the characteristic value (complexity) for the entire video data has been calculated.

[0061] The block-type 2-pass encoding and separate 2-pass encoding according to the present specification process video data by dividing it into block data, and thus has the great advantage of fast execution speed while stored in the main memory of the computing device. When the applicant of the present invention executed a 37-second FHD video (816*1920) using two RTX 3090 GPUs and excluding audio using the conventional 2-pass encoding (a method of dividing the entire frame into individual sheets and then storing it on the HDD), the total work time including model inference and encoding took 22 minutes. On the other hand, in the case of the block-type 2-pass encoding and separate 2-pass encoding according to the present specification (a method of sequentially storing only a portion in memory), the total work time including model inference and encoding took 15 minutes, showing an improvement in processing speed that was about 1.46 times faster than the conventional technology.

[0062] The video encoding method according to the present specification may include a processor, an application-specific integrated circuit (ASIC), another chipset, a logic circuit, a register, a communication modem, a data processing device, etc. known in the art to which the present invention pertains, to execute the above-described calculation and various control logic. In addition, when the above-described control logic is implemented in software, the video encoding method may be implemented as a set of program modules. In this case, the program modules may be stored in a memory device and executed by the processor.

[0063] The above-described computer program may include codes coded in a computer language, such as C / C++, C#, JAVA, Python, or machine language, that can be read by the processor (CPU) of the computer through the device interface of the computer, so that the computer reads the program and executes the methods implemented as a program. Such codes may include functional codes related to functions that define functions necessary for executing the methods, and may include control codes related to execution procedures necessary for the processor of the computer to execute the functions according to a predetermined procedure. In addition, such codes may further include memory reference-related codes regarding which location (address address) of the internal or external memory of the computer should reference additional information or media necessary for the processor of the computer to execute the functions. In addition, if the processor of the computer needs to communicate with any other computer or server located remotely in order to execute the functions, the code may further include communication-related code regarding how to communicate with any other computer or server located remotely using the communication module of the computer, and what information or media to send and receive during communication.

[0064] The above storage medium refers to a medium that stores data semi-permanently and can be read by a device, rather than a medium that stores data for a short period of time, such as a register, cache, or memory. Specifically, examples of the storage medium include, but are not limited to, ROM, RAM, CD-ROM, magnetic tape, floppy disk, and optical data storage device. That is, the program can be stored in various recording media on various servers that the computer can access or in various recording media on the user's computer. In addition, the medium can be distributed across network-connected computer systems, so that computer-readable code can be stored in a distributed manner.

[0065] While the embodiments of this specification have been described with reference to the attached drawings, those skilled in the art will appreciate that the present invention can be implemented in other specific forms without altering the technical spirit or essential features thereof. Therefore, the embodiments described above should be understood to be illustrative in all respects and not restrictive.

Claims

1. (a) A step in which the processor stores video data as input data; (b) a step in which the processor calculates characteristic values ​​for the entire video data; (c) a step in which the processor extracts block data of a preset size from the video data; (d) a step in which the processor inputs the corresponding block data into the image processing model; and (e) a step of encoding image data output from the image processing model according to the characteristic values ​​calculated in step (b); including, A video encoding method, wherein steps (c) and (e) above are repeatedly executed until block data extraction is performed for the entire video data.

2. In claim 1 The above characteristic value is the complexity of the image, a video encoding method.

3. In claim 2, Step (b) above, A video encoding method, characterized in that the processor calculates complexity by at least one of motion information analysis, variation, prediction error, quality requirement, screen division, and scene transition.

4. In claim 1, Step (d) above, A video encoding method, wherein the processor generates the corresponding block data into at least one frame data and inputs the generated frame data into an image processing model.

5. In claim 1, The above image processing model is a video encoding method that is a denoising or super-resolution model.

6. In claim 1, Step (e) above, A video encoding method, wherein the processor is a step of encoding by adjusting the bit rate according to the characteristic value calculated in the above step (e).

7. In claim 1, Step (e) above, A video encoding method, wherein the processor encodes the entire input video within pre-allocated bit resources.

8. In claim 1, Step (e) above, A video encoding method, wherein the processor outputs video data output from the above image processing model as one video data through one encoding.

9. In claim 1, (f) A video encoding method further comprising a step of merging encoded data by the processor and outputting it as one video data.

10. In claim 1, A video encoding method, characterized in that the steps (c) and (e) above are executed while stored in the main memory of a computing device.

11. A computer program written to perform each step of a video encoding method according to any one of claims 1 to 10 on a computer and recorded on a computer-readable recording medium.

Images