Apparatus and method for efficiently training neural network model to improve video quality

Abstract

The present specification discloses an apparatus and method for training an image quality improvement model. The apparatus for training an image quality improvement model, according to the present specification, may: split each of an original image frame and a high-definition image frame into areas of a preset size to generate each area as one image patch; input an original image patch as a first input into a model training unit; and input a high-definition image patch corresponding to the original image patch as a second input into the model training unit.

Description

Efficient training device and method for neural network models for improving video quality

[0001] The present invention relates to an efficient learning device and method for a neural network model for improving video quality.

[0002] The material described in this section merely provides background information on the embodiments described herein and does not necessarily constitute prior art.

[0003] With the advancement of artificial intelligence technology, research is being conducted on technologies to improve the image quality of original videos and generate high-quality videos. To train a neural network model that generates high-quality videos by improving the image quality of the original video, image frame data from the same frame in the original and high-quality videos are paired and input into the neural network model.

[0004] Training of a neural network model is primarily performed by a GPU (Graphics Processing Unit) equipped in a computer. The central processing unit (CPU) of the computer can load data of a neural network model stored in a storage medium into the memory of the GPU. Thereafter, the CPU can transfer image frame data contained in original videos and high-definition videos stored in the storage medium to the memory of the GPU. The GPU can input the image frame data of the original videos and high-definition videos into the neural network model data to train the neural network model.

[0005] However, when transmitting image frame data of original video and high-definition video to the GPU, if the size of the image frame data is larger than the memory available in the GPU, the image frame data cannot be transmitted to the GPU. Therefore, a preprocessing technology is required to transmit the image frame data of original video and high-definition video to the GPU depending on the memory size of the GPU.

[0006] When training AI models, the balance of training data can significantly impact their performance. If the data is biased, the AI ​​model may overtrain for specific situations or conditions, limiting its performance in general situations.

[0007] The purpose of this specification is to provide an efficient learning device and method for a neural network model for improving the image quality of a video.

[0008] This specification is not limited to the above-mentioned tasks, and other tasks not mentioned will be clearly understood by those skilled in the art from the description below.

[0009] In order to solve the above-described problem, the present specification provides a device for learning a quality improvement model, which may include: a storage unit that stores an original video file, a high-definition video file in which the quality of the original video is improved, and quality improvement learning model data that improves the quality of an image; a model learning unit that receives an image before quality improvement as a first input and an image after quality improvement as a second input to learn the quality improvement learning model; a frame extraction unit that extracts an image frame from the original video file to generate an original image frame, and extracts an image frame corresponding to the original image frame from the high-definition video file to generate a high-definition image frame; a patch generation unit that divides each of the original image frame and the high-definition image frame into regions of a preset size and generates each region as a single image patch; and a learning control unit that loads the quality improvement learning model into the model learning unit, inputs an original image patch generated from the original image frame as a first input to the model learning unit, and inputs a high-definition image patch corresponding to the original image patch from the high-definition image frame as a second input to the model learning unit.

[0010] At this time, the image quality of the high-definition video file may be characterized by being improved through at least one of noise removal, resolution enhancement, and color correction from the original video file.

[0011] According to one embodiment of the present specification, the patch generation unit can adjust the size of the image patch generated according to the memory size of the model learning unit.

[0012] According to one embodiment of the present specification, the size of the original image patch and the size of the high-quality image patch may be different from each other.

[0013] The image quality improvement model learning device according to the present specification may further include a learning data selection unit that selects the target image patch and the image patch corresponding thereto as learning data based on the amount of change between the target image patch generated from either the original image frame or the high-quality image frame and the preceding image patch generated from a frame preceding the frame in which the target image patch is generated in the video file.

[0014] According to one embodiment of the present specification, the learning data selection unit may calculate a change amount between the target image patch and the preceding image patch, and if the calculated change amount is greater than a preset reference value, select the target image patch and the corresponding image patch as learning data.

[0015] According to another embodiment of the present specification, the learning data selection unit may select the target image patch and the corresponding image patch as learning data when the sum or average of the changes in the corresponding pixel values ​​in the target image patch and the preceding image patch is greater than or equal to a preset reference value.

[0016] According to another embodiment of the present specification, each of the target image patch and the preceding image patch is divided into a plurality of target image sub-patches and preceding image sub-patches, and a sum of changes in pixel values ​​corresponding to each other in each of the target image sub-patches and the preceding image sub-patches is calculated, and if the average of the sum of the calculated changes is greater than or equal to a preset reference value, the target image patch and the corresponding image patch can be selected as learning data.

[0017] At this time, the learning data selection unit may set the sum of the changes in the corresponding pixel values ​​in the target image sub-patch and the preceding image sub-patch to be equal to or less than a preset upper limit when the sum of the changes is equal to or greater than a preset reference value, and may set the sum of the changes to be equal to or greater than a preset lower limit when the sum of the changes is equal to or less than a preset reference value.

[0018] According to one embodiment of the present specification, the learning control unit can input the target image patch selected by the learning data selection unit and the image patch corresponding thereto to the model learning unit.

[0019] The image quality improvement model learning device according to the present specification can store the learned image quality improvement model in the storage unit.

[0020] A method for learning a quality improvement model according to the present specification comprises: a storage unit for storing an original video file, a high-definition video file in which the quality of the original video is improved, and quality improvement learning model data for improving the quality of an image; and a model learning unit for learning the quality improvement learning model by receiving an image before quality improvement as a first input and an image after quality improvement as a second input; wherein the device for learning a quality improvement model comprises: a frame extraction step in which a processor extracts an image frame from the original video file to generate an original image frame, and extracts an image frame corresponding to the original image frame from the high-definition video file in which the quality of the original video is improved to generate a high-definition image frame; a patch generation step in which the processor divides each of the original image frame and the high-definition image frame into regions of a preset size and generates each region as a single image patch; and a learning control step in which the processor loads the quality improvement learning model into the model learning unit, inputs an original image patch generated from the original image frame as a first input to the model learning unit, and inputs a high-definition image patch corresponding to the original image patch from the high-definition image frame as a second input to the model learning unit.

[0021] At this time, the image quality of the high-definition video file may be characterized by being improved through at least one of noise removal, resolution enhancement, and color correction from the original video file.

[0022] According to one embodiment of the present specification, the patch generation step may be a step in which the processor adjusts the size of the image patch generated according to the memory size of the model learning unit.

[0023] According to one embodiment of the present specification, the size of the original image patch and the size of the high-quality image patch may be different from each other.

[0024] The image quality improvement model learning method according to the present specification is such that, after the patch generation step, the processor learns the target image patch and the corresponding image patch based on the amount of change between the target image patch generated from either the original image frame or the high-quality image frame and the preceding image patch generated from a frame preceding the frame in which the target image patch was generated in the video file.

[0025] It may further include a step of selecting learning data by selecting data;

[0026] According to one embodiment of the present specification, the learning data selection step may include the processor calculating a change amount between the target image patch and a preceding image patch, and selecting the target image patch and the corresponding image patch as learning data when the calculated change amount is greater than or equal to a preset reference value.

[0027] According to another embodiment of the present specification, the learning data selection step may be a step in which the processor selects the target image patch and the corresponding image patch as learning data when the sum or average of the changes in the corresponding pixel values ​​in the target image patch and the preceding image patch is greater than or equal to a preset reference value.

[0028] According to another embodiment of the present specification, the learning data selection step may be a step in which the processor divides each of the target image patch and the preceding image patch into a plurality of target image sub-patches and preceding image sub-patches, calculates a sum of changes in pixel values ​​corresponding to each other in each of the target image sub-patches and the preceding image sub-patches, and selects the target image patch and the corresponding image patch as learning data when the average of the sum of the calculated changes is equal to or greater than a preset reference value.

[0029] At this time, the learning data selection step may be a step in which, when the sum of the changes in the corresponding pixel values ​​in the target image sub-patch and the preceding image sub-patch is greater than or equal to a preset reference value, the processor sets the sum of the changes to be less than or equal to a preset upper limit value, and when the sum of the changes is less than or equal to the preset reference value, the processor sets the sum of the changes to be greater than or equal to a preset lower limit value.

[0030] According to one embodiment of the present specification, the learning step may be a step in which the processor inputs the target image patch selected from the learning data selection unit and the corresponding image patch to the image quality improvement learning model.

[0031] The method for learning a picture quality improvement model according to the present specification can be implemented in the form of a computer program written to perform each step on a computer and recorded on a computer-readable recording medium.

[0032] Other specific details of the present invention are included in the detailed description and drawings.

[0033] According to one aspect of the present specification, a device for learning a quality improvement model can generate image patches divided into regions of a preset size from image frames of an original video file and a high-quality video file with improved quality, depending on the memory size of a model learning unit provided therein.

[0034] According to another aspect of the present specification, the image quality improvement model learning device can learn video data more effectively than in the past by transmitting an original image patch generated from the original video image frame and a high-quality image patch generated from a high-quality video image frame and corresponding to the original image patch to the model learning unit.

[0035] According to another aspect of the present specification, the image quality improvement model learning device can reduce unnecessary learning data by selectively transmitting the image patch to the model learning unit according to the amount of change from the image patch in the previous image frame.

[0036] In AI model training, the foreground, which is a relatively important subject, and the background, which is a relatively less important subject, may differ in terms of image production and characteristics. For example, the foreground may be focused and relatively clear, while the background may be relatively defocused and less clear. Lighting differences can also occur. In a static background, the foreground may change significantly with movement, while the background may change less.

[0037] Due to the above reasons, background and foreground noise characteristics may differ, a common problem in images. If there is an imbalance in the training data between the background and foreground, which have different noise characteristics, the performance of an AI model trained with this data may be limited. The present invention fundamentally addresses this issue.

[0038] For example, when a foreground object moves only in front of a portion of a static background, if all image patches of each image frame are equally selected as training data, an imbalance in the training data may occur between the image patches containing the foreground and the background image patches. The present invention can resolve this imbalance in the training data.

[0039] The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

[0040] FIG. 1 is a block diagram of a device for learning a picture quality improvement model according to one embodiment of the present specification.

[0041] Figure 2 is an example image of generating an image patch by extracting image frames from an original video file and a high-definition video file.

[0042] Figure 3 is an example image of inputting learning data into a quality improvement learning model.

[0043] FIG. 4 is a block diagram of a device for learning a picture quality improvement model according to another embodiment of the present specification.

[0044] Figure 5 is an example image of a learning data selection unit selecting learning data.

[0045] Figure 6 is another example image of a learning data selection unit selecting learning data.

[0046] Figure 7 is a flowchart of a method for learning a quality improvement model according to one embodiment of the present specification.

[0047] FIG. 8 is a flowchart of a method for learning a quality improvement model according to another embodiment of the present specification.

[0048] The advantages and features of the invention disclosed in this specification, and the methods for achieving them, will become clearer with reference to the embodiments described in detail below together with the accompanying drawings. However, this specification is not limited to the embodiments disclosed below and may be implemented in various different forms. These embodiments are provided only to ensure that the disclosure of this specification is complete and to fully inform those of ordinary skill in the art (hereinafter referred to as "skilled workers") of the scope of this specification, and the scope of rights of this specification is defined only by the scope of the claims.

[0049] The terminology used herein is for the purpose of describing embodiments and is not intended to limit the scope of the present disclosure. In this specification, singular forms also include plural forms, unless specifically stated otherwise. As used herein, the terms "comprises" and / or "comprising" do not exclude the presence or addition of one or more other components in addition to the components mentioned.

[0050] Throughout the specification, the same reference numerals refer to the same elements, and the term "and / or" includes each and every combination of the elements mentioned. Although terms such as "first," "second," etc. are used to describe various elements, these elements are not limited by these terms. These terms are used only to distinguish one element from another. Therefore, it should be understood that a first element mentioned below may also be a second element within the technical scope of the present invention.

[0051] Unless otherwise defined, all terms (including technical and scientific terms) used herein may be used in their common sense to those skilled in the art to which this specification pertains. Furthermore, terms defined in commonly used dictionaries are not to be interpreted ideally or excessively unless explicitly and specifically defined otherwise.

[0052] Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings.

[0053] FIG. 1 is a block diagram of a device for learning a picture quality improvement model according to one embodiment of the present specification.

[0054] Referring to FIG. 1, a picture quality improvement model learning device (1) according to one embodiment of the present specification may include a storage unit (10), a model learning unit (11), a frame extraction unit (12), a patch generation unit (13), and a learning control unit (14).

[0055] The storage unit (10) above can store an original video file, a high-definition video file with improved image quality of the original video, and image quality improvement learning model data that improves the image quality of the image. The high-definition video file may be a file with improved image quality through at least one of noise removal, resolution enhancement (Super Resolution), and color grading of the original video file. This is only one example, and various algorithms such as dynamic range enhancement and edge enhancement may be used, and are not limited by the method.

[0056] The above image quality enhancement learning model may correspond to a neural network model that receives data from an original video file and generates a high-quality video file with improved image quality. The image quality enhancement learning model may learn the input data using machine learning and / or deep learning-based learning methods widely known to those skilled in the art.

[0057] The above model learning unit (11) can receive an image before image quality improvement as a first input and an image after image quality improvement as a second input to train the image quality improvement learning model. The image before image quality improvement may refer to an image frame extracted from the original video file. The image after image quality improvement may refer to an image frame extracted from the high-definition video file at the same time as an image extracted from the original video file.

[0058] The above model learning unit (11) can receive the image quality improvement learning model data from the learning control unit (14). The image data before and after image quality improvement can be input into the image quality improvement learning model to train the image quality improvement learning model.

[0059] The above model learning unit (11) can be implemented by a device such as a CPU (Central Processing Unit), a GPU (Graphic Processing Unit), a TPU (Tensor Processing Unit) and / or an NPU (Neural Processing Unit). Preferably, the model learning unit (11) can be implemented using a GPU, which is an example, and the device

[0060] is not limited by

[0061] The above frame extraction unit (12) can extract image frames from the original video file to generate original image frames, and can extract image frames corresponding to the original image frames from the high-definition video file to generate high-definition image frames. Preferably, the frame extraction unit (12) can extract original image frames and high-definition image frames according to the time order of the image frames in the original video file and the high-definition video file.

[0062] The above patch generation unit (13) can divide each of the original image frame and the high-quality image frame into areas of a preset size and generate each area as one image patch.

[0063] The learning control unit (14) can load the image quality improvement learning model into the model learning unit (11). The learning control unit (14) can input at least one original image patch generated from the original image frame as a first input to the model learning unit (11). The learning control unit (14) can input a high-quality image patch corresponding to each original image patch in the high-quality image frame as a second input to the model learning unit (11). The learning control unit (14) may correspond to a CPU device, which is an example and is not limited by the device.

[0064] Figure 2 is an example image of generating an image patch by extracting image frames from an original video file and a high-definition video file.

[0065] Referring to FIG. 2, the original video file may include a (natural number) number of image frames (100-1 to 100-a). The high-definition video file may include a number of image frames (200-1 to 200-a). The original video file and the high-definition video file may have the same number of image frames.

[0066] Additionally, the high-definition video file may be a video file with an increased frame rate as well as improved picture quality from the original video file.

[0067] In this case, the high-definition video file may have a relatively larger number of image frames than the original video file.

[0068] The first image frame (100-1) in the original video file may be an image frame corresponding to the first image frame (200-1) in the high-definition video file. The fact that the image frames correspond to each other may mean that the image frames are of the same time period in the original video file and the high-definition video file. For example, each of the first image frames (100-1, 200-1) may correspond to the first image frame in the original video file and the high-definition video file.

[0069] Alternatively, the original video file and the high-definition video file may include b (a natural number greater than a) image frames. The frame extraction unit (12) may extract a (a natural number greater than 1 and less than or equal to b) image frames at a preset frame interval from the b image frames. For example, the frame extraction unit (12) may extract a image frames at a 30-frame interval from the original video file and the high-definition video file. This is an example and is not limited by the frame interval.

[0070] Alternatively, the original video file and the high-definition video file may include b (a natural number) image frames. The frame extraction unit (12) may extract a (a natural number greater than or equal to 1 and less than or equal to bc) number of image frames excluding the first image frame to the cth (a natural number greater than or equal to 1 and less than or equal to b)-th image frame from the original video file and the high-definition video file. The first image frame to the cth image frame may correspond to an intro video. The image frame corresponding to the intro video may not correspond to learning data input to the image quality improvement learning model. The first image frame (100-1, 200-1) illustrated in FIG. 2 may correspond to the c+1th image frame from the original video file and the high-definition video file. In addition, the frame extraction unit (12) may also extract a' (a natural number greater than or equal to 1 and less than or equal to a) number of image frames excluding the image frame corresponding to the outro video from the original video file and the high-definition video file.

[0071] Alternatively, the original video file and the high-definition video file may include b (a natural number) image frames. The frame extraction unit (12) may extract d (a natural number greater than or equal to 1 and less than or equal to bc) image frames, excluding the first image frame to the c (a natural number greater than or equal to 1 and less than or equal to b) image frame, from the original video file and the high-definition video file. Thereafter, the frame extraction unit (12) may extract a (a natural number greater than or equal to 1 and less than or equal to d) image frames at a preset frame interval from the d image frames.

[0072] In this specification, the original video file and high-definition video file contain a number of image frames, and the image quality enhancement learning model is trained using all of the image frames. However, this is merely an example, and the image quality enhancement learning model is not necessarily trained using all of the image frames.

[0073] The above frame extraction unit (12) can extract the tth (a natural number greater than or equal to 1 and less than or equal to a) original image frame (110) from the original video file. The frame extraction unit (12) can extract the tth high-definition image frame (210) corresponding to the tth original image frame (110) from the high-definition video file.

[0074] The patch generation unit (13) can divide each of the original image frame (110) and the high-quality image frame (210) into regions of preset sizes. The patch generation unit (13) can generate original image patches (111 to 114) from the original image frame (110). The patch generation unit (13) can generate high-quality image patches (211 to 214) from the high-quality image frame (210). Although four image patches are illustrated in FIG. 2, this is only an example, and at least one image patch may be generated.

[0075] According to one embodiment of the present specification, the patch generation unit (13) can adjust the size of the image patch generated according to the memory size of the model learning unit (11).

[0076] For example, the model learning unit (11) may have a memory size corresponding to any one of 8 GB, 16 GB, and 32 GB.

[0077] When the memory size of the model learning unit (11) corresponds to 32 GB, the patch generation unit (13) can generate an image patch having a relatively largest size from the original image frame (110) and the high-definition image frame (210). This may mean that the maximum size of the image patch that the patch generation unit (13) can generate is relatively the largest. At this time, the patch generation unit (13) can generate one or more image patches from the original image frame (110) and the high-definition image frame (210).

[0078] When the memory size of the model learning unit (11) corresponds to 16 GB, the patch generation unit (13) can generate an image patch of a relatively smaller size than when the memory size of the model learning unit (11) is 32 GB. This may mean that the maximum size of the image patch that the patch generation unit (13) can generate is relatively smaller than when the memory size is 32 GB. At this time, the patch generation unit (13) can generate an image patch by dividing the original image frame (110) and the high-definition image frame (210) into two or more regions.

[0079] When the memory size of the above model learning unit (11) corresponds to 8 GB, the patch generation unit (13) can generate an image patch of a relatively smallest size.

[0080] This may mean that the maximum size of the image patch that the patch generation unit (13) can generate is relatively the smallest. At this time, the patch generation unit (13) may generate the image patch by dividing the original image frame (110) and the high-definition image frame (210) into four or more areas.

[0081] This is an example and is not limited by the memory size and number of image patches.

[0082] The above memory size may refer to the memory size of a GPU. Alternatively, the above memory size may refer to the memory of a storage medium widely known to those skilled in the art, such as RAM (Random Access Memory), HDD (Hard Disk Drive), SSD (Solid State Drive), etc.

[0083] According to one embodiment of the present specification, the original image frame (110) and the high-definition image frame (210) may have the same resolution. For example, the original image frame (110) and the high-definition image frame (210) may have a resolution of 1920x1080. When the patch generation unit (13) generates four original image patches (111 to 114) and four high-definition image patches (211 to 214), each image patch may have a resolution of 960x540. This is an example and is not limited by the resolution and the number of image patches.

[0084] According to another embodiment of the present specification, the original image frame (110) and the high-definition image frame (210) may have different resolutions. The resolution of the high-definition image frame (210) may be relatively greater than the resolution of the original image frame (110). For example, the original image frame (110) may have a resolution of 1920x1080. The high-definition image frame (210) may have a resolution of 3840x2160. The patch generation unit (13) may generate four original image patches (111 to 114) and four high-definition image patches (211 to 214), respectively. At this time, the four original image patches (111 to 114) may have a resolution of 960x540. The four high-definition image patches (211 to 214) may have a resolution of 1920x1080. This is an example and is not limited by the resolution and number of image patches.

[0085] Figure 3 is an example image of inputting learning data into a quality improvement learning model.

[0086] Referring to FIG. 3, the learning control unit (14) can load the image quality improvement learning model data stored in the storage unit (10) to the model learning unit (11). The learning control unit (14) can input a pair of corresponding image patches from among four original image patches (111 to 114) and four high-quality image patches (211 to 214) to the model learning unit (11). For example, the model learning unit (11) can select corresponding image patches using the coordinate values ​​of the original image patches and / or high-quality image patches and input the selected corresponding image patches to the model learning unit (11). This is merely an example, and various techniques widely known to those skilled in the art can be used to select image patches located in corresponding areas in the original image frame and the high-quality image frame.

[0087] Among the above four original image patches (111 to 114) and high-quality image patches (211 to 214), the first original image patch (111) may correspond to the first high-quality image patch (211), the second original image patch (112) may correspond to the second high-quality image patch (212), the third original image patch (113) may correspond to the third high-quality image patch (213), and the fourth original image patch (114) may correspond to the fourth high-quality image patch (214).

[0088] The learning control unit (14) can input the first original image patch (111) as a first input to the model learning unit (11). In addition, the learning control unit (14) can input the first high-definition image patch (211) as a second input to the model learning unit (11). The model learning unit (11) can input the first original image patch (111) and the first high-definition image patch (211) to the image quality improvement learning model. The image quality improvement learning model can learn the data of the first original image patch (111) and the first high-definition image patch (211). In this manner, the learning control unit (14) can input all image patches included in the original image frame and the high-definition image frame as a pair with a corresponding image patch to the model learning unit (11).

[0089] FIG. 4 is a block diagram of a device for learning a picture quality improvement model according to another embodiment of the present specification.

[0090] Referring to FIG. 4, a device (1') for learning a picture quality improvement model according to another embodiment of the present specification may include a storage unit (10), a model learning unit (11), a frame extraction unit (12), a patch generation unit (13), a learning control unit (14), and a learning data selection unit (15). Since the storage unit (10), the model learning unit (11), the frame extraction unit (12), the patch generation unit (13), and the learning control unit (14) have been described above, a repetitive description thereof will be omitted.

[0091] The above learning data selection unit (15) can select the target image patch and the image patch corresponding thereto as learning data based on the amount of change between the target image patch generated from either the original image frame or the high-definition image frame and the preceding image patch generated from a frame preceding the frame in which the target image patch is generated in the video file.

[0092] The above learning control unit (14) can input the selected target image patch and the corresponding image patch to the model learning unit (11).

[0093] Figure 5 is an example image of a learning data selection unit selecting learning data.

[0094] Referring to FIG. 5, the target image patch (300) may correspond to any one of at least one image patch generated from an original image frame. The original image frame may correspond to the t-th original image frame among a number of original image frames. The learning data selection unit (15) may compare the target image patch (300) with a preceding image patch. The preceding image patch may refer to an image patch located in the same area as the target image patch (300) in the t-1-th original image frame.

[0095] In addition, the preceding image patch may mean an image patch located in the same area as the target image patch (300) in the tu (u is a natural number greater than or equal to 1 and less than t)th original image frame. For example, when the image patch in the t-1th original image frame is not selected as learning data, the learning data selection unit (15) may calculate the amount of change between the target image patch (300) and the image patch corresponding to the target image patch (300) in the t-2th original image frame.

[0096] The above learning data selection unit (15) can calculate the amount of change between at least one target image patch included in the t-th original image frame and the preceding image patch corresponding to each other. Thereafter, the learning data selection unit (15) can calculate the amount of change between at least one target image patch included in the t+1-th original image frame and the preceding image patch corresponding to each other.

[0097] According to one embodiment of the present specification, when the first target image patch (310) is an image patch in the first (t=1) original image frame, the learning data selection unit (15) can select the first target image patch (310) and the corresponding image patch in the high-quality image frame as learning data.

[0098] Thereafter, the learning data selection unit (15) can calculate the amount of change between the second target image patch (320) and the first target image patch (310) in the second (t=2) original image frame. The amount of change between the first target image patch (310) and the second target image patch (320) may be less than or equal to a preset reference value. The learning data selection unit (15) may not select the second target image patch (320) as learning data.

[0099] Thereafter, the learning data selection unit (15) can calculate the amount of change between the third target image patch (330) and the first target image patch (310) in the third (t=3) original image frame. At this time, the amount of change between the first target image patch (310) and the third target image patch (330) may be greater than or equal to a preset reference value. The amount of change in the area occupied by an arbitrary object in the third target image patch (330) may be greater than or equal to a preset reference value. The learning data selection unit (15) can select the third target image patch (330) as learning data. In addition, the learning data selection unit (15) can select an image patch corresponding to the third target image patch (330) in the third high-definition image frame as learning data.

[0100] Thereafter, the learning data selection unit (15) can calculate the amount of change between the fourth target image patch (340) and the third target image patch (330) in the fourth (t=4) original image frame. At this time, the amount of change between the third target image patch (330) and the fourth target image patch (340) may be greater than or equal to a preset reference value. The amount of change in the position of any object in the third target image patch (330) and the fourth target image patch (340) may be greater than or equal to a preset reference value. The learning data selection unit (15) can select the fourth target image patch (340) as learning data. In addition, the learning data selection unit (15) can select an image patch corresponding to the fourth target image patch (340) in the fourth high-definition image frame as learning data.

[0101] Thereafter, the learning data selection unit (15) can calculate the amount of change between the fifth target image patch (350) and the fourth target image patch (340) in the fifth (t=5) original image frame. At this time, the amount of change between the fourth target image patch (340) and the fifth target image patch (350) may be greater than or equal to a preset reference value. The amount of change in the area and / or position of any object in the fourth target image patch (340) and the fifth target image patch (350) may be greater than or equal to a preset reference value. The learning data selection unit (15) can select the fifth target image patch (350) as learning data. In addition, the learning data selection unit (15) can select an image patch corresponding to the fifth target image patch (350) in the fifth high-definition image frame as learning data.

[0102] Thereafter, the learning data selection unit (15) can calculate the amount of change between the sixth target image patch (360) and the fifth target image patch (350) in the sixth (t=6) original image frame. At this time, the amount of change between the fifth target image patch (350) and the sixth target image patch (360) may be greater than or equal to a preset reference value. The amount of change in the area of ​​an arbitrary object in the fifth target image patch (350) and the sixth target image patch (360) may be greater than or equal to a preset reference value. The learning data selection unit (15) can select the sixth target image patch (360) as learning data. In addition, the learning data selection unit (15) can select an image patch corresponding to the sixth target image patch (360) in the sixth high-definition image frame as learning data.

[0103] Thereafter, the learning data selection unit (15) can calculate the amount of change between the seventh target image patch (370) and the sixth target image patch (360) in the seventh (t=7) original image frame. At this time, the amount of change between the sixth target image patch (360) and the seventh target image patch (370) may be greater than or equal to a preset reference value. The amount of change in the area of ​​an arbitrary object in the sixth target image patch (360) and the seventh target image patch (370) may be less than or equal to a preset reference value. The learning data selection unit (15) may not select the seventh target image patch (370) as learning data.

[0104] The above learning data selection unit (15) can select learning data by calculating the amount of change from the preceding image patch from the first target image patch to the a-th target image patch. The learning data selection unit (15) can delete target image patches and / or corresponding image patches that are not selected as learning data from the memory. This is an example and is not limited by the above method.

[0105] In cases where a foreground object moves in front of a region of a static background, the imbalance in the learning data can be corrected by distinguishing between image patches with relatively large and relatively small changes over time in the same manner as above.

[0106] For example, in a 1-hour lecture video at 60FPS, each image frame can be divided into 10 image patches for training. The total number of frames in this video can be 216,000 (60 x 3,600 seconds). In the video, the instructor can be positioned at a point corresponding to the image patch area of ​​any one of the 10 image patches against a static blackboard background for most of the lecture time. At this time, the image patches corresponding to the area where the instructor moves can be a total of 216,000 (for convenience, the effect of the instructor crossing two or more image patches depending on the movement is excluded). The image patch where the instructor is positioned can change relatively more than other image frames depending on the instructor's movement. The 1,994,000 (9 x 216,000) image patches where the instructor is not positioned can contain images with the same noise characteristics. If an AI model trains on image patches where the instructor is not present, it could end up learning 1,944,000 times more from the same image patches than from patches where the instructor is present. For AI models, this could result in some patches diluting the effectiveness of learning from diverse image patches. This could mean the AI ​​model overfits to image patches with relatively little variation.

[0107] In the above example, if an average of 1 / 9 of the entire background image patches where the instructor is not positioned is selected as learning data, the ratio of the number of image patches where the instructor is positioned and the number of background image patches selected as learning data can be adjusted from 1:9 to 1:1. As a result, the artificial intelligence model can learn the foreground and background with different image and noise characteristics in a balanced manner. The artificial intelligence model can learn the image patches with relatively large changes and the image patches with relatively small changes in a balanced manner. This is an example and is not limited by the number and ratio of the image patches.

[0108] Training data constructed using conventional methods can contain this imbalance, potentially leading to imbalances between the foreground and background. However, the aforementioned method reduces the number of repetitions of background training data, making the training data more balanced than before.

[0109] According to another embodiment of the present specification, the learning data selection unit (15) can calculate the amount of change for pixel values ​​that correspond to each other in the target image patch and the preceding image patch. The corresponding pixel values ​​may refer to pixel values ​​of the same coordinates in the target image patch and the preceding image patch. The learning data selection unit (15) can calculate the sum or average value of the amount of change for all pixel values ​​that correspond to each other in the target image patch and the preceding image patch. If the sum or average of the amount of change is greater than or equal to a preset reference value, the learning data selection unit (15) can select the corresponding target image patch as learning data. In addition, the learning data selection unit (15) can select an image patch corresponding to the corresponding target image patch as learning data.

[0110] For example, if the first target image patch (310) is an image patch in the first (t=1) original image frame, the learning data selection unit (15) can select the first target image patch (310) and the corresponding image patch in the high-quality image frame as learning data.

[0111] Thereafter, the learning data selection unit (15) can calculate the amount of change between the second target image patch (320) and the first target image patch (310) in the second (t=2) original image frame. The learning data selection unit (15) can calculate the sum and / or average of the amounts of change for pixel values ​​corresponding to each other in the first target image patch (310) and the second target image patch (320). The sum and / or average of the amounts of change in the first target image patch (310) and the second target image patch (320) may be less than or equal to a preset reference value. The learning data selection unit (15) may not select the second target image patch (320) as learning data.

[0112] Thereafter, the learning data selection unit (15) can calculate the amount of change between the third target image patch (330) and the first target image patch (310) in the third (t=3) original image frame. The learning data selection unit (15) can calculate the sum and / or average of the amounts of change for the pixel values ​​corresponding to each other in the first target image patch (310) and the third target image patch (330). The third target image patch (330) may include an arbitrary object. Due to the arbitrary object, the sum and / or average of the amounts of change in the first target image patch (310) and the third target image patch (330) may be greater than or equal to a preset reference value. The learning data selection unit (15) can select the third target image patch (330) as learning data. In addition, the learning data selection unit (15) can select an image patch corresponding to the third target image patch (330) from the third high-definition image frame as learning data.

[0113] Thereafter, the learning data selection unit (15) can calculate the amount of change between the fourth target image patch (340) and the third target image patch (330) in the fourth (t=4) original image frame. The learning data selection unit (15) can calculate the sum and / or average of the amounts of change for corresponding pixel values ​​in the third target image patch (330) and the fourth target image patch (340). The position of the arbitrary object in the fourth target image patch (340) may be different from that in the third target image patch (330). Therefore, the sum and / or average of the amounts of change in the third target image patch (330) and the fourth target image patch (340) may be greater than or equal to a preset reference value. The learning data selection unit (15) can select the fourth target image patch (340) as learning data. In addition, the learning data selection unit (15) can select an image patch corresponding to the fourth target image patch (340) from the fourth high-definition image frame as learning data. The learning data selection unit (15) can select learning data by calculating the amount of change from the preceding image patches from the first target image patch to the a-th target image patch. The learning data selection unit (15) can delete target image patches and / or image patches corresponding thereto that are not selected as learning data from the memory. This is an example and is not limited by the above method. Preferably, the learning data selection unit (15) can calculate the root mean square of the amount of change calculated by [Mathematical Formula 1] below in the preceding image patch (X) and the target image patch (Y) and / or the mean absolute error of the amount of change calculated by [Mathematical Formula 2].

[0114]

[0115] L: Horizontal resolution of the image patch

[0116] M: Vertical resolution of the image patch

[0117] i=1,2,3,...,L

[0118] j=1,2,3,...,M

[0119] k: 1 (Red pixel value), 2 (Green pixel value), 3 (Blue pixel value)

[0120]

[0121] L: Horizontal resolution of the image patch

[0122] M: Vertical resolution of the image patch

[0123] i=1,2,3,...,L

[0124] j=1,2,3,...,M

[0125] k: 1 (Red pixel value), 2 (Green pixel value), 3 (Blue pixel value)

[0126] The above learning data selection unit (15) is the Diff RMS (X,Y) and / or Diff MAE If the value derived from (X,Y) is greater than a preset reference value, the target image patch can be selected as learning data.

[0127] Figure 6 is another example image of a learning data selection unit selecting learning data.

[0128] Referring to FIG. 6, the learning data selection unit (15) can divide each of the target image patch and the preceding image patch into a plurality of target image sub-patches and preceding image sub-patches. For example, the learning data selection unit (15) can divide the third target image patch (330) and the fourth target image patch (340) into a plurality of sub-patches. The third target image patch (330) may correspond to a preceding image patch of the fourth target image patch (340). The learning data selection unit (15) can divide the fourth target image patch (340) into four target image sub-patches (341 to 344). The learning data selection unit (15) can divide the third target image patch (330) into four preceding image sub-patches (331 to 334).

[0129] The first target image sub-patch (341) may correspond to the first preceding image sub-patch (331), the second target image sub-patch (342) may correspond to the second preceding image sub-patch (332), the third target image sub-patch (343) may correspond to the third preceding image sub-patch (333), and the fourth target image sub-patch (344) may correspond to the fourth preceding image sub-patch (334).

[0130] The above learning data selection unit (15) can calculate the sum of the changes in pixel values ​​in the sub-patches that correspond to each other. The learning data selection unit (15) can calculate the sum of the changes in the pixel values ​​that correspond to each other in the first target image sub-patch (341) and the first preceding image sub-patch (331) as a first output value. The learning data selection unit (15) can calculate the sum of the changes in the pixel values ​​that correspond to each other in the second target image sub-patch (342) and the second preceding image sub-patch (332) as a second output value. The learning data selection unit (15) can calculate the sum of the changes in the pixel values ​​that correspond to each other in the third target image sub-patch (343) and the third preceding image sub-patch (333) as a third output value. The learning data selection unit (15) can calculate the sum of the changes in the corresponding pixel values ​​in the fourth target image sub-patch (344) and the fourth preceding image sub-patch (334) as the fourth output value. Thereafter, the learning data selection unit (15) can calculate the average value of the first to fourth output values. If the average value is greater than or equal to a preset reference value, the fourth target image patch (340) can be selected as learning data. In addition, the learning data selection unit (15) can select an image patch corresponding to the fourth target image patch (340) in the high-quality image frame as learning data. The learning data selection unit (15) can calculate a plurality of n-th output values ​​(subDiff(X)) from the preceding image patch (X) and the target image patch (Y). n , Y n )) for the mean (Diff sub (X, Y) can be calculated using [Mathematical Formula 3] below.

[0131]

[0132] n: natural number (number of subpatches)

[0133] For example, the upper limit may be set to '1', and the lower limit may be set to '0'. The first target image sub-patch (341) and the first preceding image sub-patch (331) may not include any object. In addition, the second target image sub-patch (342) and the second preceding image sub-patch (332) may not include any object. In this case, the first output value and the second output value may be less than or equal to a preset reference value. The learning data selection unit (15) may set the first output value and the second output value to have a value greater than or equal to '0'. At this time, the learning data selection unit (15) may set the first output value and the second output value to have a value close to '0'. A value close to '0' may mean a value less than 0.5. Preferably, the learning data selection unit (15) can set the first output value and the second output value to '0'.

[0134] The positions of any object in the third target image patch (330) and the fourth target image patch (340) may be different. Any object in the third target image patch (330) may occupy a relatively wider area in the third preceding image sub-patch (333) than in the fourth preceding image sub-patch (334). In addition, any object in the fourth target image patch (340) may occupy a relatively wider area in the fourth target image sub-patch (344) than in the third target image sub-patch (343). Therefore, the third output value and the fourth output value may be greater than or equal to a preset reference value. The learning data selection unit (15) may set the third output value and the fourth output value to have a value less than or equal to '1'. In this case, the learning data selection unit (15) may set the third output value and the fourth output value to have a value close to '1'. A value close to '1' may mean a value greater than or equal to 0.5. Preferably, the learning data selection unit (15) may set the third output value and the fourth output value to be '1'.

[0135] By setting an upper limit on the first to fourth output values, the learning data selection unit (15) can select image patches in which the pixel value difference is large only in a very small area between the target image patch and the preceding target image patch and in which the pixel values ​​are similar in the remaining areas as learning data.

[0136] The above learning data selection unit (15) selects multiple n-th output values ​​(subDiff(X n , Y n )) can be calculated using [Mathematical Formula 4] and / or [Mathematical Formula 5] below.

[0137]

[0138] I: Horizontal resolution of sub-patch

[0139] J: Vertical resolution of sub-patch

[0140] i=1,2,3,..., I

[0141] j=1,2,3,..., J

[0142] k: 1 (Red pixel value), 2 (Green pixel value), 3 (Blue pixel value)

[0143]

[0144] I: Horizontal resolution of sub-patch

[0145] J: Vertical resolution of sub-patch

[0146] i=1,2,3,..., I

[0147] j=1,2,3,..., J

[0148] k: 1 (Red pixel value), 2 (Green pixel value), 3 (Blue pixel value)

[0149] In this specification, the image patch is divided into four sub-patches, but this is only an example and is not limited by the above number. In addition, the upper limit is set to '1' and the lower limit is set to '0', but this is only an example and is not limited by the above upper and lower limits.

[0150] The above learning data selection unit (15) can select an image patch included in the first image frame from a number of original image frames and high-quality image frames as learning data. The learning data selection unit (15) can divide each image patch from a number of original image frames and high-quality image frames into n sub-patches. Thereafter, the learning data selection unit (15) can select the subDiff RMS (X n , Y n ) is calculated using the first Diff sub (X,Y) and / or the subDiff above MAE (X n ,Y n) is calculated using the second Diff sub Using (X,Y), learning data can be selected from image patches included in a-1 original image frames and high-quality image frames excluding the first image frame. The learning data selection unit (15) can delete target image patches and / or corresponding image patches that are not selected as learning data from the memory. The learning data selection unit (15) can select the Diff MAE When selecting learning data using the values ​​of (X,Y), the above Diff RMS It is possible to select learning data relatively more precisely than selecting learning data using (X,Y). For example, the first target image patch (310) and the second target image patch (320) may correspond to image patches in which only the background is captured. The first target image patch (310) and the second target image patch (320) may have a variation amount less than or equal to a preset reference value. In this case, the learning data selection unit (15) may not select the second target image patch (320) as learning data.

[0151] However, some pixel values ​​may differ between the first target image patch (310) and the second target image patch (320) due to noise caused by changes in lighting during shooting, image compression, etc. For this reason, although the first target image patch (310) and the second target image patch (320) are image patches in which the same background is shot, the amount of change may be calculated to be greater than a preset reference value due to the difference in pixel values. For this reason, the learning data selection unit (15) may select the second target image patch as learning data.

[0152] The above learning data selection unit (15) is the Diff MAE (X,Y) and / or the above Diff RMSThe target image patch can be prevented from being selected as learning data due to the difference in pixel values ​​in the preceding image patch and the target image patch in which the same image was captured through the values ​​calculated as (X,Y). At this time, the learning data selection unit (15) is configured to select the Diff MAE When calculating values ​​using (X,Y), the above Diff RMS It is possible to more effectively prevent the selection of a target image patch other than learning data as learning data due to the difference in the pixel values ​​than when calculating values ​​using (X,Y). In addition, the learning data selection unit (15) is configured to select the Diff MAE (X,Y) and / or Diff RMS In the case of calculating the value using (X,Y), the first Diff sub (X,Y) and / or 2nd Diff sub When calculating values ​​using (X,Y), it is possible to more effectively prevent the selection of a target image patch other than learning data as learning data due to the difference in the pixel values. At this time, the learning data selection unit (15) is the second Diff sub When calculating values ​​using (X,Y), the first Diff sub It is possible to more effectively prevent the selection of a target image patch, rather than learning data, as learning data due to the difference in the pixel values ​​than when calculating values ​​using (X,Y). The learning control unit (14) can input the target image patch selected by the learning data selection unit (15) and the corresponding image patch to the model learning unit (11). The model learning unit (11) can input a pair of input image patches to the image quality improvement learning model.

[0153] In FIGS. 5 and 6 of this specification, the learning data selection unit (15) is illustrated as selecting learning data using a target image patch and a preceding image patch generated from an original image frame, but this is only an example, and the learning data selection unit (15) may also select learning data using a target image patch and a preceding image patch generated from a high-definition image frame.

[0154] The above learning model unit (11) can store a quality improvement model that has learned at least one original video file and a high-definition video file in the storage unit (10). Alternatively, the learning model unit (11) can update the quality improvement learning model data stored in the storage unit (10) with quality improvement model data that has learned at least one original video file and a high-definition video file. The quality improvement model can generate a high-definition video file with improved quality by using at least one of noise removal, resolution enhancement, and color correction from the original video file.

[0155] The above model learning unit (11), frame extraction unit (12), patch generation unit (13), learning control unit (14), and learning data selection unit (15) may include a processor, ASIC (application-specific integrated circuit), other chipset, logic circuit, register, communication modem, data processing device, etc. known in the technical field to which the present invention pertains in order to execute calculation and various control logic. In addition, when the above-described control logic is implemented in software, the model learning unit (11), frame extraction unit (12), patch generation unit (13), learning control unit (14), and learning data selection unit (15) may be implemented as a set of program modules. At this time, the program modules may be stored in the memory device and executed by the processor.

[0156] Hereinafter, a method for learning an image quality improvement model using a device (1, 1') for learning an image quality improvement model according to the present specification is disclosed. However, in explaining the method for learning an image quality improvement model according to the present specification, repetitive descriptions of each component are omitted.

[0157] Figure 7 is a flowchart of a method for learning a quality improvement model according to one embodiment of the present specification.

[0158] Referring to FIG. 7, in step S10, the processor may extract the t-th image frame from an original video file including a (a natural number) number of original image frames stored in the storage unit (10) to generate an original image frame. The processor may extract the image frame corresponding to the t-th (a natural number greater than or equal to 1 and less than or equal to a) number of original image frames from a high-definition video file including a number of high-definition image frames to generate a high-definition image frame. The high-definition video file may be a file whose image quality is improved through at least one of noise removal, resolution enhancement, and color correction from the original video file. Preferably, the processor may extract the image frames in time order from the a number of image frames included in the original video file and the high-definition video file.

[0159] In step S11, the processor can divide the t-th original image frame and the t-th high-definition image frame into regions of preset sizes and generate each region as one image patch. At this time, the processor can adjust the size of the generated image patch according to the memory size of the model learning unit (11). The maximum size of the image patch can be relatively larger when the memory size of the model learning unit (11) is large than when the memory size is small.

[0160] The original image frame and the high-definition image frame may have the same resolution. In this case, the original image patch and the high-definition image patch may have the same resolution. The original image patch and the high-definition image patch may have the same image size.

[0161] Additionally, the high-definition image frame may have a relatively larger resolution than the original image frame. In this case, the high-definition image patch may have a relatively larger resolution than the original image patch. The high-definition image patch may have a relatively larger image size than the original image patch.

[0162] In step S12, the processor can load the image quality improvement learning model stored in the storage unit (10) into the model learning unit (11). In addition, the processor can input at least one original image patch generated in step S11 and a high-quality image patch corresponding to each original image patch into the model learning unit (11).

[0163] The above model learning unit (11) can input the input original image patch and the high-quality image patch corresponding to the original image patch into the image quality improvement learning model and train it.

[0164] The above processor can input all image patches generated from a number of original image frames and high-quality image frames into the model learning unit (11).

[0165] FIG. 8 is a flowchart of a method for learning a quality improvement model according to another embodiment of the present specification.

[0166] Referring to Fig. 8, steps S20 and S21 are identical to steps S10 and S11, and therefore, a repetitive description is omitted.

[0167] In step S22, the processor may select the target image patch and the image patch corresponding thereto as learning data based on the amount of change between the target image patch generated from either the original image frame or the high-definition image frame and the preceding image patch generated from a frame preceding the frame in which the target image patch was generated in the video file. When the target image patch is an image patch of the first original image frame or the first high-definition image frame, the processor may select the target image patch and the image patch corresponding thereto as learning data.

[0168] The processor may calculate a change amount between the target image patch and the preceding image patch. If the calculated change amount is greater than a preset reference value, the processor may select the target image patch and the corresponding image patch as learning data. For example, the change amount may correspond to a change amount in an area occupied by an arbitrary object included in the target image patch and the preceding image patch and / or a change amount in a position of an arbitrary object, which is merely an example and is not limited by the change amount.

[0169] Preferably, the processor can calculate a sum of changes and / or an average of changes for pixel values ​​corresponding to each other in the target image patch and the preceding image patch. If the sum of changes and / or an average of changes for the pixel values ​​is greater than or equal to a preset reference value, the processor can select the target image patch and the image patch corresponding thereto as learning data. Preferably, the processor can select learning data by calculating the root mean square of the change calculated by [Mathematical Formula 1] and / or the average absolute error of the change calculated by [Mathematical Formula 2].

[0170] Alternatively, the processor may divide each of the target image patch and the preceding image patch into a plurality of target image sub-patches and preceding image sub-patches. The processor may calculate a sum of changes in corresponding pixel values ​​in each of the target image sub-patches and the preceding image sub-patches. If the average of the calculated sum of changes is greater than or equal to a preset reference value, the processor may select the target image patch and the corresponding image patch as learning data. Preferably, the processor may calculate an average of the calculated sum of changes using [Mathematical Formula 3].

[0171] At this time, the processor may set the sum of the changes in the corresponding pixel values ​​in the target image sub-patch and the preceding image sub-patch to be equal to or less than a preset upper limit value when the sum of the changes is equal to or greater than a preset reference value. The processor may set the sum of the changes to have a value close to the upper limit value. Preferably, the processor may set the sum of the changes to have the upper limit value.

[0172] In addition, the processor may set the sum of the changes to be equal to or greater than a preset lower limit when the sum of the changes is less than or equal to a preset reference value. The processor may set the sum of the changes to have a value close to the lower limit. Preferably, the processor may set the sum of the changes to have the lower limit. The processor may set the first Diff sub (X,Y) and / or 2nd Diff subIf the value calculated using (X,Y) is greater than or equal to a preset reference value, the target image patch and the corresponding image patch can be selected as learning data. In step S22, the processor can delete the target image patch and / or the corresponding image patch that are not selected as learning data from the memory.

[0173] In step S23, the processor can load the image quality improvement learning model stored in the storage unit (10) into the model learning unit (11). In addition, the processor can input at least one target image patch selected in step S22 and an image patch corresponding to each target image patch into the model learning unit (11).

[0174] The above model learning unit (11) can input the target image patch and the corresponding image patch to the image quality improvement learning model.

[0175] The processor can select learning data by comparing a target image patch generated from a-1 image frames excluding the first original image frame and the high-quality image frame among a number of original image frames or high-quality image frames with a preceding image patch. The processor can input the selected learning data to the model learning unit (11).

[0176] The image quality improvement model learning method according to the present specification can be implemented in the form of a computer program written to perform each step on a computer and recorded on a computer-readable recording medium. The aforementioned computer program may include code coded in a computer language, such as C / C++, C#, JAVA, Python, or machine language, that can be read by the processor (CPU) of the computer through the device interface of the computer, so that the computer reads the program and executes the methods implemented as a program. Such code may include functional code related to functions that define the functions necessary to execute the methods, and may include control code related to execution procedures necessary for the processor of the computer to execute the functions according to a predetermined procedure. In addition, such code may further include memory reference-related code regarding which location (address address) in the internal or external memory of the computer should reference additional information or media necessary for the processor of the computer to execute the functions. In addition, if the processor of the computer needs to communicate with any other computer or server located remotely in order to execute the functions, the code may further include communication-related code regarding how to communicate with any other computer or server located remotely using the communication module of the computer, and what information or media to send and receive during communication.

[0177] The above storage medium refers to a medium that stores data semi-permanently and can be read by a device, rather than a medium that stores data for a short period of time, such as a register, cache, or memory. Specifically, examples of the storage medium include, but are not limited to, ROM, RAM, CD-ROM, magnetic tape, floppy disk, and optical data storage device. That is, the program can be stored in various recording media on various servers that the computer can access or in various recording media on the user's computer. In addition, the medium can be distributed across network-connected computer systems, so that computer-readable code can be stored in a distributed manner.

[0178] While the embodiments of this specification have been described with reference to the attached drawings, those skilled in the art will appreciate that the present invention can be implemented in other specific forms without altering the technical spirit or essential features thereof. Therefore, the embodiments described above should be understood to be illustrative in all respects and not restrictive.

Claims

1. A storage unit that stores an original video file, a high-definition video file with improved image quality of the original video, and image quality improvement learning model data that improves the image quality; A model learning unit that receives an image before image quality improvement as a first input and an image after image quality improvement as a second input and trains the image quality improvement learning model; A frame extraction unit that extracts an image frame from the original video file to generate an original image frame, and extracts an image frame corresponding to the original image frame from the high-definition video file to generate a high-definition image frame; A patch generation unit for dividing each of the original image frame and the high-quality image frame into areas of a preset size and generating each area as one image patch; and An image quality improvement model learning device, comprising: a learning control unit that loads the image quality improvement learning model into the model learning unit, inputs an original image patch generated from the original image frame as a first input into the model learning unit, and inputs a high-quality image patch corresponding to the original image patch from the high-quality image frame as a second input into the model learning unit.

2. In claim 1, A device for learning a picture quality improvement model, characterized in that the picture quality of the high-definition video file is improved through at least one of noise removal, resolution enhancement, and color correction from the original video file.

3. In claim 1, The above patch generation unit, An image quality improvement model learning device that adjusts the size of image patches generated according to the memory size of the above model learning unit.

4. In claim 1, A device for learning an image quality improvement model, characterized in that the size of the original image patch and the size of the high-quality image patch are different from each other.

5. In claim 1, A device for learning a picture quality improvement model, further comprising a learning data selection unit that selects the target image patch and the corresponding image patch as learning data based on the amount of change between the target image patch generated from either the original image frame or the high-quality image frame and the preceding image patch generated from a frame preceding the frame in which the target image patch was generated in the video file.

6. In claim 5, The above learning data selection unit is, An image quality improvement model learning device that calculates the amount of change between the target image patch and the preceding image patch, and selects the target image patch and the corresponding image patch as learning data if the calculated amount of change is greater than a preset reference value.

7. In claim 5, The above learning data selection unit is, An image quality improvement model learning device that selects the target image patch and the corresponding image patch as learning data when the sum or average of the changes in the corresponding pixel values ​​in the target image patch and the preceding image patch is greater than a preset reference value.

8. In claim 5, The above learning data selection unit is, An image quality improvement model learning device that divides each of the target image patch and the preceding image patch into a plurality of target image sub-patches and preceding image sub-patches, calculates the sum of the changes in the pixel values ​​corresponding to each other in each of the target image sub-patches and the preceding image sub-patches, and selects the target image patch and the corresponding image patch as learning data when the average of the sum of the calculated changes is greater than or equal to a preset reference value.

9. In claim 8, The above learning data selection unit is, If the sum of the changes in the corresponding pixel values ​​in the target image sub-patch and the preceding image sub-patch is greater than or equal to a preset reference value, the sum of the changes is set to be less than or equal to a preset upper limit, and if the sum of the changes is less than or equal to the preset reference value, the sum of the changes is set to be greater than or equal to a preset lower limit.

10. In claim 5, The above learning control unit, Image quality improvement model learning by inputting the target image patch selected from the above learning data selection unit and the corresponding image patch to the model learning unit.

11. A storage unit for storing an original video file, a high-quality video file with improved quality of the original video, and quality improvement learning model data for improving the quality of the image; and A device for learning a quality improvement model, comprising: a model learning unit that receives an image before quality improvement as a first input, an image after quality improvement as a second input, and learns the quality improvement learning model; A frame extraction step in which a processor extracts an image frame from the original video file to generate an original image frame, and extracts an image frame corresponding to the original image frame from a high-definition video file in which the image quality of the original video is improved to generate a high-definition image frame; A patch generation step in which the processor divides each of the original image frame and the high-quality image frame into regions of a preset size and generates each region as one image patch; and A method for learning an image quality improvement model, comprising: a learning control step in which a processor loads the image quality improvement learning model into the model learning unit, inputs an original image patch generated from the original image frame as a first input into the model learning unit, and inputs a high-quality image patch corresponding to the original image patch from the high-quality image frame as a second input into the model learning unit.

12. In claim 11, A method for learning a quality improvement model, characterized in that the quality of the high-definition video file is improved through at least one of noise removal, resolution enhancement, and color correction from the original video file.

13. In claim 11, The above patch creation step is: A method for learning an image quality improvement model, wherein the above processor is a step of adjusting the size of an image patch generated according to the memory size of the model learning unit.

14. In claim 11, A method for learning an image quality improvement model, characterized in that the size of the original image patch and the size of the high-quality image patch are different from each other.

15. In claim 11, After the above patch generation step, A method for learning an image quality improvement model, further comprising a learning data selection step in which the processor selects the target image patch and the image patch corresponding thereto as learning data based on the amount of change between the target image patch generated from either the original image frame or the high-quality image frame and the preceding image patch generated from a frame preceding the frame in which the target image patch was generated in the video file.

16. In claim 15, The above learning data selection step is, A method for learning an image quality improvement model, wherein the processor calculates a change amount between the target image patch and the preceding image patch, and if the calculated change amount is greater than a preset reference value, selects the target image patch and the corresponding image patch as learning data.

17. In claim 15, The above learning data selection step is, A method for learning an image quality improvement model, wherein the processor selects the target image patch and the corresponding image patch as learning data when the sum or average of the changes in the corresponding pixel values ​​in the target image patch and the preceding image patch is greater than or equal to a preset reference value.

18. In claim 15, The above learning data selection step is, A method for learning an image quality improvement model, wherein the processor divides each of the target image patch and the preceding image patch into a plurality of target image sub-patches and preceding image sub-patches, calculates a sum of changes in pixel values ​​corresponding to each other in each of the target image sub-patches and the preceding image sub-patches, and selects the target image patch and the corresponding image patch as learning data when the average of the sum of the calculated changes is greater than or equal to a preset reference value.

19. In claim 18, The above learning data selection step is, A method for learning an image quality improvement model, wherein the processor sets the sum of the changes in the corresponding pixel values ​​in the target image sub-patch and the preceding image sub-patch to be equal to or less than a preset upper limit when the sum of the changes is equal to or greater than a preset reference value, and sets the sum of the changes to be equal to or greater than a preset lower limit when the sum of the changes is equal to or less than the preset reference value.

20. In claim 15, The above learning steps are: A method for learning an image quality improvement model, wherein the processor is a step of inputting a target image patch selected from the learning data selection unit and an image patch corresponding thereto into the image quality improvement learning model.

Images