Method and apparatus for performing video enhancement

Abstract

A method and apparatus for performing image enhancement are disclosed. The method includes identifying a first image including an abnormal phenomenon among a plurality of images included in a first video, acquiring guide information corresponding to the abnormal phenomenon among a plurality of types of guide information, acquiring a second image to replace the first image based on the acquired guide information, and acquiring a second video in which the abnormal phenomenon has been removed by using the second image.

Description

Technical Field

[0001] This disclosure relates to a method and apparatus for performing video enhancement. Background Technology

[0002] Electronic devices equipped with generative artificial intelligence (AI) can generate content requested by the user. Generative AI can generate similar content using existing content. It can learn content patterns and generate new content as a perturbation.

[0003] Recently, various types of electronic devices have been developed that utilize neural network models to perform visual tasks. Furthermore, electronic devices equipped with generative artificial intelligence can reproduce images based on user input conditions, or generate videos from user-input text or sentences. Summary of the Invention

[0004] Solution to the problem

[0005] According to embodiments of this disclosure, a method for performing video enhancement is provided. The method includes identifying a first image from a plurality of images included in a first video, wherein the first image includes an anomaly. Additionally, the method includes obtaining guidance information corresponding to the anomaly from various types of guidance information. Furthermore, the method includes obtaining a second image based on the guidance information to replace the first image. Finally, the method includes obtaining a second video with the anomaly removed by using the second image.

[0006] According to embodiments of this disclosure, an apparatus for performing video enhancement is provided. The apparatus for performing video enhancement includes at least one memory storing one or more instructions; and at least one processor configured to execute the one or more instructions. When executed by the at least one processor, the one or more instructions are configured to cause the apparatus to identify a first image from a plurality of images included in a first video, wherein the first image includes an anomaly. Furthermore, when executed by the at least one processor, the one or more instructions are configured to cause the apparatus to obtain guidance information corresponding to the anomaly from various types of guidance information. Additionally, when executed by the at least one processor, the one or more instructions are configured to cause the apparatus to obtain a second image based on the guidance information to replace the first image. Furthermore, when executed by the at least one processor, the one or more instructions are configured to cause the apparatus to obtain a second video in which the anomaly has been removed using the second image.

[0007] According to embodiments of this disclosure, a computer-readable medium having instructions stored therein is provided, which, when executed by at least one processor, cause at least one processor to perform a method for performing video enhancement. The method includes identifying a first image from a plurality of images included in a first video, wherein the first image includes an anomaly. Additionally, the method includes obtaining guidance information corresponding to the anomaly from various types of guidance information. Furthermore, the method includes obtaining a second image based on the guidance information to replace the first image. Additionally, the method includes obtaining a second video with the anomaly removed by using the second image. Attached Figure Description

[0008] The above and other aspects and features of certain embodiments of this disclosure will become more apparent from the following description taken in conjunction with the accompanying drawings, in which:

[0009] Figure 1 The images from which anomalies have been detected and the images from which anomalies have been removed are shown.

[0010] Figure 2 This is a flowchart describing a method for performing video enhancement according to embodiments of the present disclosure;

[0011] Figure 3 This is a flowchart describing a process for determining a first image in a first video comprising a plurality of images in accordance with embodiments of the present disclosure, in which an anomaly has been detected.

[0012] Figure 4 This is a diagram illustrating a method for detecting a blurring effect caused by steam according to embodiments of the present disclosure;

[0013] Figure 5 This is a diagram illustrating a method for detecting color changes caused by changes in lighting according to embodiments of the present disclosure;

[0014] Figure 6 This is a diagram illustrating a method for detecting shadows that need to be removed according to embodiments of the present disclosure;

[0015] Figure 7 This is a diagram illustrating a method for detecting reflection effects according to embodiments of the present disclosure;

[0016] Figure 8 This is a diagram used to describe an anomaly detection model according to embodiments of the present disclosure;

[0017] Figure 9 This is a flowchart describing a process for obtaining boot information corresponding to an anomaly from various types of boot information according to embodiments of the present disclosure;

[0018] Figure 10This is a flowchart describing the process of generating a second image to replace a first image based on obtained guidance information according to embodiments of the present disclosure;

[0019] Figure 11 This is a diagram illustrating an example of obtaining a second video, which has been enhanced based on a guide image, from a first video from which an anomaly has been detected, according to an embodiment of this disclosure;

[0020] Figure 12 This is a block diagram illustrating an electronic device for performing video enhancement according to an embodiment of the present disclosure;

[0021] Figure 13 This is a block diagram illustrating the configuration and operation of an electronic device for performing video enhancement according to embodiments of the present disclosure; and

[0022] Figure 14 This is a block diagram illustrating the operation of a guide information obtaining module according to embodiments of the present disclosure. Detailed Implementation

[0023] The terminology used herein is briefly described, and the disclosure is described in detail. Throughout the disclosure, the expression "at least one of a, b, or c" indicates "only a", "only b", "only c", "both a and b", "both a and c", "both b and c", "all of a, b, and c", or any variation thereof.

[0024] Regarding the terminology used in this disclosure, as many commonly used terms as possible that are currently widely used have been selected while taking into account the function of this disclosure. However, terms may vary depending on the intent of those skilled in the art, precedents, the emergence of new technologies, etc. Furthermore, in certain cases, there may be terms that are arbitrarily chosen by the applicant. In such cases, the meaning of the term will be described in detail in the description of this disclosure. Therefore, the terms used herein should be defined based on the meaning of the term and the description of the entire disclosure, rather than simply based on the name of the term.

[0025] Unless the context clearly indicates otherwise, the singular form used herein is intended to include the plural form as well. All terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art. It will be understood that although terms such as “first,” “second,” etc., may be used to describe various elements, these elements should not be limited by these terms. These terms are used only to distinguish one element from another.

[0026] Throughout this specification, unless otherwise stated, the phrase "a portion includes a certain element" means that the portion may further include other elements rather than exclude them. Furthermore, terms such as "...machine" and "module" used in this specification mean a unit that performs at least one function or operation and can be implemented as hardware, software, or a combination of hardware and software.

[0027] The artificial intelligence (AI) related functions disclosed herein operate via a processor and memory. The processor can be implemented as one or more processors. In this case, the one or more processors can be general-purpose processors (such as central processing units (CPUs), application processors (APs), or digital signal processors (DSPs), graphics-specific processors (such as graphics processing units (GPUs) or vision processing units (VPUs)), or AI-specific processors (such as neural processing units (NPUs)). The one or more processors can perform control to process input data based on an AI model stored in memory or predefined operating rules. Alternatively, when the one or more processors are AI-specific processors, the AI-specific processors can be designed with hardware architectures dedicated to processing specific AI models.

[0028] AI models and predefined operating rules are built through learning. The phrase "built through learning" means that the AI ​​model or predefined operating rules configured to perform desired characteristics (or purposes) are built in such a way that a basic AI model is trained using a large amount of training data via a learning algorithm. According to this disclosure, learning can be performed within the device performing the AI ​​itself, or it can be performed via a separate server and / or system. Examples of learning algorithms include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but this disclosure is not limited to the examples described above.

[0029] AI models can include multiple neural network layers. Each neural network layer has multiple weight values ​​and performs neural network operations through operations between these weight values ​​and the results of the previous layer. The weight values ​​of neural network layers can be optimized using the learning results of the AI ​​model. For example, the weight values ​​can be updated to reduce or minimize the loss or cost values ​​obtained by the AI ​​model during the learning process. Artificial neural networks can include deep neural networks (DNNs), such as convolutional neural networks (CNNs), deep neural networks (DNNs), recurrent neural networks (RNNs), restricted Boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), or deep Q-networks, but this disclosure is not limited to the examples described above.

[0030] In the following, one or more embodiments of the present disclosure will be described in detail with reference to the accompanying drawings, enabling those skilled in the art to implement the disclosure. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein.

[0031] This disclosure is described in detail below with reference to the accompanying drawings.

[0032] Figure 1 The image shows the one from which anomalies have been detected and the one from which anomalies have been removed.

[0033] In this disclosure, a video includes multiple images. A video can be a series of continuously captured moving images or a set of intermittently captured still images. A video can include consecutive images. For example, a video can be a series of images captured with zero shutter lag, or it can be images recorded together at a time before and after the moment a photograph is captured by pressing the shutter. An image can be a frame of a video or a still image such as a photograph. An image can be at least one photograph or still image selected from a plurality of images included in a video.

[0034] refer to Figure 1 The first video may include multiple images captured at different times. For example, the first video may be a cooking video, in which cooking actions or processes for preparing food are captured sequentially, such as... Figure 1 As shown. For ease of illustration, the description is given assuming the first video is a cooking video, but the videos in this disclosure are not limited to this.

[0035] Videos that capture images of the same object, whether continuously or intermittently, create a sense of unity in the colors or tones of the images or the shapes of objects within them. Because of this perceived unity within the video, users can distinguish objects in the video as the same object without feeling discomfort or inconsistency.

[0036] refer to Figure 1 In the image corresponding to the frame at time point n of the first video, the cooking utensils or food being cooked are difficult to identify due to the steam generated during cooking. Objects appear blurry and image quality is degraded in the image corresponding to the frame at time point n; therefore, the uniformity between adjacent images corresponding to frames before and after time point n is reduced. Thus, the user can perceive that the first video does not continue naturally. The user can also perceive anomalies in the image corresponding to the frame at time point n.

[0037] In this disclosure, anomalies may be caused by factors such as objects in the image, changes in the capture environment, or other objects outside the image. For example, anomalies may be caused by steam generated from food, changes in lighting during image capture, shadows on objects unrelated to the video content, or the appearance of other objects reflected on the surface of objects in the image.

[0038] To ensure that the images in the first video connect naturally to each other and to facilitate user identification of objects within the first video, video enhancement can be performed on the image corresponding to the frame at time point n in the first video. Video enhancement can be performed on regions or the entire image corresponding to anomalies within the image.

[0039] In this disclosure, video enhancement refers to the task of adjusting an original video to make it easier to identify objects included in the video. For example, video enhancement may include a series of processes that modify an original video to obtain information needed for a specific purpose from the video. Video enhancement may be the process of removing anomalies or noise included in a video. Video enhancement may include the process of improving image quality by correcting sharpness, contrast, color, etc., or adjusting brightness values.

[0040] In embodiments of this disclosure, video enhancement can be understood as image enhancement directed at an image. Image enhancement can involve removing anomalies or noise or improving image quality for one or more images stored in a storage space or storage path. For example, image enhancement can involve modifying one or more images selected by a user among images captured sequentially with zero shutter lag. As another example, image enhancement can involve modifying one or more images selected by a user among images recorded together at a time before and after the moment a photo is captured by pressing the shutter. Methods and apparatus for performing video enhancement are described below.

[0041] Figure 2 This is a flowchart describing a method for performing video enhancement according to embodiments of the present disclosure.

[0042] Reference Figure 2In operation S210, the device for performing video enhancement can determine a first image from a first video comprising multiple images in which an anomaly has been detected. The device for performing video enhancement can be an electronic device including a camera that captures the video, or it can be an electronic device or server that obtains the video from an external source. For example, the device for performing video enhancement can be a video processing device, such as a smartphone, smart glasses, wearable device, digital camera, laptop computer, augmented reality (AR) device, and virtual reality (VR) device. The device for performing video enhancement can be provided with various types of neural network models. For example, the device for performing video enhancement can be provided with at least one model, such as a deep neural network (DNN), convolutional neural network (CNN), recurrent neural network (RNN), and bidirectional recurrent deep neural network (BRDNN), and these networks can be used in combination.

[0043] The first video, including images, can be video captured by a device used to perform video enhancement, or it can be video received from an external device. The device used to perform video enhancement can analyze the images included in the first video, determine whether an anomaly is detected, and identify images where an anomaly has been detected. (Referring below...) Figures 3 to 8 Describe in detail a method for detecting anomalies in a video that includes multiple images.

[0044] Figure 3 This is a flowchart, according to embodiments of the present disclosure, describing the identification of a first image in a first video comprising multiple images in which an anomaly has been detected.

[0045] refer to Figure 3 In operation S310, the device for performing video enhancement can extract feature information from each of the images included in the first video. The device for performing video enhancement can extract information about at least one of the components constituting the image (e.g., edges, colors, etc.). The device for performing video enhancement can extract feature information for each of the images by using information about the components constituting the image. The feature information required to detect anomalies can vary depending on the type of anomaly. The device for performing video enhancement can extract at least one piece of feature information to detect anomalies.

[0046] In operation S320, the device used to perform video enhancement can detect anomalies based on the extracted feature information. The device used to perform video enhancement can detect anomalies in various ways depending on the type of anomaly.

[0047] Devices used for video enhancement can perform an operation to detect anomalies based on extracted feature information. For example, a device for video enhancement can measure the edge intensity of each element in an image. A device for video enhancement can obtain a color distribution close to a specific value by using the RGB color code of each element in the image. A device for video enhancement can obtain an image histogram of each element in the image and perform an operation to determine the characteristics of the image.

[0048] Devices used for video enhancement can detect anomalies by employing an anomaly detection model that uses extracted feature information as input. The anomaly detection model can be trained using a set of images containing the anomaly to be detected and images not containing the anomaly to be detected. The anomaly detection model can receive feature information extracted from the images as input and can output whether a specific type of anomaly was detected. The anomaly detection model can detect various types of anomalies based on the feature information extracted from the images and can classify the types of detected anomalies.

[0049] Figure 4 This is a diagram illustrating a method for detecting a blurring effect caused by steam according to embodiments of the present disclosure.

[0050] Figure 4 The results are shown for performing an operation using dark channel priors and the variance of the Laplacian operator to detect images in which steam-induced blurring has already occurred. The dark channel prior analyzes the R, G, and B channel values ​​of each pixel in the image. The dark channel prior measures the intensity of the steam-induced blurring effect by using the statistical property that at least one channel value converges to 0 in the non-degraded region. Regions where steam-induced blurring has occurred have values ​​close to (R, G, B) = (255, 255, 255). By using this, it can be determined whether steam-induced blurring has occurred in the image. The variance of the Laplacian operator measures the edge strength within the image by performing Laplacian filtering as the second derivative on the image. Fewer edges are detected in images where steam-induced blurring has occurred. By using this, it can be determined whether steam-induced blurring has occurred in the image. However, the method for detecting images in which steam-induced blurring has occurred is not limited to this.

[0051] like Figure 4 As shown, as a result of performing a certain operation on the three images using the dark channel prior and the variance of the Laplacian operator, it was confirmed that the first image (image #1) with the blurring effect caused by steam has a lower value than the second image (image #2) and the third image (image #3) without the blurring effect caused by steam.

[0052] Figure 5 This is a diagram illustrating a method for detecting color changes caused by changes in lighting according to embodiments of the present disclosure.

[0053] Figure 5 This diagram illustrates the results of performing an operation using an image histogram for each image to detect images where color changes caused by illumination variations have occurred. An image histogram displays the characteristics of an image by representing the values ​​of image pixels on the horizontal axis and the number of image pixels on the vertical axis. An image histogram is a graph showing the distribution of the number of bright and dark pixels in an image. However, methods for detecting images where color changes caused by illumination variations have occurred are not limited to this.

[0054] For each of the multiple images constituting the video, an image histogram can be obtained for each of the R, G, and B channels, and the histogram correlation values ​​between adjacent frames can be obtained. At moments when the lighting environment changes, the histogram correlation values ​​between adjacent frames decrease briefly. For example... Figure 5 As shown, because the histogram correlation value of image #115 decreases briefly, it can be confirmed that a color change caused by the lighting change has already occurred in frame 115 of the video.

[0055] Figure 6 This is a diagram illustrating a method for detecting shadows that need to be removed according to embodiments of the present disclosure.

[0056] Shadows can appear on objects and the lighting that illuminates them, and these shadows can appear along with other objects within the image. When all shadows in an image are removed, the image may become unnatural because objects within the image may be perceived as copied and pasted. Therefore, it is necessary to selectively remove only the shadows within the image that are considered abnormal.

[0057] Shadows that need to be removed as anomalies within an image can be located at the edges of the image and can change frequently. Objects located at the edges of an image, or their shadows, may have low importance within the image. Shadows that appear and disappear within the image for a short period of time may be caused by external factors such as lighting or other objects.

[0058] To detect shadows that need to be removed as anomalies, the device performing video enhancement can find all shadow regions in an image through shadow segmentation and can distinguish between shadow regions that need to be removed and those that do not by using algorithms or neural networks capable of measuring the degree of shadow change or movement. When shadow regions that need to be removed are present in the image, the device performing video enhancement can determine that shadows have been detected as anomalies and need to be removed.

[0059] like Figure 6 As shown, a device for performing video enhancement can use an anomaly detection model to detect shadows that need to be removed as anomalies. Shadows that need to be removed as anomalies are located at the edges of the image and change frequently. This shadow corresponds to the target shadow. On the other hand, shadows that do not need to be removed are located in the center of the image and appear continuously within the image. This shadow corresponds to the non-target shadow. The anomaly detection model can be a model trained using a set of images that include shadows that need to be removed as anomalies (i.e., target shadows) and images that do not include target shadows. The anomaly detection model can be trained by giving high scores to images that include target shadows and low scores to images that do not include target shadows. The anomaly detection model can receive feature information extracted from the image as input and can output whether a shadow that needs to be removed as an anomaly is detected in the image.

[0060] Figure 7 This is a diagram illustrating a method for detecting reflection effects according to embodiments of the present disclosure.

[0061] When the surface of the object being photographed is made of a material such as glass, or when there is a glass window between the camera and the object, a reflection effect can occur in the captured image. That is, the image of another object reflected from the surface of the object or the glass window can overlap with the captured image. For example, when capturing video of cooking inside an oven, the image of the photographer or another object may be reflected on the glass oven door.

[0062] like Figure 7 As shown, a device for performing video enhancement can use an anomaly detection model to detect reflection effects that need to be removed as anomalies. The anomaly detection model can be a model trained using a set of images that include images with reflection effects that need to be removed as anomalies and images that do not include reflection effects. Images including reflection effects can be captured images in which images of the photographer or other objects are reflected, or images in which images of people or other objects are combined with images that do not include reflection effects. The anomaly detection model can be trained by assigning high scores to images including reflection effects and low scores to images that do not include reflection effects. The anomaly detection model can receive feature information extracted from the images as input and can output whether reflection effects that need to be removed as anomalies are detected in the image.

[0063] Figure 8 This is a diagram used to describe an anomaly detection model according to embodiments of the present disclosure.

[0064] Devices used for video enhancement can detect anomalies by employing an anomaly detection model that uses feature information extracted from an image as input. The anomaly detection model can detect at least one anomaly based on the feature information extracted from the image and can classify the type of the detected anomaly. Devices used for video enhancement can use the anomaly detection model to obtain information about the image in which anomalies have been detected, the type of anomaly, the region in which anomalies have been detected, etc.

[0065] like Figure 8 As shown, an anomaly detection model can take the form of a multi-task model. A multi-task model can be in the form of a neural network comprising multiple layers. A multi-task model can include a shared backbone layer and additional layers that receive the output of the shared backbone as input for each task. A multi-task model can include a feature extractor that extracts feature information and a predictor that detects anomalies using the extracted feature information. For example, an anomaly detection model can be a multi-task model where a prediction head for each type of anomaly is connected to a feature extractor that extracts feature information from an image, such as... Figure 8 As shown.

[0066] A feature extractor can extract information about the components that make up each of the multiple images included in a first video. For example, the feature extractor can extract information about the edges of each image. The feature extractor can extract information about the color of each image. The feature extractor can extract information about the brightness or contrast of each image. The feature extractor may include an image encoder.

[0067] The predictor comprises multiple prediction heads, each capable of performing the task of detecting different types of anomalies. Each prediction head can be a calculator performing a specific operation or a neural network comprising layers. For example, in Figure 8 In the task, Task-A could be a task to detect blurring effects caused by steam. Task-B could be a task to detect color changes caused by changes in lighting. Task-C could be a task to detect shadows that need to be removed. Task-D could be a task to detect reflection effects.

[0068] Refer again Figure 3 In operation S330, the device for performing video enhancement can select a first image from the first video in which an anomaly has been detected. The device for performing video enhancement can select the first image from the images constituting the first video by generating information about the anomaly detected from the first image. For example, the device for performing video enhancement can generate information about the image in which an anomaly has been detected, information identifying the type of anomaly, information about the region in which an anomaly has been detected, etc.

[0069] Refer again Figure 2In operation S220, the device used to perform video enhancement can obtain guidance information corresponding to the anomaly from various types of guidance information. Guidance information can be guidance images, guidance masks, guidance features, or combinations of two or more of them. The type of guidance information required to perform video enhancement can vary depending on the type of anomaly.

[0070] A device for performing video enhancement can generate first guidance information from a guidance image, a guidance mask, and guidance features based on the type of anomaly. The guidance image may be information about at least one image that can be used as a reference for video enhancement of the first image. The guidance mask may be information identifying a specific object or region within the image. The guidance features may be information informing the characteristics of the image or objects within the image.

[0071] The device used to perform video enhancement can determine whether the generated first guidance information is valid. For example, when no guidance information is actually generated or when guidance information without any information is generated, the device used to perform video enhancement can determine that the generated first guidance information is invalid. The device used to perform video enhancement can also determine that the generated first guidance information is invalid when the guidance information is outside a certain valid range.

[0072] The device for performing video enhancement can determine, based on the result of determining whether the generated first guidance information is valid, either the first guidance information or a second guidance information different from the first guidance information, as guidance information. When the generated first guidance information is valid, the device for performing video enhancement can determine that the generated first guidance information is guidance information. When the generated first guidance information is valid, the device for performing video enhancement can also determine whether additional guidance information is needed. When no additional guidance information is needed, the device for performing video enhancement can determine that the generated first guidance information is guidance information. When additional guidance information is needed, the device for performing video enhancement can change the guidance information generation method, generate second guidance information, and determine both the generated first and second guidance information as guidance information. When the generated first guidance information is invalid, the device for performing video enhancement can change the guidance information generation method, generate second guidance information, and determine that the generated second guidance information is guidance information.

[0073] Figure 9 This is a flowchart describing a process for obtaining boot information corresponding to an exception from various types of boot information according to embodiments of the present disclosure.

[0074] In operation S910, the device used to perform video enhancement can identify the type of anomaly detected in the first image. This is because the type of guidance information to be generated is determined based on the type of anomaly.

[0075] In operation S920, the device for performing video enhancement can generate guidance information from the guidance image, guidance mask, and guidance features based on the type of anomaly. For example, when the anomaly is a blurring effect caused by steam, the device for performing video enhancement can generate guidance information of the type of guidance image. When the anomaly is a color change caused by a change in lighting, the device for performing video enhancement can generate guidance information of the type of guidance feature. When the anomaly is the appearance of a shadow that needs to be removed, the device for performing video enhancement can generate guidance information of the type of guidance mask. When the anomaly is a reflection effect, the device for performing video enhancement can generate guidance information of both the type of guidance image and the type of guidance mask.

[0076] In operation S930, the device used to perform video enhancement can determine whether the generated guidance information is valid. For example, when no guidance information is actually generated, when guidance information without any information is generated, or when the guidance information is not within a certain valid range, the device used to perform video enhancement can determine that the generated guidance information is invalid.

[0077] In operation S940, when the generated guidance information is valid, the device used to perform video enhancement can determine whether additional guidance information is needed. For example, to remove or improve anomalies, the device used to perform video enhancement can determine whether more guidance information is needed based on the type of anomaly.

[0078] In operation S950, when the device used to perform video enhancement determines that the generated guidance information is valid and no further additional guidance information is required, the device used to perform video enhancement can determine the guidance information.

[0079] In operation S960, when the device used to perform video enhancement determines that the generated guidance information is invalid or that additional guidance information is required even if the generated guidance information is valid, the device used to perform video enhancement may change the guidance information generation method and may generate other guidance information.

[0080] Embodiments of this disclosure for obtaining bootstrapping information for each type of exception are described below with reference to specific examples. Depending on the type of exception, [the following may be omitted]. Figure 9 Some of the processes shown are related to obtaining guidance information.

[0081] When the anomaly is a blurring effect caused by steam, the device for performing video enhancement can obtain, from the first video image in which no steam-induced blurring effect is detected, an image that is most similar in shape and contour of objects within the image to a first image in which steam-induced blurring has been detected, as the first guiding image. When a steam-induced blurring effect occurs, the loss of color information in the image is significant, but edges can be extracted to the extent necessary to determine the shape and contour of objects within the image. The device for performing video enhancement can determine whether the first guiding image is valid based on the similarity between the first guiding image and the first image. Based on the determined result, the device for performing video enhancement can identify the first guiding image as guiding information, or it can identify a guiding mask that performs segmentation on objects within the first image, guiding features including a color map of the objects, and a second guiding image that does not include the objects as guiding information. When generating guiding features such as a color map of the objects, a guiding mask that has undergone segmentation corresponding to the objects can be used.

[0082] When the anomaly is a color change caused by a change in lighting, the device for performing video enhancement can generate a first guiding feature based on the most common color information extracted from the color information of each of the images included in the first video. The device for performing video enhancement can extract color information from each image, cluster similar color information together, and generate the first guiding feature based on the representative color information of the largest cluster. The device for performing video enhancement can determine whether the generated first guiding feature is within a certain range. Based on the determination, the device for performing video enhancement can determine the generated first guiding feature as guiding information, or it can determine a second guiding feature based on a user-specified color range as guiding information. Alternatively, when the device for performing video enhancement determines that the first guiding feature is not within a certain range, the device for performing video enhancement can determine the second guiding feature as guiding information based on the most common color information extracted from the color information of each of the neighboring images within a certain range of the first image that detected the color change caused by the change in lighting.

[0083] When the anomaly is the presence of a shadow that needs to be removed, the device for performing video enhancement can generate a first guiding mask corresponding to each shadow region in a first image where the shadow that needs to be removed has been detected. When multiple guiding masks are generated based on multiple shadow regions, the first guiding mask can be generated by fusing the guiding masks. The device for performing video enhancement can determine whether the first guiding mask is valid based on the extent and position of the generated first guiding mask. The device for performing video enhancement can determine the generated first guiding mask as guiding information based on the determined result, or it can determine a second guiding mask generated by changing the mask generation model as guiding information. The device for performing video enhancement can generate an image with no shadow region or with the smallest shadow region in the first video as a guiding image by using information about the entire shadow region, the target shadow region, and the non-target shadow region within the image generated during the generation of the first guiding mask as additional guiding information, and can use the generated image to prevent unnecessary shadow removal.

[0084] When the anomaly is a reflection effect, the device performing video enhancement can generate a first guide image as the most similar image to the first image in which the reflection effect was detected, from images in the first video where no reflection effect was detected. It can also generate a first guide mask corresponding to the region in the first image with the reflection effect. The device performing video enhancement can determine whether the first guide mask is valid based on whether it targets a portion of the first image that is not reflected due to the presence of an object. If the first guide mask is generated even if the surface of an object in the first image includes materials that do not cause reflection, the device performing video enhancement can determine that the generated first guide mask is invalid. Based on this determination, the device performing video enhancement can determine the first guide mask and the first guide image as guide information, or it can determine the generated first guide image and a second guide mask generated by changing the mask generation model as guide information.

[0085] Refer again Figure 2 In operation S230, the device for performing video enhancement can generate a second image to replace the first image based on the obtained guidance information. The device for performing video enhancement can generate the second image from the first image based on the guidance information, or it can generate the second image by combining the guidance information.

[0086] Figure 10 This is a flowchart describing the process of generating a second image to replace a first image based on obtained guidance information according to embodiments of the present disclosure.

[0087] In operation S1010, the device for performing video enhancement can determine the video enhancement function to be performed based on information about the anomaly and the obtained guidance information. The information about the anomaly may include information about the image in which the anomaly has been detected, information identifying the type of anomaly, and information about the region in which the anomaly has been detected, but this disclosure is not limited thereto.

[0088] In operation S1020, the device for performing video enhancement can correct or replace regions in the first image that have been detected as abnormal based on the obtained guidance information, or it can generate a second image by combining the obtained guidance information. For example, the device for performing video enhancement can generate a second image by correcting at least one of the color, color features, or sharpness of regions in the first image that have been detected as abnormal based on the obtained guidance information. The device for performing video enhancement can extract regions corresponding to regions in the first image that have been detected as abnormal from the obtained guidance information, and can generate a second image by replacing the regions that have been detected as abnormal with the extracted regions. The device for performing video enhancement can use the guidance information to remove regions in the first image that have been detected as abnormal. The device for performing video enhancement can generate a second image by combining the obtained guidance information with regions in the first image.

[0089] Figure 11 This is a diagram illustrating an example of obtaining a second video, which has undergone video enhancement based on a guide image, from a first video where an anomaly has been detected, according to an embodiment of this disclosure.

[0090] Reference Figure 11 The device for performing video enhancement can detect a blurring effect caused by steam in a first image corresponding to a frame at time point n in a first video. To ensure the images in the first video connect naturally and to facilitate user identification of objects within the first video, the device for performing video enhancement can perform video enhancement on the first image corresponding to the frame at time point n in the first video. The device for performing video enhancement can perform video enhancement on the entire first image or on a region within the first image corresponding to the blurring effect caused by steam. For example, the device for performing video enhancement can extract the region corresponding to the blurring effect caused by steam from an acquired guide image and generate a second image by replacing the region corresponding to the blurring effect in the first image with the extracted region. Alternatively, the device for performing video enhancement can generate the second image by combining the acquired guide images. The device for performing video enhancement can obtain a second video that has undergone video enhancement by replacing the first image in the first video with the second image.

[0091] Refer again Figure 2In operation S240, the device for performing video enhancement can use the second image to obtain a second video with anomalies removed. Anomaly removal includes not only completely removing anomalies, but also weakening or reducing the severity of anomalies. The device for performing video enhancement can obtain a second video by replacing the first image in the first video with the second image.

[0092] A device used to perform video enhancement is a device capable of acquiring video and processing it using an AI model, and can be in the form of a server or a user-operable electronic device 100. In the following text, references... Figures 12 to 14 The configuration and operation of electronic device 100, which is used to perform video enhancement, are described below. Even if omitted below, the description of the method for performing video enhancement can be applied to electronic device 100 used to perform video enhancement.

[0093] Figure 12 This is a block diagram illustrating an electronic device 100 for performing video enhancement according to an embodiment of the present disclosure.

[0094] refer to Figure 12 An electronic device 100 for performing video enhancement according to embodiments of the present disclosure may include a memory 110, a processor 120, a communication interface 130, an input / output interface 140, and a camera 150.

[0095] The memory 110 may store instructions, data structures, and program code that can be read by the processor 120. In embodiments of this disclosure, operations performed by the processor 120 may be implemented by executing instructions or program code stored in the memory 110.

[0096] The memory 110 may include flash memory, hard disk memory, multimedia card micro-memory or card-type memory (e.g., Secure Digital (SD) or Extreme Digital (XD) memory), may include non-volatile memory (including at least one of read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk or optical disk), and may include volatile memory such as random access memory (RAM) or static random access memory (SRAM).

[0097] According to embodiments of the present disclosure, the memory 110 may store one or more instructions and / or programs for execution control, causing the electronic device 100 for performing video enhancement to process images. For example, the memory 110 may store an anomaly detection module, a guidance information acquisition module, and a video enhancement module. When AI model learning is required within the electronic device 100 to perform video enhancement, a model learning module may be further installed thereon.

[0098] Processor 120 can control the operations or functions performed by electronic device 100 for performing video enhancement by executing instructions stored in memory 110 or programmed software modules. Processor 120 may include hardware components that perform arithmetic, logic, and input / output operations, as well as signal processing. Processor 120 can control the overall operation of electronic device 100 for performing video enhancement by executing one or more instructions stored in memory 110.

[0099] Processor 120 may include at least one of the following: CPU, microprocessor, GPU, application-specific integrated circuit (ASIC), DSP, digital signal processing device (DSPD), programmable logic device (PLD), field-programmable gate array (FPGA), AP, neural processing unit, or AI-specific processor designed with a hardware architecture dedicated to processing AI models, but this disclosure is not limited thereto. Each processor constituting processor 120 may be a dedicated processor for performing a specific function.

[0100] Communication interface 130 can perform wired or wireless communication with other devices or networks. Communication interface 130 may include communication circuitry or modules supporting at least one of various wired and wireless communication methods. For example, communication interface 130 can use at least one of the data communication schemes for data communication between electronic device 100 for performing video enhancement and other devices, including wired local area network (LAN), wireless LAN, Wi-Fi, Bluetooth, ZigBee, Wi-Fi Direct (WFD), Infrared Data Association (IrDA), Bluetooth Low Energy (BLE), Near Field Communication (NFC), Wireless Broadband Internet (Wibro), Global Microwave Access Interoperability (WiMAX), Shared Wireless Access Protocol (SWAP), Wireless Gigabit Alliance (WiGig), and radio frequency (RF) communication.

[0101] The communication interface 130 according to embodiments of this disclosure can receive AI models or images for performing video enhancement from external devices. For example, the communication interface 130 can receive a database or AI model trained on a server. The communication interface 130 can receive images from other electronic devices. The communication interface 130 can send images generated by the electronic device 100 for performing video enhancement to external devices, such that the images are shared with another user or displayed on external devices associated with the images.

[0102] The communication interface 130 according to embodiments of this disclosure can send images or information about images obtained by the electronic device 100 for performing video enhancement to a server, or can receive images from the server. For example, the communication interface 130 can send images captured by the electronic device 100 for performing video enhancement or images received from another electronic device to the server. The communication interface 130 can send information about images input through the input / output interface 140 of the electronic device 100 for performing video enhancement to the server. The communication interface 130 can receive images stored in the server from the server.

[0103] The input / output interface 140 may be provided in the form of an output interface that provides information or images and an input interface that receives input. The output interface may include a display panel and a controller for controlling the display panel, and may be implemented in various ways, such as an organic light-emitting diode (OLED) display, an active-matrix organic light-emitting diode (AM-OLED) display, and a liquid crystal display (LCD). The input interface may receive various types of input from a user and may include at least one of a touch panel, a keyboard, or a pen recognition panel. The input / output interface 140 may be provided in the form of a touchscreen, wherein the display panel and the touch panel are combined with each other, and may be implemented as flexible or foldable.

[0104] The input / output interface 140 according to embodiments of this disclosure can obtain information about an image from a user. The user can select the video to be enhanced from a plurality of videos through the input / output interface 140. The input / output interface 140 can receive user input for information or control commands required during the video enhancement process.

[0105] Camera 150 is a hardware module for capturing images. Camera 150 can capture images. Camera 150 may include at least one camera module and may support functions such as depth of field, telephoto, wide-angle, and ultra-wide-angle, depending on the specifications of the electronic device 100 for performing video enhancement. Camera 150 can be excluded from the electronic device 100 for performing video enhancement unless images are captured directly using the electronic device 100 for performing video enhancement.

[0106] Figure 13 This is a block diagram illustrating the configuration and operation of an electronic device 100 for performing video enhancement according to embodiments of the present disclosure.

[0107] refer to Figure 13An electronic device 100 for performing video enhancement may include a memory 110 storing one or more instructions and a processor 120 executing one or more instructions stored in the memory 110. The processor 120 may load instructions or code from the memory 110 for the anomaly detection module 121, the boot information acquisition module 123, and the video enhancement module 129, and may execute the loaded instructions or code.

[0108] Processor 120 can execute one or more instructions to determine a first image in a first video comprising multiple images in which an anomaly is detected. Processor 120 can receive the first video stored in memory 110 and can perform an anomaly detection task for each image included in the first video via anomaly detection module 121. Processor 120 can extract feature information from each image included in the first video via anomaly detection module 121, can detect anomalies based on the extracted feature information, and can select a first image in the first video in which an anomaly has been detected. Processor 120 can perform an anomaly detection operation based on the extracted feature information via anomaly detection module 121, or it can detect anomalies by using an anomaly detection model that uses the extracted feature information as input.

[0109] Processor 120 can execute one or more instructions to obtain guidance information corresponding to an exception from various types of guidance information. Processor 120 can generate first guidance information from a guidance image, guidance mask, and guidance features based on the type of exception via guidance information acquisition module 123. Processor 120 can determine whether the first guidance information generated by guidance information acquisition module 123 is valid. Based on the determination result, processor 120 can identify the generated first guidance information or second guidance information different from the generated first guidance information as guidance information.

[0110] When the first boot information generated by the boot information acquisition module 123 is valid, the processor 120 can determine the generated first boot information as boot information. When the first boot information generated by the boot information acquisition module 123 is valid, the processor 120 can determine whether additional boot information is needed. When no additional boot information is needed, the processor 120 can determine the generated first boot information as boot information. When the first boot information generated by the boot information acquisition module 123 is valid and additional boot information is needed, the processor 120 can change the boot information generation method, generate second boot information, and determine both the generated first and second boot information as boot information. When the first boot information generated by the boot information acquisition module 123 is invalid, the processor 120 can change the boot information generation method, generate second boot information, and determine the generated second boot information as boot information.

[0111] Processor 120 can execute one or more instructions to generate a second image to replace the first image based on obtained guidance information. Processor 120 can generate the second image from the first image via video enhancement module 129 based on the guidance information, or it can generate the second image via video enhancement module 129 by combining the guidance information. Processor 120 can determine the video enhancement function to be performed via video enhancement module 129 based on information about anomalies and the obtained guidance information. Information about anomalies may include information about images where anomalies have been detected, information identifying the type of anomaly, and information about the regions where anomalies have been detected, but this disclosure is not limited thereto. Processor 120 can correct or replace regions in the first image where anomalies have been detected via video enhancement module 129 based on the obtained guidance information, or it can generate the second image via video enhancement module 129 by combining the obtained guidance information.

[0112] Processor 120 can execute one or more instructions to obtain a second video with anomalies removed by using a second image. Processor 120 can replace the first image in the first video with the second image generated by video enhancement module 129 to obtain a second video that has undergone video enhancement, and can store the second video in memory 110.

[0113] Figure 14 This is a block diagram describing the operation of the guide information obtaining module 123 according to embodiments of the present disclosure.

[0114] refer to Figure 14 The guidance information acquisition module 123 may include a guidance information generation module 125, a module controller 126, and a verification module 127. The guidance information acquisition module 123 can receive a first video stored in the memory 110. The guidance information acquisition module 123 may receive information about anomalies from the anomaly detection module 121, or it may receive control signals based on user input. The guidance information acquisition module 123 can output guidance information to the video enhancement module 129.

[0115] The guidance information generation module 125 may include a guidance image generator, a guidance mask generator, a guidance feature generator, and an additional guidance generator. Based on control signals output from the module controller 126, the guidance information generation module 125 can generate guidance information corresponding to the detected anomaly in a first image of the first video. The module controller 126 can receive information about the anomaly, such as information about the image where the anomaly has been detected, information identifying the type of anomaly, or information about the region where the anomaly has been detected. The module controller 126 can control the guidance information generation module 125 to generate guidance information corresponding to the detected anomaly in the first image where the anomaly has been detected. The guidance information generation module 125 can output the generated guidance information to the verification module 127.

[0116] The guidance information generation module 125 can generate guidance information from various types of guidance information based on the type of detected anomaly. For example, when the anomaly is a blurring effect caused by steam, the guidance information generation module 125 can generate a guidance image using a guidance image generator. The guidance image generator can generate an image that is most similar in shape and outline to the first image in which the blurring effect caused by steam is detected, from images included in the first video where no blurring effect caused by steam is detected, as the first guidance image. When the anomaly is a color change caused by a change in lighting, the guidance information generation module 125 can generate guidance features using a guidance feature generator. The guidance feature generator can generate the first guidance feature based on the color information with the highest frequency of occurrence from each color information extracted from the images included in the first video. When the anomaly is the appearance of a shadow that needs to be removed, the guidance information generation module 125 can generate a guidance mask using a guidance mask generator. The guidance mask generator can generate a first guidance mask corresponding to each shadow area in the first image where a shadow that needs to be removed has been detected. When the anomaly is a reflection effect, the guidance information generation module 125 can generate a guidance image and a guidance mask using both the guidance image generator and the guidance mask generator. The guide image generator can generate an image most similar to the first image in which reflections were detected, from images in the first video that did not show any reflections. The guide mask generator can generate a first guide mask corresponding to the regions in the first image that show reflections.

[0117] The guide image generator, guide mask generator, guide feature generator, and additional guide generator included in the guide information generation module 125 can share the guide information generated by the guide image generator, guide mask generator, guide feature generator, and additional guide generator. For example, when the guide image generator generates a guide image, it can use the guide mask generated by the guide mask generator.

[0118] The verification module 127 can determine whether the boot information received from the boot information generation module 125 is valid. For example, when no boot information is actually generated, when boot information without any information is generated, or when the boot information is outside a certain valid range, the verification module 127 can determine that the generated boot information is invalid. The verification module 127 can output the determined result to the module controller 126.

[0119] When the boot information is valid, module controller 126 can determine whether additional boot information is needed. When module controller 126 determines that the boot information is valid and no additional boot information is needed, module controller 126 can control the boot information to obtain the boot information generated by module 123. When module controller 126 determines that the boot information is invalid, or that additional boot information is needed even if the generated boot information is valid, module controller 126 can change the boot information generation method and can generate other boot information.

[0120] On the other hand, embodiments of this disclosure can be implemented in the form of a computer-readable recording medium including computer-executable instructions, such as program modules executable by a computer. A computer-readable recording medium can be any available medium accessible to a computer and can include any volatile and non-volatile medium, as well as any removable and non-removable medium. Furthermore, a computer-readable recording medium can include computer storage media and communication media. A computer storage medium can include any volatile, non-volatile, removable, and non-removable medium implemented using any method or technology for storing information, such as computer-readable instructions, data structures, program modules, or other data. Communication media typically include computer-readable instructions, data structures, or other data such as program modules that modulate data signals.

[0121] Furthermore, computer-readable recording media may be provided in the form of non-transitory computer-readable recording media. Non-transitory storage media are tangible devices and simply mean that they do not include signals (e.g., electromagnetic waves). This term does not distinguish between cases where data is stored semi-permanently in a storage medium and cases where data is temporarily stored in a storage medium. For example, a non-transitory storage medium may include a buffer for temporarily storing data.

[0122] According to embodiments of this disclosure, methods according to various embodiments of this disclosure can be provided by being included in a computer program product. The computer program product can be traded as a commodity between a seller and a buyer. The computer program product can be distributed in the form of a machine-readable storage medium (e.g., an optical disc read-only memory (CD-ROM)), or can be distributed online (e.g., downloaded or uploaded) between two user devices (e.g., smartphones) via an app store or directly. In the case of online distribution, at least a portion of the computer program product (e.g., a downloadable application) is at least temporarily stored on a machine-readable storage medium, such as the memory of a manufacturer's server, an app store's server, or a relay server, or can be temporarily generated.

[0123] According to embodiments of this disclosure, a method for performing video enhancement is provided. The method may include identifying a first image from a plurality of images included in a first video, wherein the first image includes an anomaly (S210). Additionally, the method may include obtaining guidance information corresponding to the anomaly from various types of guidance information (S220). Furthermore, the method may include obtaining a second image based on the guidance information to replace the first image (S230). Finally, the method may include obtaining a second video with the anomaly removed using the second image (S240).

[0124] Furthermore, according to embodiments of this disclosure, obtaining guidance information (S220) may include generating first guidance information from the guidance image, guidance mask, and guidance features based on the type of anomaly (S910, S920). Additionally, obtaining guidance information (S220) may include identifying whether the first guidance information is valid (S930). Furthermore, obtaining guidance information (S220) may include identifying the first guidance information as guidance information based on the result of identifying whether the first guidance information is valid, or identifying second guidance information different from the first guidance information as guidance information (S950).

[0125] Additionally, identifying as guidance information (S950) may include identifying whether additional guidance information is needed based on the identification that the first guidance information is valid; identifying the first guidance information as guidance information based on the identification that no additional guidance information is needed; and changing the guidance information generation method to generate second guidance information based on the identification that additional guidance information is needed, and identifying the first guidance information and the second guidance information as guidance information.

[0126] Additionally, identifying as guidance information (S950) may also include identifying whether additional guidance information is needed based on the identification that the first guidance information is valid (S940). Furthermore, identifying as guidance information (S950) may also include identifying the first guidance information as guidance information based on the identification that additional guidance information is not needed; and based on the identification that additional guidance information is needed, changing the guidance information generation method to generate second guidance information, and identifying the first guidance information and the second guidance information as guidance information.

[0127] Furthermore, according to embodiments of this disclosure, obtaining guidance information (S220) based on anomalies including blurring effects caused by steam may include obtaining a first guidance image from a plurality of images based on a comparison of a first image with one or more images from a plurality of images in which no blurring effect caused by steam was detected, wherein the comparison of the first image with one or more images is based on a comparison of the shape or contour of objects included in the first image and one or more images. Additionally, obtaining guidance information (S220) may include identifying whether the first guidance image is valid based on the similarity between the first guidance image and the first image. Furthermore, obtaining guidance information (S220) may include identifying the first guidance image as guidance information based on the result of identifying whether the first guidance image is valid, or identifying a guidance mask configured to perform segmentation on objects within the first image, guidance features including a color map of the object, and a second guidance image without objects as guidance information.

[0128] Furthermore, according to embodiments of this disclosure, obtaining guidance information (S220) based on anomalies including color changes caused by lighting variations may include generating a first guidance feature based on color frequency information extracted from multiple images. Additionally, obtaining guidance information (S220) may include identifying whether the first guidance feature falls within a certain range. Moreover, obtaining guidance information (S220) may include identifying a second guidance feature or the first guidance feature based on a user-specified color range as guidance information, based on the result of identifying whether the first guidance feature falls within a certain range.

[0129] Furthermore, according to embodiments of this disclosure, obtaining guidance information (S220) based on the anomaly including a shadow region may include generating a first guidance mask corresponding to the shadow region. Additionally, obtaining guidance information (S220) may include identifying whether the first guidance mask is valid based on its range and position. Furthermore, obtaining guidance information (S220) may include identifying the first guidance mask as guidance information based on the result of identifying whether the first guidance mask is valid, or identifying a second guidance mask generated by changing the mask generation model as guidance information.

[0130] Furthermore, according to embodiments of this disclosure, obtaining guidance information based on anomalies including reflection effects (S220) may include obtaining a first guidance image from multiple images based on a comparison of a first image with one or more images from a plurality of images in which no reflection effect was detected, wherein the comparison is based on the similarity between the first image and one or more images; and generating a first guidance mask corresponding to a region in the first image that includes a reflection effect. Additionally, obtaining guidance information (S220) may include identifying whether the first guidance mask is valid based on whether it corresponds to a portion of the first image in which no reflection occurs. Furthermore, the step of obtaining guidance information (S220) may include identifying the first guidance mask and the first guidance image as guidance information based on the result of identifying whether the first guidance mask is valid, or identifying a second guidance mask generated by changing the mask generation model and the first guidance image as guidance information.

[0131] Additionally, according to embodiments of this disclosure, identifying the first image (S210) may include extracting feature information from each of the plurality of images (S310). Furthermore, determining the first image (S210) may include identifying the presence of anomalies in the images among the plurality of images based on the feature information (S320). Additionally, determining the first image (S210) may include identifying an image containing anomalies as the first image (S330).

[0132] Additionally, identifying the presence of an anomaly (S320) may include performing specific operations based on feature information or using an anomaly detection model that utilizes feature information as input to detect the anomaly.

[0133] According to embodiments of this disclosure, a device 100 for performing video enhancement is provided. The device 100 for performing video enhancement may include at least one memory 110 storing one or more instructions; and at least one processor 120 configured to execute one or more instructions. Additionally, the one or more instructions, when executed by the at least one processor 120, are configured to cause the device 100 to identify a first image from a plurality of images included in a first video, wherein the first image includes an anomaly. Furthermore, the one or more instructions, when executed by the at least one processor 120, are configured to cause the device 100 to obtain guidance information corresponding to the anomaly from various types of guidance information. Additionally, the one or more instructions, when executed by the at least one processor 120, are configured to cause the device 100 to obtain a second image based on the guidance information to replace the first image. Furthermore, the one or more instructions, when executed by the at least one processor 120, are configured to cause the device 100 to obtain a second video in which the anomaly has been removed by using the second image.

[0134] Additionally, according to embodiments of this disclosure, one or more instructions, when executed by at least one processor 120, are configured to cause device 100 to generate first boot information from a boot image, a boot mask, and boot features based on the type of anomaly. Furthermore, one or more instructions, when executed by at least one processor 120, are configured to cause device 100 to identify whether the first boot information is valid, and based on the result of identifying whether the first boot information is valid, to identify the first boot information or second boot information different from the first boot information as boot information.

[0135] Furthermore, according to embodiments of this disclosure, one or more instructions, when executed by at least one processor 120, are configured to cause device 100 to recognize the first boot information as boot information based on the recognition of the first boot information being valid. Additionally, one or more instructions, when executed by at least one processor 120, are configured to cause device 100 to change the boot information generation method, generate second boot information, and recognize the second boot information as boot information based on the recognition of the first boot information being invalid.

[0136] Furthermore, one or more instructions, when executed by at least one processor 120, are configured to cause device 100 to determine whether additional boot information is needed based on the recognition that the first boot information is valid. Furthermore, one or more instructions, when executed by at least one processor 120, are configured to cause device 100 to recognize the first boot information as boot information based on the recognition that additional boot information is not needed. Furthermore, one or more instructions, when executed by at least one processor 120, are configured to cause device 100 to change the boot information generation method, generate second boot information, and recognize the first and second boot information as boot information based on the recognition that additional boot information is needed.

[0137] Furthermore, according to embodiments of this disclosure, one or more instructions, when executed by at least one processor 120, are configured to cause device 100 to obtain a first guide image from a plurality of images based on a comparison of a first image with one or more images from a plurality of images in which no blurring effect caused by steam is detected, based on an anomaly including a blurring effect caused by steam. The comparison of the first image with the one or more images is based on a comparison of the shape or contour of objects included in the first image and the one or more images. Additionally, one or more instructions, when executed by at least one processor 120, are configured to cause device 100 to identify whether the first guide image is valid based on the similarity between the first guide image and the first image. Furthermore, one or more instructions, when executed by at least one processor 120, are configured to cause device 100 to identify the first guide image as guide information based on the result of identifying whether the first guide image is valid, or to identify a guide mask that performs segmentation on objects within the first image, guide features including a color map of the objects, and a second guide image without objects as guide information.

[0138] Additionally, one or more instructions, when executed by at least one processor 120, are configured to cause device 100 to generate a first guidance feature based on color frequency information extracted from multiple images, based on anomalies including color changes caused by lighting variations. Furthermore, one or more instructions, when executed by at least one processor 120, are configured to cause device 100 to identify whether the first guidance feature is within a certain range. Moreover, one or more instructions, when executed by at least one processor 120, are configured to cause device 100 to identify a second guidance feature or the first guidance feature based on a user-specified color range as guidance information, based on the result of identifying whether the first guidance feature is within a certain range.

[0139] Additionally, according to embodiments of this disclosure, one or more instructions, when executed by at least one processor 120, are configured to cause device 100 to generate a first boot mask corresponding to a shadowed region based on an anomaly including the shadowed region. Furthermore, one or more instructions, when executed by at least one processor 120, are configured to cause device 100 to identify whether the first boot mask is valid based on the extent and location of the first boot mask. Furthermore, one or more instructions, when executed by at least one processor 120, are configured to cause device 100 to: identify the first boot mask as boot information based on the result of identifying whether the first boot mask is valid, or identify a second boot mask generated by changing the mask generation model as boot information.

[0140] Furthermore, according to embodiments of this disclosure, one or more instructions, when executed by at least one processor 120, are configured to cause device 100 to obtain a first guide image from a plurality of images based on a comparison of a first image with one or more images from a plurality of images in which no reflection effect was detected, based on an anomaly including a reflection effect, wherein the comparison is based on the similarity between the first image and one or more images. Additionally, one or more instructions, when executed by at least one processor 120, are configured to cause device 100 to obtain a first guide mask corresponding to a region in the first image that includes a reflection effect. Furthermore, one or more instructions, when executed by at least one processor 120, are configured to cause device 100 to identify whether the first guide mask is valid based on whether it corresponds to a portion of the first image in which no reflection occurs. Furthermore, one or more instructions, when executed by at least one processor 120, are configured to cause device 100 to identify the first guide mask and the first guide image as guide information based on the result of identifying whether the first guide mask is valid, or to identify a second guide mask and the first guide image generated by changing the mask generation model as guide information.

[0141] Furthermore, according to embodiments of this disclosure, one or more instructions, when executed by at least one processor 120, are configured to cause device 100 to extract feature information from each of a plurality of images. Additionally, one or more instructions, when executed by at least one processor 120, are configured to cause device 100 to identify, based on the feature information, whether an anomaly exists in one of the plurality of images, and to identify the image including the anomaly as the first image.

[0142] According to embodiments of this disclosure, a computer-readable medium having instructions stored therein, which, when executed by at least one processor, cause at least one processor to perform a method for performing video enhancement. The method may include identifying a first image from a plurality of images included in a first video, wherein the first image includes an anomaly. Additionally, the method may include obtaining guidance information corresponding to the anomaly from various types of guidance information. Furthermore, the method may include obtaining a second image based on the guidance information to replace the first image. Additionally, the method may include obtaining a second video with the anomaly removed by using the second image.

[0143] According to embodiments of this disclosure, a method for performing video enhancement is provided. The method for performing video enhancement may include identifying a first image including an anomaly from a plurality of images. Additionally, the method may include identifying the type of the anomaly. Furthermore, the method may include generating first guidance information based on the type of the anomaly using a first guidance information generation method. Additionally, the method may include generating a second image to replace the first image based on the identified valid first guidance information. Furthermore, the method may include generating second guidance information using a second guidance information generation method different from the first guidance information generation method based on the identified invalid first guidance information, and generating a second image to replace the first image based on the second guidance information. Additionally, the method may include obtaining a second video with the anomaly removed by using the second image.

[0144] The foregoing description is for illustrative purposes only, and those skilled in the art will understand that other specific modifications can be made without altering the technical spirit or essential characteristics of this disclosure. Therefore, it should be understood that the embodiments disclosed above are illustrative in all respects and not restrictive. For example, a component described as singular may be implemented in a distributed manner. Similarly, a component described as distributed may be implemented in a composite manner.

[0145] The scope of this disclosure is defined by the appended claims rather than the detailed description above, and all changes or modifications derived from the meaning and scope of the claims and their equivalents shall be construed as falling within the scope of this disclosure.

Claims

1. A method for performing video enhancement, the method comprising: Identify a first image from a plurality of images included in a first video, wherein the first image includes an anomaly (S210). Obtain the guidance information corresponding to the exception from various types of guidance information (S220); The second image is obtained based on the guidance information to replace the first image (S230); and A second video with the anomalies removed is obtained by using the second image (S240).

2. The method according to claim 1, wherein, Obtaining guidance information (S220) includes: Based on the type of anomaly, first guidance information is generated from the guidance image, guidance mask, and guidance features (S910, S920). Identify whether the first guidance information is valid (S930); and Based on the result of identifying whether the first guidance information is valid, the first guidance information is identified as guidance information, or the second guidance information that is different from the first guidance information is identified as guidance information (S950).

3. The method according to claim 1 or 2, wherein, The identification of guidance information (S950) also includes: Based on the recognition that the first guidance information is valid, the first guidance information is identified as guidance information; and Based on the identification that the first guidance information is invalid, the guidance information generation method is changed to generate a second guidance information, and the second guidance information is identified as guidance information.

4. The method according to any one of claims 1 to 3, in, The identification of guidance information (S950) also includes: Based on the identification that the first guidance information is valid, determine whether additional guidance information is needed (S940). Based on the identification that no additional guidance information is needed, the first guidance information is identified as guidance information; and Based on the identification that additional guidance information is needed, the guidance information generation method is changed to generate second guidance information, and the first and second guidance information are identified as guidance information.

5. The method according to any one of claims 1 to 4, wherein, Based on the anomaly, including the blurring effect caused by steam, the guidance information obtained (S220) includes: A first guide image is obtained from multiple images based on a comparison of a first image with one or more images in which no blurring effect caused by steam is detected, wherein the comparison of the first image with one or more images is based on a comparison of the shape or outline of objects included in the first image and one or more images. Based on the similarity between the first guidance image and the first image, determine whether the first guidance image is valid; and Based on the result of identifying whether the first guide image is valid, the first guide image is identified as guide information, or a guide mask configured to perform segmentation on objects within the first image, guide features including the color map of the object, and a second guide image without objects are identified as guide information.

6. The method according to any one of claims 1 to 5, wherein, Based on the anomaly, including color changes caused by lighting variations, the guidance information obtained (S220) includes: The first guiding feature is generated based on the color frequency information extracted from multiple images; Identify whether the first guiding feature is within a certain range; and Based on the result of identifying whether the first guiding feature is within a certain range, the second guiding feature or the first guiding feature based on the color range specified by the user is identified as guiding information.

7. The method according to any one of claims 1 to 6, wherein, Based on the anomaly including the shadowed area, the following guidance information is obtained (S220): Generate a first guiding mask corresponding to the shadow area; Based on the range and position of the first guiding mask, determine whether the first guiding mask is valid; and Based on the result of identifying whether the first guiding mask is valid, the first guiding mask is identified as guiding information, or the second guiding mask generated by changing the mask generation model is identified as guiding information.

8. The method according to any one of claims 1 to 7, wherein, Based on the anomaly, including reflection effects, the guidance information obtained (S220) includes: A first guiding image is obtained from multiple images based on a comparison between a first image and one or more images from which no reflection effect was detected, wherein the comparison is based on the similarity between the first image and one or more images; Generate a first guiding mask corresponding to the region in the first image that includes a reflective effect; Based on whether the first guiding mask corresponds to the non-reflective portion of the first image, the validity of the first guiding mask is determined; and Based on the result of identifying whether the first guiding mask is valid, the first guiding mask and the first guiding image are identified as guiding information, or the second guiding mask and the first guiding image generated by changing the mask generation model are identified as guiding information.

9. The method according to any one of claims 1 to 8, wherein, Recognizing the first image (S210) includes: Extract feature information from each of the multiple images (S310); Identify the presence of anomalies in multiple images based on feature information (S320); and The image containing the anomaly is identified as the first image (S330).

10. The method according to any one of claims 1 to 9, wherein, Identifying the presence of an anomaly (S320) includes: performing an operation based on feature information or using an anomaly detection model that uses feature information as input to detect the anomaly.

11. An apparatus (100) for performing video enhancement, the apparatus comprising: At least one memory (110) stores one or more instructions; and At least one processor (120) is configured to execute the one or more instructions, wherein the one or more instructions, when executed by the at least one processor (120), are configured to cause the device (100): Identify a first image from among multiple images included in a first video, wherein the first image includes an anomaly. Obtain the corresponding guidance information from various types of guidance information. A second image is obtained based on the guidance information to replace the first image, and A second video with the anomalies removed was obtained by using a second image.

12. The device (100) according to claim 11, wherein, The one or more instructions, when executed by the at least one processor (120), are further configured to cause the device (100): Based on the type of anomaly, first guidance information is generated from the guidance image, guidance mask, and guidance features. Identify whether the first guidance message is valid, and Based on the result of identifying whether the first guidance information is valid, the first guidance information or the second guidance information that is different from the first guidance information is identified as guidance information.

13. The device (100) according to claim 11 or 12, wherein, The one or more instructions, when executed by the at least one processor (120), are further configured to cause the device (100): Based on the recognition that the first guidance information is valid, the first guidance information is identified as guidance information, and Based on the identification that the first guidance information is invalid, the guidance information generation method is changed to generate a second guidance information, and the second guidance information is identified as guidance information.

14. The device (100) according to any one of claims 11 to 13, wherein, The one or more instructions, when executed by the at least one processor (120), are further configured to cause the device (100): Extract feature information from each of multiple images; Identify the presence of anomalies in multiple images based on feature information; and The image containing the anomaly is identified as the first image.

15. A computer-readable medium having instructions stored thereon, the instructions, when executed by at least one processor, causing the at least one processor to perform a method for performing video enhancement, the method comprising: Identify a first image from a plurality of images included in a first video, wherein the first image includes an anomaly; Obtain guidance information corresponding to the exception from various types of guidance information; A second image is obtained based on the guidance information to replace the first image; and A second video with the anomalies removed was obtained by using a second image.

Images