System and method for processing input video

Abstract

A system and method for processing an input video is provided. The method includes: converting the input video from a first frame rate to a second frame rate, wherein the second frame rate is a frame rate faster than the first frame rate; generating processed frames of the input video at the second frame rate; and aggregating the processed frames using a temporal sliding window aggregation to produce a processed output video at a third frame rate.

Description

Technical Field

[0001] The present disclosure relates generally to video processing. In particular, the present disclosure relates to a system and method for processing video based on temporal characteristics between video frames. Background Art

[0002] A video is a series of frames played at a specific frame rate. A video can be processed frame by frame using known video processing techniques such as video editing, video colorization, video restoration, video enhancement, video super-resolution processing, and video style transfer processing.

[0003] Video processed using known processing techniques often results in temporally uncorrelated video due to changes between consecutive frames, resulting in flickering during video playback. Therefore, post-processing may be required to make the video frames temporally consistent. Post-processing may require a large amount of computer processing resources to consider all the information provided in each video frame.

[0004] Therefore, it may be advantageous to provide video processing techniques capable of generating video frames with enhanced temporal consistency, thereby alleviating the need to provide additional processing (ie, post-processing) for correcting temporal variations between video frames. Summary of the invention

[0005] According to one embodiment, a method for processing an input video while maintaining temporal consistency across video frames is provided. The method includes: converting the input video from a first frame rate to a second frame rate, wherein the second frame rate is a frame rate faster than the first frame rate; generating processed frames of the input video at the second frame rate; and aggregating the processed frames using a temporal sliding window aggregation to produce a processed output video at a third frame rate.

[0006] According to one embodiment, a system for processing an input video while maintaining temporal consistency across video frames is provided. The system includes a non-transitory computer-readable memory and a processor, wherein the processor is configured to: convert the input video from a first frame rate to a second frame rate, wherein the second frame rate is a frame rate faster than the first frame rate; generate processed frames of the input video at the second frame rate; and aggregate the processed frames using a temporal sliding window aggregation to produce a processed output video at a third frame rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The above and other aspects, features and advantages of certain embodiments of the present disclosure will become more apparent from the following detailed description taken in conjunction with the accompanying drawings, in which:

[0008] Figure 1 shows a flow chart for temporal up-conversion and aggregation according to an embodiment;

[0009] Figure 2 illustrates a frame during up-conversion and aggregation in time according to an embodiment;

[0010] Figure 3 shows an input video frame and a mask for performing video inpainting according to an embodiment;

[0011] Figure 4A illustrates a frame interpolation step for performing video inpainting according to an embodiment;

[0012] Figure 4B Frame-by-frame restoration steps for performing video restoration according to an embodiment are shown;

[0013] Figure 4C illustrates a time aggregation step according to an embodiment; and

[0014] Figure 5 A block diagram showing an electronic device in a network environment according to an embodiment. DETAILED DESCRIPTION

[0015] Hereinafter, embodiments of the present disclosure are described in detail with reference to the accompanying drawings. It should be noted that even if the same elements are shown in different drawings, they will be indicated by the same reference numerals. In the following description, only specific details such as detailed configuration and components are provided to help a comprehensive understanding of the embodiments of the present disclosure. Therefore, it should be clear to those skilled in the art that various changes and modifications of the embodiments described herein may be made without departing from the scope of the present disclosure. In addition, for clarity and conciseness, the description of well-known functions and structures is omitted. The terms described below are defined in consideration of the functions in the present disclosure, and may be different according to the user, the user's intention or custom. Therefore, the definition of the terms should be determined based on the content throughout this specification.

[0016] The present disclosure may have various modifications and various embodiments, wherein embodiments of the various embodiments are described in detail below with reference to the accompanying drawings. However, it should be understood that the present disclosure is not limited to the embodiments, but includes all modifications, equivalents and substitutes within the scope of the present disclosure.

[0017] Although terms including ordinal numbers such as first, second, etc. can be used to describe various elements, structural elements are not limited by terms. Terms are only used to distinguish one element from another element. For example, without departing from the scope of the present disclosure, a first structural element may be referred to as a second structural element. Similarly, a second structural element may also be referred to as a first structural element. As used herein, the term "and / or" includes any combination and all combinations of one or more related items.

[0018] The terms used herein are only used to describe various embodiments of the present disclosure and are not intended to limit the present disclosure. Unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In the present disclosure, it should be understood that the term "including" or "having" indicates the presence of a feature, number, step, operation, structural element, component, or a combination thereof, and does not exclude the possibility of the presence or addition of one or more other features, numbers, steps, operations, structural elements, components, or a combination thereof.

[0019] Unless defined differently, all terms used herein have the same meaning as understood by those skilled in the art to which the present disclosure belongs. Unless clearly defined in the present disclosure, terms (such as those defined in general dictionaries) will be interpreted as having the same meaning as the contextual meaning in the relevant art, and will not be interpreted as having an ideal or overly formal meaning.

[0020] The electronic device according to one embodiment may be one of various types of electronic devices. The electronic device may include, for example, a portable communication device (e.g., a smart phone), a computer, a portable multimedia device, a portable medical device, a camera, a wearable device, or a household appliance. According to one disclosed embodiment, the electronic device is not limited to those described above.

[0021] The terms used in this disclosure are not intended to limit the disclosure, but are intended to include various changes, equivalents or substitutes of corresponding embodiments. Regarding the description of the accompanying drawings, similar reference numerals may be used to represent similar or related elements. Unless the relevant context clearly indicates otherwise, the singular form of the noun corresponding to the item may include one or more of the things. As used herein, each of such phrases such as "A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C" and "at least one of A, B or C" may include all possible combinations of the items listed together in the corresponding phrases in these phrases. As used herein, terms such as "the 1st", "the 2nd", "the first" and "the second" may be used to distinguish a corresponding component from another component, but are not intended to limit the component in other aspects (e.g., importance or order). If an element (e.g., a first element) is referred to as being “coupled with”, “coupled to”, “connected with”, or “connected to” another element (e.g., the second element), with or without the term “operably” or “communicatively”, it is intended to indicate that the element may be coupled with the other element directly (e.g., by wire), wirelessly, or via a third element.

[0022] As used herein, the term "module" may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with other terms (e.g., "logic," "logic block," "component," and "circuit"). A module may be a single integrated component or its smallest unit or part suitable for performing one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application specific integrated circuit (ASIC).

[0023] In an era of deep learning where many algorithms are based on learning algorithms, it can be difficult to collect the data needed to train the algorithm to produce a satisfactory output video. In addition, when the video processing algorithm considers information from neighboring frames, temporal information is often ignored and video processing is still performed frame by frame. In addition, if consecutive frames include significant changes, the processed video may be inconsistent or unrelated in time.

[0024] Multi-frame processing may refer to an algorithm that processes multiple frames in a sequence of frames (i.e., video frames). The frames may have temporal coherence between the frames. However, the frame rate and the speed of the objects in the frames may determine how much change can be observed between consecutive frames.

[0025] Thus, if a sequence of frames is incomplete with missing pixels, multi-frame processing systems and methods may be used to replace the missing regions (ie, regions of the frame corresponding to the missing pixels) with accurate (ie, realistic) values ​​(ie, pixel values).

[0026] The present disclosure provides a system and method including a frame processing algorithm for processing video frames at a higher frame rate and aggregating the results across time to a target frame rate. The system may include an electronic device and / or a processor for performing the operations of the method.

[0027] For example, image inpainting can be an image processing algorithm that can be performed on video frames at a higher frame rate.

[0028] Image inpainting can be defined as the task of replacing a set of missing pixels in an image with actual values. For example, image inpainting can be performed to accomplish watermark removal and / or object removal in a video.

[0029] Image inpainting can be characterized by a two-stage network architecture, where the first stage generates a coarse prediction and the second stage refines the predicted pixel values ​​by considering two streams (i.e., the foreground stream and the background stream of the image).

[0030] In order to improve the features inside the region of the image to be corrected, the repair module can reconstruct the foreground block inside the repair region by obtaining the weighted sum of the background patches outside the repair region. The weight used to determine the weighted sum can be calculated as the correlation between the foreground patch and the background patch. In addition, the correlation operation and the reconstruction operation of the image repair can be performed in parallel by stacking the background patches to perform convolution with the foreground (e.g., unoccluded) area, other background patches, and / or correlation values.

[0031] In addition, by replacing the normal convolution layer of the existing encoder-decoder network with a partial convolution layer, image inpainting can be performed without relying on the placeholder (or position marker) value of the occluded area, which can cause unexpected behavior during inference. The partial convolution layer can take both the input tensor (e.g., data formed as a matrix or vector) and the mask (or mask) as parameters, and calculates the value of each output position as the normalized weighted sum of the unmasked input values.

[0032] Furthermore, image inpainting can successfully reconstruct images by considering two types of losses: reconstruction loss and adversarial loss.

[0033] The reconstruction loss can be used to determine the pixel-by-pixel distance between the predicted image and the full frame of the actual image, and scale the contribution of each pixel based on its distance to the closest known pixel. The adversarial loss can jointly update the discriminator with the image inpainting model. The reconstruction loss and the adversarial loss can be jointly optimized for improved image inpainting.

[0034] When considering large, continuous occluded regions of an image, such as consecutive frames in a video sequence, it may be beneficial to identify context frames that contain similar information in corresponding regions of the image from one frame to the next.

[0035] Image inpainting can be extended to consider a video consisting of multiple image frames (e.g., consecutive frames) in the temporal direction by reconstructing features of a reference frame based on the reference frame (e.g., a specific frame in a video sequence) itself and multiple context frames. To achieve this, background patches can be extracted from both the reference frame and the context frame and convolved with the foreground region of the reference frame. The result can then be normalized based on weights assigned according to the location within the image.

[0036] However, with respect to video restoration, the added temporal dimension introduces additional criteria that must be met: for example, the content between adjacent frames should be consistent and the apparent motion between them should be realistic.

[0037] The present disclosure provides systems and methods for performing image processing (eg, inpainting) on ​​video frames that take into account the temporal characteristics of individual frames without having to rely on complex time-dependent algorithms to perform the image processing.

[0038] Figure 1 A flow chart for temporal up-conversion and aggregation according to an embodiment is shown.

[0039] Reference Figure 1 In step 101, the system increases the frame rate of the occluded input video using a frame interpolation method, which temporally upsamples the unprocessed video to generate an interpolated video with a high frame rate.

[0040] In step 102, the system applies a frame-by-frame processing algorithm to the high frame rate interpolated video to generate a high frame rate output video. Each frame in the high frame rate interpolated video is independently processed using frame-by-frame processing (eg, an image restoration application or an application for applying a style to each frame).

[0041] In step 103 , frames in the output video at a high frame rate are aligned and aggregated within overlapping windows of a suitable step size (eg, a plurality of frames) to obtain an output video at a target frame rate.

[0042] Can be viewed by Figure 2 The frames shown are used to visualize step 101 , step 102 and step 103 .

[0043] Figure 2 Frames during up-conversion and aggregation in time are shown according to an embodiment.

[0044] Figure 2 Steps 201, 202 and 203 in the embodiment may correspond to Figure 1 Steps 101, 102 and 103 in FIG.

[0045] Reference Figure 2 , an unprocessed video V running at an original frame rate R_o may be provided as input, and after performing steps 201 , 202 , and 203 , a processed video Y running at a target frame rate R_t may be provided as output.

[0046] In step 201 , a frame rate up conversion (FRUC) is performed to upsample the frame rate of an input unprocessed video V from R_o to R_u, where R_u is a higher video frame rate than R_o.

[0047] Temporal up-conversion (e.g., FRUC) can interpolate changes in illumination, color, and motion between consecutive frames and produce intermediate frames with less drastic changes in motion, illumination, and / or color than the consecutive frames. In addition, the interpolated frames can carry temporal information from left (L) and right (R) (e.g., forward and backward) reference frames.

[0048] FRUC can be performed using conventional frame interpolation techniques. Additionally or alternatively, FRUC can be performed using deep learning techniques by predicting a pair of optical flows and thereby predicting a warping grid for warping the input frame to a specific time step.

[0049] For example, the forward optical flow and the backward optical flow (e.g., the optical flow in equations (1) and (2) below) between a first frame (e.g., the L frame) at a first time step and a second frame (e.g., the R frame) at a second time step may be estimated. _t->L and O _t->R ), and estimating a forward deformed mesh and a backward deformed mesh between a first frame at a first time step and a second frame at a second time step to perform FRUC. The forward optical flow and the forward deformed mesh may correspond to a forward direction, and the backward optical flow and the backward deformed mesh may correspond to a backward direction.

[0050] The optical flow can be interpolated for each intermediate frame between the first frame at the first time step and the second frame at the second time step. The second time step can be any time step relative to the first time step. According to equations (1) and (2), the interpolated optical flow O _t->L and O _t->R can be estimated as the forward optical flow O estimated from L frames _L->R and the backward optical flow O estimated from the R frame _R->L The weighted sum of:

[0051] O _t->L = -(1-t) t × O _L->R – t 2 O _R->L (1)

[0052] O _t->R = (1-t 2 ) _L->R – t (1-t) O _R->L (2)

[0053] In Equation (1) and Equation (2), the interpolated optical flow from the left frame is O _t->L , the interpolated optical flow from the right frame is O _t->R . We can use the interpolated optical flow O _t->Land O _t->R The resulting deformed mesh deforms the left and right frames to give a deformed frame F interpolated in the forward (e.g., from L) direction _t (L) and the deformed frame F interpolated in the backward (e.g., from R) direction _t (R).

[0054] In addition, a weight mask can be estimated and used to transform the frame F from the L direction _t (L) and the deformed frame F from the R direction _t (R) is summed, so the frame F at the intermediate time step t is defined by equation (3) _t :

[0055] F _t = w _L ×F _t (L) + w _R ×F _t (R) (3)

[0056] In equation (3), w _L is the weight mask used to sum the warped frames from the left direction, w _R is the weight mask used to sum the warped frames from the right direction.

[0057] In step 202, frame-by-frame conversion is performed on the interpolated frames.

[0058] Frame-by-frame conversion (e.g., processing) of intermediate (e.g., interpolated) frames may produce results that are different from those obtained on reference non-interpolated frames because the intermediate frames may carry characteristics and / or features from left and right frames (e.g., context frames) because the intermediate frames are obtained from interpolation in both the forward and backward directions.

[0059] To perform frame-by-frame processing, the temporal interpolation frame V S The composed high frame rate video may be processed at time t using a frame-by-frame algorithm including one or more frame-by-frame processing techniques, such that at least one unprocessed interpolated frame F_t is processed into at least one processed frame P(F_t) by the frame-by-frame algorithm P. _t ).

[0060] The frame-by-frame algorithm P can be a number of different types of frame-by-frame image processing algorithms. For example, the frame-by-frame algorithm P can be an inpainting algorithm that can inpaint each frame independently using a contextual attention inpainting network. Additionally or alternatively, the frame-by-frame algorithm P can be an algorithm that can apply a particular style to the interpolated frames (such as a blur removal algorithm or a darkness removal algorithm). Many different types of frame-by-frame algorithms can be applied to the upsampled video frames V S .

[0061] In step 203, frame rate down-conversion is performed by aggregating frames. Specifically, the processed high frame rate video Y can be down-converted by temporally combining the interpolated processed frames using sliding window aggregation. S The temporal aggregation is performed to produce a processed video Y at a target frame rate R_t. The target frame rate R_t may be the original frame rate R_o.

[0062] In the temporal aggregation process, the high frame rate video Y from the processing S Information of the processed interpolated frames can be carried backward to the previous (ie, left) reference frame and forward to the next (ie, right) reference frame, thereby improving the temporal consistency between the processed frames at the target frame rate R_t after aggregation.

[0063] The processed video Y at the target frame rate R_t may be a combination of processed frames when t corresponds to a temporal interpolation around a target frame given a particular upsampling rate u (eg, corresponding to a particular time step).

[0064] The processed video can be aggregated to the target frame rate R_t according to equation (4):

[0065] P(F _L ) = Agg( P(F _{uL-m} ), P(F _{uL-m+1} ), …, P(F _{uL} ), P(F _{uL+1}), ) .. P(F _{uL+m} ) ) (4)

[0066] In equation (4), m may be a function of the window size, such that "2m+1" is the window size for aggregation, and b may represent a number between -m and m to indicate the index of the processed superframe (e.g., upsampled frame) in the target window. The aggregation function (Agg()) may calculate each processed frame P(F _{uL+b} ) to the processed target frame P(F _{uL} ) between the optical flow O _(uL+b)->uL The optical flow O can be calculated using _(uL+b)->uL The interpolated frame F _(uL+b) Warp to target frame F _uL , thus generating the deformed frame W _(uL+b)->uL . This can be done based on equation (5) by using the weight r _b Sum the deformed frames to get the processed frame P(F _L ) to aggregate over time:

[0067] P(F _L ) = SUM_( {-m:m}中的b) (r _b W _(uL+b)->uL ) (5)

[0068] In equation (5), for frames that are farther from the target frame or have a larger absolute value of b, the weight r _b Can have a smaller weight value.

[0069] Different types of frame-by-frame processing may be used to process the interpolated frames. For example, image inpainting may be a single-frame process in which spatial holes (collections or regions of pixels) in an image are replaced with actual values. Additionally or alternatively, style transfer processing, stylization processing, super-resolution processing, denoising processing, enhancement processing, sharpening processing, inpainting processing, and shadow reduction processing may be types of frame-by-frame processing used to process the interpolated frames.

[0070] Video inpainting is another type of process applicable to the present disclosure. Video inpainting is an example of a video multi-frame process that can replace spatio-temporal holes in a video with actual and plausible content (ie, value).

[0071] Compared to image inpainting, video inpainting, which performs processing between frames in time, is more complex. For example, the information replaced in one frame should be temporally consistent with similar information replaced in adjacent frames. This becomes particularly challenging when there is significant motion between frames.

[0072] Figure 3 An input video frame and a mask for performing video inpainting according to an embodiment are shown.

[0073] Reference Figure 3 To perform video inpainting, each of the unprocessed occluded input video frames 301 may be accompanied by a mask 302 (e.g., an occlusion mask). The mask 302 corresponds to the region of the input video frame 301 to be inpainted. For each time step in the input video, the mask 302 may exist for each input video frame 301. In addition, the mask 302 may be upsampled using the input video frame (e.g., step 101), so that video inpainting can be performed at a high upsampled frame rate R_u.

[0074] Accordingly, FIG. 4A to FIG. 4C A video repair is shown, where Figure 4A Corresponding to step 101, Figure 4B Corresponding to step 102, Figure 4C Corresponding to step 103.

[0075] Figure 4A A frame interpolation step for performing video inpainting according to an embodiment is shown.

[0076] Reference Figure 4A, V represents the input video (i.e., red green blue (RGB) video) and can be represented by {v 1 , . . . ,v N} frame, X represents the occlusion mask and can be represented by {x 1 , . . . ,x N} frame composition. 1 , . . . , v N} and X{x 1 , .. . ,x N} together indicate the area to be inpainted when pixels are occluded in the input video V. In addition, occluded pixels may have placeholder values ​​to identify their location in the frame.

[0077] To perform video restoration techniques, the controller may accept V{v 1 , . . . , v N} and X{x 1 , . . . ,x N} as input, and output Y = { Y1 , . . . , YN}, where Y = { Y1 , . . . , YN} including V{v 1 , . . ., v N}same value.

[0078] For example, video inpainting techniques can be performed by applying the same deformation operation to the video V{v 1 , . . . ,v N} and included in the mask X{x 1 , . . . ,x N}, we artificially increase the occlusion video V{v 1 , . . . , v N} and the occlusion mask X{x 1 , . . . ,x N} frame rate. Figure 4A As shown in , it is possible to target occluded video V at high frames per second (FPS) S Generate a high FPS mask X for each frame S .

[0079] Additionally or alternatively, to perform temporal upsampling, a frame upconversion method (such as Super SloMo, FRUC, and / or frame interpolation) may be used to insert multiple frames between each pair of frames in the occlusion video V and the occlusion mask X to create a high FPS version of the occlusion video V S and the occlusion mask X S . High FPS version of the occlusion video V S It can be obtained by {v s1 , . . . , v sN} frame, and the occlusion mask of the high FPS version (denoted as X S ) can be represented by {x s1 , . . . , x sN}Frame composition.

[0080] The frame up-conversion method may be able to generate multiple intermediate frames among each pair of frames in the occlusion video V and the occlusion mask X. In addition, for a given time step, a pair of optical flow deformation grids may be used to deform the frames in the occlusion video V and the occlusion mask X to that time step.

[0081] Figure 4B Frame-by-frame restoration steps for performing video restoration according to an embodiment are shown.

[0082] Reference Figure 4B , after obtaining the high FPS version of the occlusion video V S and the occlusion mask X S Afterwards, each frame of the high FPS occluded video can be repaired independently to produce a high FPS output video Y S In one example, after obtaining the occlusion video V S and the occlusion mask X S Afterwards, the occlusion mask X S With occlusion video V S Each frame of the corresponding frame independently repairs the occluded video V S Each frame is used to generate the output video Y S .

[0083] Inpainting frames of occluded video at high FPS independently advantageously uses less processing power than temporal video inpainting, which must consider differences between frames based on time. However, the video inpainting embodiments of the present disclosure do consider the temporal factors between frames when aggregating frames of a video at high FPS.

[0084] Figure 4C The temporal aggregation steps according to an embodiment are shown.

[0085] like Figure 4CAs shown in , when aggregating processed frames (e.g., after performing restoration on interpolated frames or after performing another type of image processing on interpolated frames), temporal aggregation can be applied. Specifically, when generating a high FPS output video Y S After that, the high FPS output video Y S The frames in can be aligned and averaged over a sliding window with an appropriate step size (ie, length or number of frames) to produce frames of the final output video Y.

[0086] To perform temporal aggregation, the processed frames can be output in the high FPS output video Y S The overlapping windows of a predetermined step size (i.e., length or number of frames) are aligned and averaged to obtain the output Y = { Y1 , . . . , YN}.

[0087] Specifically, the output video Y at high FPS can be S At least one reference frame 501 is confirmed in the context frame 501. At least one context frame 502 surrounds the reference frame 501 before or after the reference frame 501 in the time direction.

[0088] like Figure 4C As shown in , a sliding window 503 having a length of five reference frames (including one reference frame 501 and four context frames) is selected. The sliding window 503 can be oriented relative to the reference frame 501 so that the reference frame is aligned (e.g., centered) within the window 503. That is, when aligned, the same number of context frames can be located on the left and right sides of the reference frame 501.

[0089] exist Figure 4C In the output video Y of high FPS, every three frames S The reference frame 501 is confirmed once in the sliding window 503. Therefore, because the number of frames included in the sliding window 503 (e.g., five frames) is greater than the number of frames between the reference frames (e.g., two frames), at least some of the context frames can be aligned with more than one reference frame. Aligning the same context frame with respect to the reference frame in the forward direction and with respect to the reference frame in the backward direction improves image consistency and smoothness between frames after aggregation.

[0090] Output video Y at high FPS SAfter the frames have been aligned within each sliding window, the optical flow between each frame in the sliding window 503 can be estimated, and the frames can then be warped based on the optical flow. The final output frame included in the output video Y can be determined by averaging the warped frame (e.g., the warped context frame) with the reference frame 501. In one embodiment, the first set of processed frames can be aligned and averaged within the temporal sliding window, and the second set of processed frames can be aligned and averaged within the temporal sliding window, wherein the first set of processed frames and the second set of processed frames include at least one shared context frame.

[0091] Therefore, according to embodiments of the present disclosure, additional (ie, post) processing may not be necessary to enforce temporal consistency between video frames.

[0092] Figure 5 FIG. 5 is a block diagram showing an electronic device 501 in a network environment 500 according to an embodiment. Figure 5 , the electronic device 501 under the network environment 500 may communicate with another electronic device 502 via a first network 598 (e.g., a short-range wireless communication network), or communicate with another electronic device 504 or a server 508 via a second network 599 (e.g., a long-range wireless communication network). The electronic device 501 may also communicate with the electronic device 504 via the server 508. The electronic device 501 may include a processor 520, a memory 530, an input device 550, a sound output device 555, a display device 560, an audio module 570, a sensor module 576, an interface 577, a haptic module 579, a camera module 580, a power management module 588, a battery 589, a communication module 590, a user identification module (SIM) 596, and / or an antenna module 597. In one embodiment, at least one of the components (e.g., the display device 560 or the camera module 580) may be omitted from the electronic device 501, or one or more other components may be added to the electronic device 501. In one embodiment, some of the plurality of components may be implemented as a single integrated circuit (IC). For example, the sensor module 576 (eg, a fingerprint sensor, an iris sensor, or an illumination sensor) may be embedded in the display device 560 (eg, a display).

[0093] The processor 520 may execute, for example, software (e.g., program 540) to control at least one other component (e.g., hardware or software component) of the electronic device 501 in combination with the processor 520, and may perform various data processing or calculations. As at least part of the data processing or calculation, the processor 520 may load a command or data received from another component (e.g., sensor module 576 or communication module 590) in the volatile memory 532, process the command or data stored in the volatile memory 532, and store the resultant data in the non-volatile memory 534. The processor 520 may include a main processor 521 (e.g., a central processing unit (CPU) or an application processor (AP)), and an auxiliary processor 510 (e.g., a graphics processor (GPU), an image signal processor (ISP), a sensor hub processor, or a communication processor (CP)) that may operate independently of the main processor 521 or in combination with the main processor 521. Additionally or alternatively, the auxiliary processor 510 may be adapted to consume less power than the main processor 521, or to perform a specific function. The auxiliary processor 510 may be implemented separately from the main processor 521 or as a part of the main processor 521 .

[0094] The auxiliary processor 510 may control at least some of the functions or states related to at least one of the components of the electronic device 501 (e.g., the display device 560, the sensor module 576, or the communication module 590) instead of the main processor 521 when the main processor 521 is in an inactive (e.g., sleep) state, or control at least some of the functions or states related to at least one of the components of the electronic device 501 (e.g., the display device 560, the sensor module 576, or the communication module 590) together with the main processor 521 when the main processor 521 is in an active state (e.g., executing an application). According to one embodiment, the auxiliary processor 510 (e.g., ISP or CP) may be implemented as a part of another component (e.g., camera module 580 or communication module 590) that is functionally related to the auxiliary processor 510.

[0095] The memory 530 may store various data used by at least one component of the electronic device 501 (e.g., the processor 520 or the sensor module 576). The various data may include, for example, software (e.g., the program 540) and input data or output data for commands related thereto. The memory 530 may include a volatile memory 532 or a nonvolatile memory 534.

[0096] The program 540 may be stored as software in the memory 530 , and may include, for example, an operating system (OS) 542 , middleware 544 , or an application 546 .

[0097] The input device 550 may receive commands or data to be used by other components (eg, the processor 520) of the electronic device 501 from outside (eg, a user) of the electronic device 501. The input device 550 may include, for example, a microphone, a mouse, or a keyboard.

[0098] The sound output device 555 can output a sound signal to the outside of the electronic device 501. The sound output device 555 may include, for example, a speaker or a receiver. The speaker can be used for general purposes (such as playing multimedia or recording), and the receiver can be used to receive an incoming call. According to one embodiment, the receiver can be implemented as a part of the speaker or as a separate speaker.

[0099] The display device 560 may visually provide information to the outside of the electronic device 501 (e.g., a user). The display device 560 may include, for example, a display, a holographic device, or a projector, and a control circuit for controlling a corresponding one of the display, the holographic device, and the projector. According to one embodiment, the display device 560 may include a touch circuit adapted to detect a touch, or a sensor circuit (e.g., a pressure sensor) adapted to measure the strength of a force caused by a touch.

[0100] The audio module 570 may convert sound into an electrical signal and vice versa. According to one embodiment, the audio module 570 may obtain sound via the input device 550 or output sound via the sound output device 555 or an earphone of an external electronic device 502 directly (eg, wired) or wirelessly coupled to the electronic device 501.

[0101] The sensor module 576 may detect an operating state (e.g., power or temperature) of the electronic device 501 or an environmental state (e.g., a state of a user) outside the electronic device 501, and then generate an electrical signal or data value corresponding to the detected state. The sensor module 576 may include, for example, a gesture sensor, a gyroscope sensor, an atmospheric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an infrared (IR) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illumination sensor.

[0102] The interface 577 may support one or more designated protocols to be used to directly (e.g., wired) or wirelessly combine the electronic device 501 with the external electronic device 502. According to one embodiment, the interface 577 may include, for example, a high-definition multimedia interface (HDMI), a universal serial bus (USB) interface, a secure digital (SD) card interface, or an audio interface.

[0103] The connection terminal 578 may include a connector via which the electronic device 501 may be physically connected to the external electronic device 502. According to one embodiment, the connection terminal 578 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (eg, a headphone connector).

[0104] The haptic module 579 may convert the electrical signal into mechanical stimulation (eg, vibration or movement) or electrical stimulation that can be recognized by the user via tactile or kinesthetic sense. According to one embodiment, the haptic module 579 may include, for example, a motor, a piezoelectric element, or an electrical stimulator.

[0105] The camera module 580 may capture a still image or a moving image. According to one embodiment, the camera module 580 may include one or more lenses, an image sensor, an ISP, or a flash.

[0106] The power management module 588 may manage power supplied to the electronic device 501. The power management module 588 may be implemented as, for example, at least a portion of a power management integrated circuit (PMIC).

[0107] The battery 589 may supply power to at least one component of the electronic device 501. According to one embodiment, the battery 589 may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0108] The communication module 590 may support establishing a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device 501 and an external electronic device (e.g., electronic device 502, electronic device 504, or server 508), and perform communication via the established communication channel. The communication module 590 may include one or more CPs that may operate independently of the processor 520 (e.g., AP) and support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module 590 may include a wireless communication module 592 (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 594 (e.g., a local area network (LAN) communication module or a power line communication (PLC) module). The corresponding communication modules among these communication modules may communicate via a first network 598 (e.g., a short-range communication network (such as, Bluetooth TM, Wireless Fidelity (Wi-Fi) Direct, or Infrared Data Association (IrDA) standards)) or a second network 599 (e.g., a long-distance communication network (such as a cellular network, the Internet) or a computer network (e.g., a LAN or a wide area network (WAN))) to communicate with an external electronic device. These various types of communication modules may be implemented as a single component (e.g., a single IC), or may be implemented as multiple components separated from each other (e.g., multiple ICs). The wireless communication module 592 may use user information (e.g., an International Mobile Subscriber Identity (IMSI)) stored in the user identification module 596 to identify and authenticate the electronic device 501 in a communication network (such as, the first network 598 or the second network 599).

[0109] The antenna module 597 may transmit or receive a signal or power to or from the outside of the electronic device 501 (e.g., an external electronic device). According to one embodiment, the antenna module 597 may include one or more antennas, and thus at least one antenna suitable for a communication scheme used in a communication network (such as the first network 598 or the second network 599) may be selected, for example, by the communication module 590 (e.g., the wireless communication module 592). The signal or power may then be transmitted or received between the communication module 590 and the external electronic device via the selected at least one antenna.

[0110] At least some of the above components may be coupled to each other and transmit signals (eg, commands or data) therebetween via an inter-peripheral communication scheme (eg, a bus, a general purpose input output (GPIO), a serial peripheral interface (SPI), or a mobile industry processor interface (MIPI)).

[0111] According to one embodiment, commands or data may be sent or received between the electronic device 501 and the external electronic device 504 via a server 508 combined with the second network 599. Each of the electronic devices 502 and 504 may be a device of the same type or a different type as the electronic device 501. All or some of the operations to be performed at the electronic device 501 may be performed at one or more of the external electronic devices 502, 504, or the server 508. For example, if the electronic device 501 should automatically perform a function or service or perform a function or service in response to a request from a user or another device, the electronic device 501 may request one or more external electronic devices to perform at least a portion of the function or service instead of or in addition to performing the function or service. The one or more external electronic devices receiving the request may perform at least a portion of the requested function or service, or an additional function or additional service related to the request, and transmit the result of the execution to the electronic device 501. The electronic device 501 may provide the result as at least a part of the reply to the request with or without further processing the result. To this end, for example, cloud computing, distributed computing, or client-server computing technology may be used.

[0112] One embodiment may be implemented as software (e.g., program 540) including one or more instructions stored in a storage medium (e.g., internal memory 536 or external memory 538) readable by a machine (e.g., electronic device 501). For example, the processor of the electronic device 501 may call at least one of the one or more instructions stored in the storage medium, and execute the at least one instruction under the control of the processor with or without one or more other components. Thus, the machine may be operated to perform at least one function according to the at least one instruction called. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. The term "non-transitory" indicates that the storage medium is a tangible device and does not include signals (e.g., electromagnetic waves), but this term does not distinguish between data being semi-permanently stored in the storage medium and data being temporarily stored in the storage medium.

[0113] According to one embodiment, the disclosed method may be included and provided in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., a compact disc read-only memory (CD-ROM)), or via an application store (e.g., Play Store TM) or distributed online (e.g., downloaded or uploaded) directly between two user devices (e.g., smart phones). If distributed online, at least a portion of the computer program product may be temporarily generated or at least temporarily stored in a machine-readable storage medium (such as a memory of a manufacturer's server, an application store's server, or a relay server).

[0114] According to one embodiment, each of the above-mentioned components (e.g., a module or a program) may include a single entity or multiple entities. One or more of the above-mentioned components may be omitted, or one or more other components may be added. Alternatively or optionally, multiple components (e.g., modules or programs) may be integrated into a single component. In this case, the integrated component may still perform one or more functions of each of the multiple components in the same or similar manner as the one or more functions of each of the multiple components are performed by the corresponding components in the multiple components before integration. The operations performed by a module, a program, or another component may be performed sequentially, in parallel, repeatedly, or heuristically, or one or more of the operations may be performed in a different order or omitted, or one or more other operations may be added.

[0115] Although specific embodiments of the present disclosure have been described in the detailed description of the present disclosure, the present disclosure may be modified in various forms without departing from the scope of the present disclosure. Therefore, the scope of the present disclosure should not be determined based solely on the described embodiments, but on the attached claims and their equivalents.

Claims

1. A method for processing an input video while maintaining temporal consistency between video frames, the method comprising: Converting an input video from a first frame rate to a second frame rate, wherein the second frame rate is a higher frame rate than the first frame rate; generating processed frames of the input video at a second frame rate; and aggregating the processed frames using a temporal sliding window aggregation to produce a processed output video at a third frame rate, The step of aggregating the processed frames using time sliding window aggregation includes: aligning and averaging the first set of processed frames within a temporal sliding window oriented relative to a first reference frame; and aligning and averaging a second set of processed frames within a temporal sliding window oriented relative to a second reference frame, The first group of processed frames and the second group of processed frames include at least one shared context frame.

2. The method according to claim 1, wherein: The step of converting an input video from a first frame rate to a second frame rate comprises: An intermediate frame is determined by computing a weighted sum of a first video frame at a first time step and a second video frame at a second time step.

3. The method according to claim 2, wherein: The step of calculating a weighted sum of a first video frame at a first time step and a second video frame at a second time step comprises: calculating a forward deformation grid from the first video frame to a third time step corresponding to the intermediate frame, and obtaining a deformation frame in a forward direction based on the forward deformation grid; calculating a backward deformation grid from the second video frame to a third time step corresponding to the intermediate frame, and obtaining a deformed frame in a backward direction based on the backward deformation grid; An intermediate frame at a third time step is determined by performing a weighted sum of the deformed frames in the forward direction and the deformed frames in the backward direction.

4. The method according to claim 1, wherein: The step of converting an input video from a first frame rate to a second frame rate comprises: converting both the input video and the occlusion mask from a first frame rate to a second frame rate, Therein, the occlusion mask indicates the area of ​​the input video that will be processed.

5. The method according to claim 4, wherein: The step of generating processed frames of the input video at a second frame rate comprises: Using an occlusion mask at the second frame rate corresponding to each frame of the input video at the second frame rate, each frame of the input video at the second frame rate is independently processed, wherein the step of generating the processed frames includes: at least one of enhancement processing, sharpening processing, denoising processing, super-resolution processing and restoration processing.

6. The method according to claim 1, wherein: The step of aggregating the processed frames using a temporal sliding window aggregation further includes: selecting a temporal sliding window having a predetermined step size; and At least some of the processed frames are aligned and averaged within the temporal sliding window.

7. The method according to claim 6, wherein: The processed frames aligned and averaged within the temporal sliding window include: a reference frame and at least two context frames.

8. The method according to claim 7, wherein: The at least two context frames include: at least one context frame located before the reference frame and at least one context frame located after the reference frame.

9. The method according to any one of claims 1 to 8, wherein: The first frame rate is equal to the third frame rate.

10. A system for processing an input video while maintaining temporal consistency between video frames, the system comprising: A non-transitory computer readable memory and a processor, wherein the processor, when executing instructions stored in the non-transitory computer readable memory, is configured to: Converting an input video from a first frame rate to a second frame rate, wherein the second frame rate is a higher frame rate than the first frame rate; generating processed frames of the input video at a second frame rate; and aggregating the processed frames using a temporal sliding window aggregation to produce a processed output video at a third frame rate, Wherein, when the processed frames are aggregated using time sliding window aggregation, the processor is further configured to: aligning and averaging the first set of processed frames within a temporal sliding window oriented relative to a first reference frame; and aligning and averaging a second set of processed frames within a temporal sliding window oriented relative to a second reference frame, The first group of processed frames and the second group of processed frames include at least one shared context frame.

11. The system according to claim 10, wherein: When converting the input video from the first frame rate to the second frame rate, the processor is further configured to: An intermediate frame is determined by computing a weighted sum of a first video frame at a first time step and a second video frame at a second time step.

12. The system according to claim 11, wherein: When calculating the weighted sum of the first video frame at the first time step and the second video frame at the second time step, the processor is further configured to: calculating a forward deformation grid from the first video frame to a third time step corresponding to the intermediate frame, and obtaining a deformation frame in a forward direction based on the forward deformation grid; calculating a backward deformation grid from the second video frame to a third time step corresponding to the intermediate frame, and obtaining a deformed frame in a backward direction based on the backward deformation grid; An intermediate frame at a third time step is determined by performing a weighted sum of the deformed frames in the forward direction and the deformed frames in the backward direction.

13. The system according to claim 10, wherein: When converting the input video from the first frame rate to the second frame rate, the processor is further configured to: converting both the input video and the occlusion mask from a first frame rate to a second frame rate, Therein, the occlusion mask indicates the area of ​​the input video that will be processed.

14. The system according to claim 13, wherein: When generating processed frames of the input video at the second frame rate, the processor is further configured to: Using an occlusion mask at the second frame rate corresponding to each frame of the input video at the second frame rate, each frame of the input video at the second frame rate is independently processed, wherein the step of generating the processed frames includes: at least one of enhancement processing, sharpening processing, denoising processing, super-resolution processing and restoration processing.

15. The system according to claim 10, wherein: When aggregating the processed frames using a temporal sliding window aggregation, the processor is further configured to: selecting a temporal sliding window having a predetermined step size; and At least some of the processed frames are aligned and averaged within the temporal sliding window.

16. The system of claim 15, wherein: The processed frames aligned and averaged within the temporal sliding window include: a reference frame and at least two context frames.

17. The system of claim 16, wherein: The at least two context frames include: at least one context frame located before the reference frame and at least one context frame located after the reference frame.

18. A system according to any one of claims 10 to 17, wherein: The first frame rate is equal to the third frame rate.

Images