Method and electronic device for tilt correction of video

Abstract

A method for tilt correction of video by an electronic device. The method may include receiving a user selection of a video. The method may include identifying a sequence of image frames of a video representing a scene in the video. The method may include determining an inclination in each image frame of the sequence of image frames. The method may include determining whether a tilt cause in a sequence of image frames is unintentional by analyzing a tilt in each image frame using a trained artificial intelligence (AI) model. The method may include performing tilt correction on the sequence of image frames based on the cause of tilt being unintentional.

Description

Technical Field

[0001] Certain example embodiments relate to an electronic device, and for example, to a method and / or an electronic device for tilt correction of a video. Background Art

[0002] The user intentionally or unintentionally changes the orientation of the camera away from a reference plane (e.g., a horizontal plane) when capturing an image / video, which results in a tilt in the image / video. One goal of intentionally creating a tilt in an image / video (e.g., intentional tilt) is to improve the aesthetics of the image / video. However, the tilt in the image / video (e.g., unintentional tilt) may be attributed to human error in using the camera, which is undesirable. In addition, the unintentional tilt in the image / video caused by human error greatly reduces the aesthetics of the image / video.

[0003] Figures 1(a) and 1(b) illustrate tilt in an image. As shown in (11), images with pitch confuse existing heuristic-based methods, which detect tilt in images poorly by understanding visuals at the object level and due to the presence of clustered vertical / horizontal lines in the image. In addition, existing methods cannot effectively distinguish between intentional and unintentional tilt in images / videos. As shown in (12), existing heuristic-based methods also fail to understand imaginary horizontal and vertical references and detect tilt in images where orthogonality is a property shared by only a few elements in an image. As shown in (13), objects in the foreground of an image have higher importance to the image. In some other images, objects in the background of the image may have higher importance to the image. In existing heuristic-based methods, applying tilt correction by understanding important objects in the image is neglected.

[0004] A user can manually correct the tilt in an image, as shown in (15, 16), where (14) is a ground truth image. Manually fixing such a tilt in an image is cumbersome and time-consuming. Typically, a user manually rotates and / or crops an image frame to correct the tilt in the image frame, which results in data loss of the image frame, as shown in (16). Alternatively, a user can only manually rotate the image frame to correct the tilt in the image frame, thereby limiting data loss. However, as shown in (15), black patches (or black fill) are formed at the four corners of the image frame due to the rotation of the image frame. Consider that the tilt in each image frame of the video will be different. Therefore, when manually correcting the tilt, the rotation and / or cropping on the image frame of the video will also be uneven, which results in zoom-in / zoom-out jitter in the video, which gives the user an unsteady viewing experience and reduces the aesthetics of the video. Therefore, it is desirable to provide a useful solution for correcting unintentional tilt in images / videos. Summary of the invention

[0005] Technical Solutions

[0006] Certain example embodiments provide a method and / or electronic device for tilt correction of a video, wherein the electronic device can make the tilt angle as close to 0° as possible for each scene span without generating zoom-in / zoom-out jitter in the video during tilt correction, which improves the video aesthetics.

[0007] Certain example embodiments may perform tilt correction by considering the relative spatial positioning of objects, the correlation of objects, and the relative relationship between objects.

[0008] Certain example embodiments may use a self-attention model to accurately predict tilt angles in image frames.

[0009] Certain example embodiments may minimize or reduce pixel and data loss by applying outer padding on black patches formed during deflection correction and center cropping image frames, maintaining uniform resolution of image frames and maintaining smooth transitions between image frames after deflection correction.

[0010] Therefore, some example embodiments herein may provide a method for tilt correction of a video by an electronic device. The method may include receiving, by an electronic device, a user selection of a video. The method may include identifying, by an electronic device, a sequence of image frames of a video representing a scene in the video. The method may include determining, by the electronic device, the tilt in each image frame in the sequence of image frames. The method may include determining, by the electronic device, whether the cause of the tilt in the sequence of image frames is unintentional by analyzing the tilt in each image frame using a trained artificial intelligence (AI) model. The method may include performing, by the electronic device, the tilt correction on the sequence of image frames based on the cause of the tilt being unintentional.

[0011] In an example embodiment, determining by the electronic device whether the cause of the tilt in the image frame sequence is unintentional by analyzing the tilt in each image frame using a trained AI model may include: determining by the electronic device using the trained AI model whether the tilt in each image frame satisfies (e.g., or "corresponds to" or "fits" or "satisfies") a threshold condition; and determining by the electronic device that the cause of the tilt in the image frame sequence is unintentional based on the tilt in each image frame satisfying the threshold condition.

[0012] In an example embodiment, performing tilt correction on a sequence of image frames by an electronic device may include: estimating, by the electronic device, the tilt angle of each image frame for maximum / maximum tilt correction at each image frame; determining, by the electronic device, a tilt angle deviation for each image frame, wherein the tilt angle deviation of an image frame in the sequence of image frames is the absolute value of the difference between the tilt angle of the image frame and an average value of the tilt angles of the image frame and adjacent image frames in the sequence of image frames; estimating, by the electronic device, a pixel shift and loss in each image frame and an area to be cropped on each image frame based on the tilt angle deviation and the tilt angle; determining, by the electronic device, a target tilt correction required for each image frame based on the pixel shift and loss; applying, by the electronic device, the target tilt correction, and at least one of cropping and out-painting, to the determined area of ​​each image frame; splicing, by the electronic device, each tilt-corrected image frame to reconstruct the scene; and generating a video by sequentially joining the spliced ​​image frames.

[0013] In an example embodiment, estimating, by an electronic device, the tilt angle of each image frame for maximum / maximum tilt correction at each image frame may include: predicting, by the electronic device, a range of possible tilt angles for each image frame using a trained self-attention model; and determining, by the electronic device, the tilt angle by determining a weighted average of the predicted tilt angles having a standard deviation of zero or greater.

[0014] In an example embodiment, a self-attention model may be trained for tilt angle prediction by an electronic device determining a range of possible tilt angles of a training image frame based on relative positions of objects in the training image frame, correlations of objects, and relationships between objects; generating training data by the electronic device using the range of possible tilt angles; generating an output vector with distribution probabilities by the electronic device performing multi-label training on the training data; and training the self-attention model using the output vector using an S-shaped activation for tilt angle prediction.

[0015] In an example embodiment, estimating, by an electronic device, pixel shift and loss in each image frame and an area to be cropped on each image frame based on a tilt angle deviation and a tilt angle may include: identifying, by the electronic device, a small (e.g., minimum) tilt angle of an image frame sequence based on the tilt angle of each image frame; determining, by the electronic device, an average tilt angle deviation of the image frame sequence based on the tilt angle deviation of each image frame; determining, by the electronic device, whether the small (e.g., minimum) tilt angle is greater than the average tilt angle deviation; performing, by the electronic device, at least one of the following: setting the average tilt angle deviation to a small (e.g., minimum) tilt angle, and estimating an angle for correcting the tilt of each image frame based on the small (e.g., minimum) tilt angle being greater than the average tilt angle deviation, and estimating an angle for correcting the tilt of each image frame based on the small (e.g., minimum) tilt angle being not greater than the average tilt angle deviation; and determining, by the electronic device, an area to be cropped on each image frame based on the average tilt angle deviation, wherein a target tilt correction has minimal or reduced pixel shift and loss.

[0016] In an example embodiment, estimating the angle used to correct the tilt of each image frame may include: determining, by an electronic device, whether the tilt angle of an image frame in an image frame sequence is one of equal to zero, less than or equal to an average tilt angle deviation, and greater than an average tilt angle deviation; and performing, by the electronic device, at least one of the following: estimating the angle used to correct the tilt of the image frame to be zero based on the tilt angle of the image frame being equal to zero, estimating the angle used to correct the tilt of the image frame to be the tilt angle based on the tilt angle of the image frame being less than or equal to the average tilt angle deviation, and estimating the angle used to correct the tilt of the image frame to be the average tilt angle deviation based on the tilt angle of the image frame being greater than the average tilt angle deviation.

[0017] In an exemplary embodiment, estimating, by an electronic device, pixel shift and loss in each image frame and an area to be cropped on each image frame based on a tilt angle deviation and a tilt angle may include: identifying, by an electronic device, a maximum / largest tilt angle of an image frame sequence based on the tilt angle of each image frame; determining, by the electronic device, the average tilt angle deviation of the image frame sequence based on the tilt angle deviation of each image frame; determining, by the electronic device, whether the tilt angle of an image frame in the image frame sequence is equal to zero and less than or equal to one of the maximum / largest tilt angle; performing, by the electronic device, one of the following: estimating, by the electronic device, that an angle for correcting the tilt of an image frame is zero based on that the tilt angle of the image frame is zero, and estimating, by the electronic device, that an angle for correcting the tilt of an image frame is a tilt angle based on that the tilt angle of the image frame is less than or equal to the maximum / largest tilt angle; and determining, by the electronic device, an area to be cropped on each image frame based on the average tilt angle deviation, wherein a target tilt correction has a maximum / largest pixel shift and loss.

[0018] In an example embodiment, determining, by an electronic device, a required target tilt correction for each image frame based on pixel offset and loss may include: determining, by the electronic device, based on the pixel offset and loss, whether a maximum / large tilt angle and an average tilt angle deviation of an image frame sequence satisfy a threshold condition; and performing, by the electronic device, one of the following: performing a target tilt correction with maximum / large pixel offset and loss based on the maximum / large tilt angle and the average tilt angle deviation of the image frame sequence satisfying the threshold condition, and performing a target tilt correction with minimum or reduced pixel offset and loss based on the maximum / large tilt angle and the average tilt angle deviation of the image frame sequence not satisfying the threshold condition.

[0019] In an example embodiment, applying, by the electronic device, target tilt correction and cropping to each image frame on a determined area of ​​each image frame may include: rotating, by the electronic device, each image frame based on an estimated angle to correct the tilt of the image frame, and cropping, by the electronic device, the determined area on each image frame based on an average tilt angle deviation.

[0020] In an example embodiment, applying, by the electronic device, at least one of cropping and external padding to the determined area of ​​each image frame may include: determining, by the electronic device, whether the area to be cropped on each image frame satisfies a threshold condition; and performing, by the electronic device, at least one of: cropping the image frame sequence based on the area to be cropped on each image not satisfying the threshold condition, and cropping and external padding the image frame sequence based on the area to be cropped on each image satisfying the threshold condition.

[0021] Therefore, certain example embodiments may provide an electronic device for tilt correction of a video. The electronic device may include a tilt correction controller, a memory, and a processor, wherein the tilt correction controller may be directly or indirectly coupled to the memory and the processor. The tilt correction controller may be configured to receive a user selection of a video. The tilt correction controller may be configured to identify a sequence of image frames of a video representing a scene in the video. The tilt correction controller may be configured to determine the tilt in each image frame of the image frame sequence. The tilt correction controller may be configured to determine whether the cause of the tilt in the image frame sequence is unintentional by analyzing the tilt in each image frame using a trained AI model. The tilt correction controller may be configured to perform tilt correction on the image frame sequence based on the cause of the tilt being unintentional.

[0022] These and other aspects of the embodiments herein will be better appreciated and understood when considered in conjunction with the following description and accompanying drawings. However, it should be understood that the following description, although indicating preferred embodiments and many of their specific details, is given by way of illustration and not limitation. Many changes and modifications may be made within the scope of the embodiments, and the embodiments herein include all such modifications. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Exemplary methods and / or electronic devices are shown in the accompanying drawings, and the same reference numerals indicate corresponding parts in the various drawings throughout the drawings. The exemplary embodiments of the present invention will be better understood from the following description with reference to the accompanying drawings, in which:

[0024] Figure 1A and 1B shows the tilt in the image according to the prior art;

[0025] Figure 2A is a block diagram of an electronic device for tilt correction of a video according to an example embodiment;

[0026] Figure 2B is a block diagram of a tilt correction controller for tilt correction of a video containing image frames having unintentional tilt according to an example embodiment;

[0027] Figure 3 is a flow chart illustrating a method for tilt correction of a video according to an example embodiment;

[0028] Figure 4 A method of determining a tilt angle in an example scenario according to an example embodiment is shown;

[0029] Figure 5 shows a schematic diagram of a self-attention model according to an example embodiment;

[0030] Figure 6 illustrates a tilt angle of an image frame of a 360 degree scale video according to an example embodiment;

[0031] Figure 7 shows scene spans and image frames in each scene span of a video according to an example embodiment;

[0032] Figure 8 is a flow chart illustrating a method for tilt correction with minimal or reduced pixel shift and loss at an image frame according to an example embodiment;

[0033] Fig. 9 is a flow chart illustrating a method for large (eg, maximum) tilt correction at an image frame according to an example embodiment;

[0034] Fig.10 is a flow chart illustrating a method for selecting a tilt correction having a minimal or reduced pixel shift and loss or a large (eg, maximum) pixel shift and loss according to an example embodiment;

[0035] Fig.11 is a flow chart illustrating a method for selecting cropping and / or outer padding of an image frame for skew correction according to an example embodiment;

[0036] Fig.12 is a flow chart illustrating a method for tilt correction of an image frame according to an example embodiment;

[0037] Fig.13 An image frame having a tilt and an image frame corrected for the tilt are shown, wherein the tilt is corrected by minimizing or reducing pixel shift and loss from the image frame according to an example embodiment;

[0038] Fig.14 An image frame having a tilt and a tilt-corrected image frame is shown, wherein the tilt is corrected by maximizing the tilt correction of the image frame according to an example embodiment; and

[0039] Fig.15 An example scenario of correcting tilt in an image according to an example embodiment is shown.

[0040] Fig.16 is a flowchart illustrating a method for controlling an electronic device according to example embodiments. DETAILED DESCRIPTION

[0041] The example embodiments of this article and their various features and advantageous details are explained more fully with reference to the non-limiting embodiments shown in the accompanying drawings and described in detail in the following description. The description of well-known components and processing techniques is omitted so as not to unnecessarily obscure the embodiments of this article. In addition, the various embodiments described herein are not necessarily mutually exclusive, because some embodiments can be combined with one or more other embodiments to form new embodiments. Unless otherwise stated, the term "or" as used herein refers to a non-exclusive or. The examples used herein are intended only to facilitate understanding of the manner in which the embodiments of this article can be practiced, and further enable those skilled in the art to practice the embodiments of this article. Therefore, the examples should not be interpreted as limiting the scope of the embodiments of this article.

[0042] As is conventional in the art, embodiments may be described and illustrated in terms of blocks that perform one or more functions described. These blocks (which may be referred to herein as managers, units, modules, hardware components, etc.) are physically implemented by analog and / or digital circuits (such as logic gates, integrated circuits, microprocessors, microcontrollers, memory circuits, passive electronic components, active electronic components, optical components, hard-wired circuits, etc.) and may optionally be driven by firmware. The circuits may be embodied, for example, in one or more semiconductor chips, or on a substrate support such as a printed circuit board. The circuits that make up the blocks may be implemented by dedicated hardware, or by a processor (e.g., one or more programmed microprocessors and associated circuits), or by a combination of dedicated hardware that performs some functions of the blocks and a processor that performs other functions of the blocks. Without departing from the scope of the present disclosure, each block of the embodiment may be physically divided into two or more interactive and discrete blocks. Similarly, without departing from the scope of the present disclosure, the blocks of the embodiment may be physically combined into more complex blocks.

[0043] The accompanying drawings are used to help easily understand various technical features, and it should be understood that the embodiments presented herein are not limited by the accompanying drawings. Therefore, the present disclosure should be interpreted as extending to any changes, equivalents and substitutes other than those specifically set forth in the accompanying drawings. Although the terms first, second, etc. can be used herein to describe various elements, these elements should not be limited by these terms. These terms are usually only used to distinguish one element from another element.

[0044] Throughout this disclosure, the terms "image frame" and "image" are used interchangeably and mean the same thing.

[0045] Therefore, embodiments of the present invention provide a method for tilt correction of a video by an electronic device. The method includes receiving a user selection of a video by an electronic device. The method includes identifying, by an electronic device, a sequence of image frames of a video representing a scene in the video. The method includes determining, by an electronic device, a tilt in each image frame in the sequence of image frames. The method includes determining, by the electronic device, whether a cause of the tilt in the sequence of image frames is unintentional by analyzing the tilt in each image frame using a trained artificial intelligence (AI) model. The method includes, in response to determining that the cause of the tilt is unintentional, performing tilt correction on the sequence of image frames by the electronic device.

[0046] Therefore, an embodiment of the present invention provides an electronic device for tilt correction of a video. The electronic device includes a tilt correction controller, a memory, and a processor, wherein the tilt correction controller is directly or indirectly coupled to the memory and the processor. The tilt correction controller is configured to receive a user selection of a video. The tilt correction controller is configured to identify a sequence of image frames of a video representing a scene in the video. The tilt correction controller is configured to determine the tilt in each image frame of the image frame sequence. The tilt correction controller is configured to determine whether the cause of the tilt in the image frame sequence is unintentional by analyzing the tilt in each image frame using a trained AI model. The tilt correction controller is configured to perform tilt correction on the image frame sequence in response to determining that the cause of the tilt is unintentional.

[0047] Unlike existing methods and systems, the electronic device brings the tilt angle as close to 0° as possible for each scene span without generating zoom-in / zoom-out jitter in the video during tilt correction, which improves video aesthetics.

[0048] Unlike existing methods and systems, the electronic device minimizes or reduces pixel and data loss, maintains uniform resolution of image frames, and maintains smooth transitions between image frames after tilt correction by applying outer padding on black patches formed during tilt correction and center cropping the image frames.

[0049] Referring now to the drawings, and more particularly to Figures 2A to 15 , showing a preferred embodiment.

[0050] Figure 2A is a block diagram of an electronic device (100) for tilt correction of video according to an embodiment as disclosed herein. Examples of electronic devices (100) include, but are not limited to, smart phones, tablet computers, personal digital assistants (PDAs), desktop computers, Internet of Things (IoT), wearable devices, etc. In an embodiment, the electronic device (100) includes a tilt correction controller (110), a memory (120), a processor (130), a communicator (140), and a display (150), wherein the display is a physical hardware component that can be used to display a video to a user. Examples of a main display include, but are not limited to, a light emitting diode display, a liquid crystal display, a projector, etc. The tilt correction controller (110) is implemented by processing circuits such as logic gates, integrated circuits, microprocessors, microcontrollers, memory circuits, passive electronic components, active electronic components, optical components, hard-wired circuits, etc., and can be optionally driven by firmware. The circuit can be embodied, for example, in one or more semiconductor chips, or on a substrate support such as a printed circuit board.

[0051] The tilt correction controller (110) receives a user's selection of a video. In an embodiment, the display (150) displays the video. In addition, the user can select an area of ​​the displayed video. In addition, the tilt correction controller (110) determines the selected area on the displayed video as the user's selection of the video.

[0052] In addition, the tilt correction controller (110) identifies a sequence of image frames of the video representing a scene in the video. In one embodiment, the tilt correction controller (110) extracts image frames of the video and identifies image frames having similar scenes in sequence. In addition, the tilt correction controller (110) groups the image frames in a sequence of scenes similar to a scene span. The tilt correction controller (110) determines the tilt in each image frame of the scene span or the sequence of image frames.

[0053] The tilt correction controller (110) performs the following steps by using a trained artificial intelligence (AI) model (e.g., Figure 2B The tilt correction controller (110) analyzes the tilt in each image frame to determine whether the cause of the tilt in the scene span or the image frame sequence is unintentional. In an embodiment, the tilt correction controller (110) uses the trained AI model (119) to determine whether the tilt in each image frame satisfies a first threshold condition. In addition, the tilt correction controller (110) determines that the cause of the tilt in the image frame sequence is unintentional in response to determining that the tilt in each image frame satisfies the threshold condition.

[0054] The tilt correction controller (110) performs tilt correction on the sequence of image frames in response to determining that the cause of the tilt is unintentional. In an embodiment, the tilt correction controller (110) estimates the tilt angle (also denoted as α) of each image frame for a large (e.g., maximum) tilt correction at each image frame. In addition, the tilt correction controller (110) determines a tilt angle deviation (also denoted as δ) of each image frame, wherein the tilt angle deviation of an image frame of the sequence of image frames is the absolute value of the difference between the tilt angle of the image frame and the average value of the tilt angles of the image frame and adjacent image frames of the sequence of image frames. In addition, the tilt correction controller (110) estimates the pixel shift and loss in each image frame and the area to be cropped on each image frame based on the tilt angle deviation and the tilt angle. In addition, the tilt correction controller (110) determines the target tilt correction required for each image frame based on the pixel shift and loss. In addition, the tilt correction controller (110) applies the target tilt correction and cropping and / or external padding to the determined area of ​​each image frame. In addition, the tilt correction controller (110) stitches each tilt-corrected image frame to reconstruct the scene. Furthermore, the tilt correction controller (110) generates a video by sequentially joining the spliced ​​image frames.

[0055] In an embodiment, in order to estimate the tilt angle of each image frame for maximum tilt correction at each image frame, the tilt correction controller (110) uses a trained self-attention model to predict the range of possible tilt angles for each image frame. In addition, the tilt correction controller (110) determines the tilt angle by determining a weighted average of predicted tilt angles with a standard deviation of zero or greater values. In an embodiment, the tilt correction controller (110) determines the range of possible tilt angles of the training image frames based on the relative position of the object in the training image frame, the correlation of the object, and the relationship between the objects for training the self-attention model. In addition, the tilt correction controller (110) generates training data using the range of possible tilt angles. In addition, the tilt correction controller (110) generates an output vector with distribution probability by performing multi-label training on the training data. In addition, the tilt correction controller (110) trains the self-attention model using the output vector using an S-shaped activation for tilt angle prediction.

[0056] In an embodiment, in order to estimate pixel shift and loss in each image frame and an area to be cropped on each image frame based on the tilt angle deviation and the tilt angle, the tilt correction controller (110) identifies a small (e.g., minimum) tilt angle of the image frame sequence based on the tilt angle of each image frame. In addition, the tilt correction controller (110) determines an average tilt angle deviation of the image frame sequence based on the tilt angle deviation of each image frame. In addition, the tilt correction controller (110) determines whether the small (e.g., minimum) tilt angle is greater than the average tilt angle deviation. In response to determining that the small (e.g., minimum) tilt angle is greater than the average tilt angle deviation, the tilt correction controller (110) sets the average tilt angle deviation to the small (e.g., minimum) tilt angle and estimates an angle for correcting the tilt of each image frame. In response to determining that the small (e.g., minimum) tilt angle is not greater than the average tilt angle deviation, the tilt correction controller (110) estimates an angle for correcting the tilt of each image frame. In addition, the tilt correction controller (110) determines an area to be cropped on each image frame based on the average tilt angle deviation, wherein the target tilt correction has a minimum or reduced pixel shift and loss.

[0057] In an embodiment, in order to estimate the angle of the tilt correction of each image frame, the tilt correction controller (110) determines whether the tilt angle of the image frame in the image frame sequence is equal to zero, or is less than the average tilt angle deviation, or is equal to the average tilt angle deviation, or is greater than the average tilt angle deviation. The tilt correction controller (110) estimates the angle of the tilt correction of the image frame to be zero in response to determining that the tilt angle of the image frame is equal to zero. The tilt correction controller (110) estimates the angle of the tilt correction of the image frame to be the tilt angle in response to determining that the tilt angle of the image frame is less than or equal to the average tilt angle deviation. The tilt correction controller (110) estimates the angle of the tilt correction of the image frame to be the average tilt angle deviation in response to determining that the tilt angle of the image frame is greater than the average tilt angle deviation.

[0058] In another embodiment, in order to estimate the pixel shift and loss in each image frame and the area to be cropped on each image frame based on the tilt angle deviation and the tilt angle, the tilt correction controller (110) identifies the maximum tilt angle of the image frame sequence based on the tilt angle of each image frame. In addition, the tilt correction controller (110) determines the average tilt angle deviation of the image frame sequence based on the tilt angle deviation of each image frame. In addition, the tilt correction controller (110) determines whether the tilt angle of the image frame of the image frame sequence is equal to zero, or is less than the maximum tilt angle or equal to the maximum tilt angle. The tilt correction controller (110) estimates the angle of the tilt of the corrected image frame to be zero in response to determining that the tilt angle of the image frame is equal to zero. The tilt correction controller (110) estimates the angle of the tilt of the corrected image frame to be the tilt angle in response to determining that the tilt angle of the image frame is less than the maximum tilt angle or equal to the maximum tilt angle. In addition, the tilt correction controller (110) determines the area to be cropped on each image frame based on the average tilt angle deviation, where the target tilt correction has the maximum pixel shift and loss.

[0059] In an embodiment, in order to determine the target tilt correction required for each image frame based on the pixel offset and loss, the tilt correction controller (110) determines whether the maximum tilt angle and the average tilt angle deviation of the image frame sequence meet a second threshold condition based on the pixel offset and loss. In response to determining that the maximum tilt angle and the average tilt angle deviation of the image frame sequence meet the second threshold condition, the tilt correction controller (110) performs the target tilt correction with the maximum pixel offset and loss. In response to determining that the maximum tilt angle and the average tilt angle deviation of the image frame sequence do not meet the second threshold condition, the tilt correction controller (110) performs the target tilt correction with a minimum or reduced pixel offset and loss.

[0060] In an embodiment, in order to apply target tilt correction and cropping on the determined area of ​​each image frame, a tilt correction controller (110) including a processing circuit rotates each image frame based on the estimated angle to correct the tilt of the image frame. In addition, the tilt correction controller (110) crops the determined area on each image frame based on the average tilt angle deviation.

[0061] In an embodiment, in order to apply cropping and / or external padding on the determined area of ​​each image frame, the tilt correction controller (110) determines whether the area to be cropped on each image frame satisfies a third threshold condition. In response to determining that the area to be cropped on each image does not satisfy the threshold condition, the tilt correction controller (110) crops the sequence of image frames. In addition, in response to determining that the area to be cropped on each image satisfies the threshold condition, the tilt correction controller (110) crops and externally pads the sequence of image frames.

[0062] The memory (120) stores video, and image frames are processed in the intermediate stage of tilt correction of the video. The memory (120) stores instructions to be executed by the processor (130). The memory (120) may include a non-volatile storage element. Examples of such non-volatile storage elements may include a magnetic hard disk, an optical disk, a floppy disk, a flash memory, or a form of an electrically programmable memory (EPROM) or an electrically erasable programmable (EEPROM) memory. In addition, in some examples, the memory (120) may be considered a non-transitory storage medium. The term "non-transitory" may indicate that the storage medium is not embodied in a carrier or propagating signal. However, the term "non-transitory" should not be interpreted as the memory (120) being non-removable. In some examples, the memory (120) may be configured to store a larger amount of information than its storage space. In some instances, a non-transitory storage medium may store data that may change over time (e.g., in a random access memory (RAM) or a cache memory). The memory ( 120 ) may be an internal storage unit, or it may be an external storage unit of the electronic device ( 100 ), cloud storage, or any other type of external storage.

[0063] The processor (130) is configured to execute instructions stored in the memory (120). The processor (130) may be a general-purpose processor (such as a central processing unit (CPU), an application processor (AP), etc.), a graphics processing unit (such as a graphics processing unit (GPU), a visual processing unit (VPU), etc.). The processor (130) may include multiple cores for executing instructions. The communicator (140) is configured to perform internal communication between hardware components in the electronic device (100). In addition, the communicator (140) is configured to facilitate communication between the electronic device (100) and other devices via one or more networks (e.g., radio technologies). The communicator (140) includes electronic circuits specific to standards for implementing wired or wireless communication.

[0064] although Figure 2A The hardware components of the electronic device (100) are shown, but it should be understood that other embodiments are not limited thereto. In other embodiments, the electronic device (100) may include fewer or greater numbers of components. In addition, the labels or names of the components are for illustrative purposes only and do not limit the scope. One or more components may be combined to perform the same or substantially similar functions for tilt correction of video.

[0065] Figure 2B The invention is a block diagram of a tilt correction controller (110) for tilt correction of a video containing an image frame with unintentional tilt according to an embodiment disclosed herein. In one embodiment, the tilt correction controller (110) includes a frame extractor (111), a scene change detector (112), a tilt angle error detector (113), a scene frame tilt recorder (114), a frame and tilt checker (115), a smooth scene frame tilt recorder (116), a frame tilt corrector (117), a video reconstruction controller (118) and an AI model (119). The frame extractor (111), the scene change detector (112), the tilt angle error detector (113), the scene frame tilt recorder (114), the frame and tilt checker (115), the smooth scene frame tilt recorder (116), the frame tilt corrector (117) and the video reconstruction controller (118) are implemented by processing circuits, such as logic gates, integrated circuits, microprocessors, microcontrollers, memory circuits, passive electronic components, active electronic components, optical components, hard-wired circuits, etc., and can be optionally driven by firmware. The circuit may, for example, be embodied in one or more semiconductor chips, or on a substrate support such as a printed circuit board or the like.

[0066] At least one of the multiple modules of the tilt correction controller (110) can be implemented by an AI model (119). Functions associated with the AI ​​model (119) can be executed by a non-volatile / volatile memory (120) and a processor (130). One or more processors (130) control the processing of input data according to a predefined operating rule or AI model (119) stored in the non-volatile / volatile memory (120). The predefined operating rule or AI model (119) is provided by training or learning. Here, it is provided by a learning device, and the predefined operating rule or AI model (119) of the desired characteristics is formed by applying a learning method to a plurality of learning data. Learning can be performed in the electronic device 100 itself that executes the AI ​​model 119 according to the embodiment, and / or can be implemented by a separate server / system. Each "model" and each "module" herein may include a circuit, such as a processing circuit.

[0067] The AI ​​model (119) may be or include a plurality of neural network layers. Each layer has a plurality of weight values, and the layer operation is performed by the calculation of the previous layer and the operation of the plurality of weights. Examples of neural networks include, but are not limited to, convolutional neural networks (CNNs), deep neural networks (DNNs), recurrent neural networks (RNNs), restricted Boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), generative adversarial networks (GANs), and deep Q networks. A learning method is a method of using a plurality of learning data to train a predetermined target device (e.g., a robot) to enable, allow, or control the target device to make a determination or prediction. Examples of learning methods include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

[0068] A frame extractor (111) receives a video and a user's selection of a video. The frame extractor (111) extracts image frames of the video. A scene change detector (112) receives the extracted image frames with time stamps and predicts tilt angles in the image frames. The scene change detector (112) detects video scenes using temporal and spatial features and identifies image frames with similar scenes in sequence. Temporal features mean features based on time. For example, a scene is defined by a specific time span in a video. Spatial features imply content in an image frame of a video. For example, a scene depicting a character performing a certain action. The scene change detector (112) provides a range of frame identifiers (IDs) belonging to similar scenes and groups the frame IDs under a scene span.

[0069] The tilt error detector (113) receives a red-green-blue (RGB) frame buffer having dimensions (e.g., length and / or width). The tilt error detector (113) uses trained weights to perform deep learning model operations to detect unintentional tilt errors in the range of 0 to 360 degrees. The tilt error detector (113) generates 360 class label probabilities for the image frame. If the value of the class label is detected to be higher than a threshold, the tilt error detector (113) determines a class label of a predicted tilt angle of the image frame.

[0070] The tilt angle error detector (113) determines the tilt in each image frame of the scene span or the image frame sequence. The tilt angle error detector (113) determines whether the cause of the tilt in the scene span or the image frame sequence is unintentional by analyzing the tilt in each image frame using the trained AI model (119). In an embodiment, the tilt angle error detector (113) determines whether the tilt in each image frame satisfies a first threshold condition (e.g., a value of a class label is detected to be higher than a threshold) using the trained AI model (119). In addition, the tilt angle error detector (113) determines that the cause of the tilt in the image frame sequence is unintentional in response to determining that the tilt in each image frame satisfies the threshold condition.

[0071] The tilt angle error detector (113) estimates the tilt angle of each image frame for (e.g., maximum) tilt correction at each image frame. In addition, the tilt angle error detector (113) determines the tilt angle deviation of each image frame. In an embodiment, in order to estimate the tilt angle of each image frame for maximum / large tilt correction at each image frame, the tilt angle error detector (113) uses the trained self-attention model to predict the range of possible tilt angles of each image frame. In addition, the tilt angle error detector (113) determines the tilt angle by determining a weighted average of predicted tilt angles with a standard deviation of zero or greater. In an embodiment, the tilt angle error detector (113) determines the range of possible tilt angles of the training image frames based on the relative positions of objects in the training image frames, the correlation of objects, and the relationship between objects for training the self-attention model. In addition, the tilt angle error detector (113) generates training data using the range of possible tilt angles. In addition, the tilt correction controller (110) generates an output vector with distribution probability by performing multi-label training on the training data. Furthermore, the tilt angle error detector (113) uses a sigmoid activation for tilt angle prediction and utilizes the output vector to train a self-attention model.

[0072] The frame and tilt checker (115) receives the tilt angle of the image frame in the scene span. and tilt angle deviation The frame and tilt checker (115) determines an appropriate tilt correction method given the input scene information and calculates a target tilt angle for each frame under observation. In addition, the frame and tilt checker 115 provides a target tilt angle for each frame under observation.

[0073] The frame and tilt checker (115) estimates pixel shift and loss in each image frame and an area to be cropped on each image frame based on the tilt angle deviation and the tilt angle. In addition, the frame and tilt checker (115) determines a target tilt correction required for each image frame based on the pixel shift and loss. In an embodiment, the frame and tilt checker (115) identifies a minimum tilt angle of the image frame sequence based on the tilt angle of each image frame. In addition, the frame and tilt checker (115) determines an average tilt angle deviation of the image frame sequence based on the tilt angle deviation of each image frame. In addition, the frame and tilt checker (115) determines whether the minimum tilt angle is greater than the average tilt angle deviation. The frame and tilt checker (115) sets the average tilt angle deviation as the minimum tilt angle, and in response to determining that the minimum tilt angle is greater than the average tilt angle deviation, estimates an angle to correct the tilt of each image frame. In response to determining that the minimum tilt angle is not greater than the average tilt angle deviation, the frame and tilt checker (115) estimates an angle to correct the tilt of each image frame. Furthermore, a frame and tilt checker (115) determines a region to be cropped on each image frame based on the average tilt angle deviation, where the target tilt correction has the minimum pixel shift and loss.

[0074] In an embodiment, the frame and tilt checker (115) determines whether the tilt angle of an image frame in the sequence of image frames is equal to zero, or is less than an average tilt angle deviation, or is equal to the average tilt angle deviation, or is greater than the average tilt angle deviation. In response to determining that the tilt angle of the image frame is equal to zero, the frame and tilt checker (115) estimates the angle of the tilt of the corrected image frame to be zero. In response to determining that the tilt angle of the image frame is less than or equal to the average tilt angle deviation, the frame and tilt checker (115) estimates the angle of the tilt of the corrected image frame to be the tilt angle. In response to determining that the tilt angle of the image frame is greater than the average tilt angle deviation, the frame and tilt checker (115) estimates the angle of the tilt of the corrected image frame to be the average tilt angle deviation.

[0075] In another embodiment, the frame and tilt checker (115) identifies a maximum tilt angle of the image frame sequence based on the tilt angle of each image frame. In addition, the frame and tilt checker (115) determines an average tilt angle deviation of the image frame sequence based on the tilt angle deviation of each image frame. In addition, the frame and tilt checker (115) determines whether the tilt angle of the image frame of the image frame sequence is equal to zero, or is less than the maximum tilt angle or equal to the maximum tilt angle. In response to determining that the tilt angle of the image frame is equal to zero, the frame and tilt checker (115) estimates that the angle of the tilt of the corrected image frame is zero. In response to determining that the tilt angle of the image frame is less than the maximum tilt angle or is equal to the maximum tilt angle, the frame and tilt checker (115) estimates that the angle of the tilt of the corrected image frame is the tilt angle. In addition, the frame and tilt checker (115) determines an area to be cropped on each image frame based on the average tilt angle deviation, where the target tilt correction is the largest and where the pixel shift and loss are also the largest.

[0076] In an embodiment, the frame and tilt checker (115) determines whether a maximum tilt angle and an average tilt angle deviation of the image frame sequence meet a second threshold condition based on the pixel offset and the loss. In response to determining that the maximum tilt angle and the average tilt angle deviation of the image frame sequence meet the second threshold condition, the frame and tilt checker (115) performs a target tilt correction with a maximum pixel offset and the loss. In response to determining that the maximum tilt angle and the average tilt angle deviation of the image frame sequence do not meet the second threshold condition, the frame and tilt checker (115) performs a target tilt correction with a minimum pixel offset and the loss.

[0077] In response to determining that the cause of the tilt is unintentional, the frame tilt corrector (117) performs tilt correction on the sequence of image frames. The frame tilt corrector (117) uses the extracted RGB frame buffer and applies the target tilt correction, and performs cropping and / or external padding on the determined area of ​​each image frame to generate a tilt-corrected image frame. In an embodiment, the frame tilt corrector (117) rotates each image frame based on the estimated angle to correct the tilt of the image frame. In addition, the frame tilt corrector (117) crops the determined area on each image frame based on the average tilt angle deviation. In another embodiment, the frame tilt corrector (117) determines whether the area to be cropped on each image frame satisfies a third threshold condition. In response to determining that the area to be cropped on each image does not satisfy the threshold condition, the frame tilt corrector (117) crops the sequence of image frames. In addition, in response to determining that the area to be cropped on each image satisfies the threshold condition, the frame tilt corrector (117) crops and externally pads the sequence of image frames.

[0078] A video reconstruction controller (118) receives the extracted and corrected RGB frame buffers and splices the corrected and uncorrected image frames to reconstruct the video, wherein the corrected video frames are inserted in place of the skewed frames on the video, wherein the video is regenerated by sequentially connecting the spliced ​​image frames.

[0079] although Figure 2B The hardware components of the tilt correction controller (110) are shown, but it should be understood that other embodiments are not limited thereto. In other embodiments, the tilt correction controller (110) may include fewer or greater numbers of components. In addition, the labels or names of the components are for illustrative purposes only and do not limit the scope. One or more components may be combined to perform the same or substantially similar functions for tilt correction of a video containing image frames with unintentional tilt.

[0080] Figure 3 is a flowchart (300) illustrating a method for tilt correction of a video according to an embodiment disclosed herein. In an embodiment, the method allows a tilt correction controller (110) to perform steps 301-305 of the flowchart (300). At step 301, the method includes receiving a user selection of a video. At step 302, the method includes identifying a sequence of image frames of the video representing a scene in the video. At step 303, the method includes determining a tilt in each image frame of the sequence of image frames. At step 304, the method includes determining whether a cause of the tilt in the sequence of image frames is unintentional by analyzing the tilt in each image frame using a trained AI model (119). At step 305, the method includes performing tilt correction on the sequence of image frames in response to determining that the cause of the tilt is unintentional.

[0081] Figure 4 An example scenario of determining a tilt angle according to an embodiment as disclosed herein is shown. In the proposed method, a tilt angle range of higher than 15° relative to 0° in a 360-degree scale in a clockwise or counterclockwise direction is considered to intentionally produce a tilt in an image / video, while a tilt angle range of higher than 0° to 15° relative to 0° in a 360-degree scale in a clockwise or counterclockwise direction is considered to unintentionally produce a tilt in an image / video. An electronic device (100) (e.g., a deep tilt detection neural engine) extracts features of each image frame when obtaining image frames of a video. In addition, the electronic device (100) uses a self-attention model to predict the tilt in each image frame. In addition, the electronic device 100 determines the tilt angle α using Equation 1 for processing each image frame.

[0082] (1)

[0083] The variable i represents the prediction of the i-th label among the 5 labels, w represents the weight of the predicted value of the i-th label, and X represents the predicted value of the i-th label.

[0084] Consider an example scenario in which the electronic device (100) predicts the tilt angles of image frames (401-404). The electronic device (100) predicts the tilt angle of the image frame (401) as 1° clockwise in the first iteration, which means that the relevant object in the image frame (401) is tilted 1° clockwise relative to 0° in the 360-degree scale. In consecutive iterations, the electronic device (100) predicts the tilt angle of the image frame (402) as 2° clockwise, the tilt angle of the image frame (403) as 4° clockwise, and the tilt angle of the image frame (404) as 4° clockwise. After predicting all the tilt angles, the electronic device (100) determines a weighted average of the predictions as the tilt angle.

[0085] Figure 5 A schematic diagram of a self-attention model according to an embodiment disclosed herein is shown. The layers, outputs, and expansions of the self-attention model are given in Table 1.

[0086] layer Output expansion enter <![CDATA[224 2 x 3]]> - Convolutional Block <![CDATA[112 2 x 16]]> 1 Reverse bottleneck layer_no_expansion <![CDATA[112 2 x 16]]> 1 Reverse bottleneck layer <![CDATA[56 2 x 16]]> 1 Reverse bottleneck layer <![CDATA[[28 2 x 32]x 4 blocks]]> [1,1,1,1] Reverse bottleneck layer <![CDATA[[14 2 x 64]x 4 blocks]]> [1,1,2,4] Self-Attention Model Block <![CDATA[[14 2 x 64]x 3 blocks]]> [1,2,4] Self-Attention Model Block <![CDATA[[7 2 x 96]x 4 blocks]]> [2,2,4,4] Separable Convolutional Layer num_classes -

[0087] Table 1

[0088] Figure 6 The tilt angle of an image frame of a video at a 360-degree scale according to an embodiment as disclosed herein is shown. The tilt angle (α) of the image frame is measured at a scale of 360 degrees. Even if the tilt angle (α2) is in the clockwise direction or the tilt angle (α1) is in the counterclockwise direction, the tilt angle deviation relative to 0° in the 360-degree scale is an absolute value. Specifically, an image frame tilted 10 degrees clockwise and an image frame tilted 10 degrees counterclockwise present a tilt angle deviation of 10 degrees relative to 0° in the 360-degree scale, where 0° is the absolute value of the tilt angle (+10° or -10°). By determining the tilt angle deviation relative to 0°, the tilt angle deviation relative to adjacent frames is implicitly taken into account.

[0089] Figure 7 The scene spans of a video and the image frames in each scene span according to an embodiment as disclosed herein are shown. Consider three scene spans of a video, where scene span 1 includes 5 consecutive image frames, scene span 2 includes the remaining 6 consecutive image frames, and scene span includes another 4 consecutive image frames, where the number of each image frame in the scene span represents the frame identifier of the corresponding image frame. The 6 consecutive image frames of scene span 2 are in Figure 7 The tilt angle and tilt angle deviation for each image frame of scene span 2 are given below, one for each image frame.

[0090] Figure 8is a flow chart (800) showing a method for tilt correction with minimum pixel shift and loss at an image frame according to an embodiment as disclosed herein. In an embodiment, the method allows a tilt correction controller (110) to perform steps 801-812 of the flow chart (800). In step 801, the method includes identifying a minimum tilt angle (α_ ) of a sequence of image frames based on the tilt angle of each image frame. min At step 802, the method includes determining an average tilt angle deviation (δ_) of the image frame sequence based on the tilt angle deviation of each image frame. mean ). At step 803, the method includes determining whether the minimum tilt angle is greater than the average tilt angle deviation.

[0091] When the scene does not end after step 803, at steps 804-806, the method includes, in response to determining that the minimum tilt angle is not greater than the average tilt angle deviation, determining whether the tilt angle of the image frame of the image frame sequence is one of equal to zero, less than or equal to the average tilt angle deviation, and greater than the average tilt angle deviation. At step 807, the method includes setting the average tilt angle deviation to the minimum tilt angle, and in response to determining that the minimum tilt angle is greater than the average tilt angle deviation, proceeding to steps 804-806 when the scene does not end after step 807. At step 808, the method includes estimating that the angle of the tilt of the corrected image frame is zero (e.g., correction angle = 0) in response to determining that the tilt angle of the image frame is equal to zero.

[0092] At step 809, the method includes, in response to determining that the tilt angle of the image frame is less than or equal to the average tilt angle deviation, estimating the angle of the tilt of the corrected image frame as the tilt angle (e.g., correction angle=tilt angle). At step 810, the method includes, in response to determining that the tilt angle of the image frame is greater than the average tilt angle deviation, estimating the angle of the tilt of the corrected image frame as the average tilt angle deviation (e.g., correction angle=δ_ mean ). In step 811, the method includes determining a region to be cropped on each image frame based on the average tilt angle deviation. In step 812, the method includes performing tilt correction in the image frame based on the correction angle and cropping the cropping region.

[0093] Fig. 9 is a flow chart (900) illustrating a method for maximum tilt correction at an image frame according to an embodiment as disclosed herein. In an embodiment, the method allows a tilt correction controller (110) to perform steps 901-908 of the flow chart (900). In step 901, the method includes identifying a maximum tilt angle (α_ ) of a sequence of image frames based on the tilt angle of each image frame. max At step 902, the method includes determining an average tilt angle deviation (δ_) of the image frame sequence based on the tilt angle deviation of each image frame. mean). When the scene does not end after step 902, in steps 903 to 904, the method includes determining whether the tilt angle of the image frame of the sequence of image frames is one of equal to zero and less than or equal to the maximum tilt angle.

[0094] At step 905, the method includes estimating the angle of tilt of the corrected image frame to be zero (e.g., correction angle = 0) in response to determining that the tilt angle of the image frame is equal to zero. At step 906, the method includes: in response to determining that the tilt angle of the image frame is less than or equal to the maximum tilt angle, estimating the angle of tilt of the corrected image frame to be the tilt angle (e.g., correction angle = tilt angle). At step 907, the method includes determining an area to be cropped on each image frame based on the average tilt angle deviation. At step 908, the method includes performing tilt correction in the image frame based on the correction angle and cropping the cropping area.

[0095] Fig.10 is a flow chart (1000) showing a method for selecting a tilt correction with minimum pixel shift and loss or maximum pixel shift and loss according to an embodiment as disclosed herein. In an embodiment, the method allows a tilt correction controller (110) to perform steps 1001-1003 of the flow chart (1000). At step 1001, the method includes determining a maximum tilt angle (α_ max ) and the mean tilt angle deviation (δ_ mean ) satisfies the second threshold condition. The second threshold condition is the minimum tilt angle threshold (α _min_threshold )≤Maximum tilt angle (α_ max )≤maximum tilt angle threshold (α_ max_threshold ) and the minimum tilt angle deviation threshold (δ_ min_threshold )≤mean tilt angle deviation(δ_ mean )≤maximum tilt angle threshold (δ_ max_threshold ). As an example, the minimum tilt angle threshold may be set to 0.90, and the maximum tilt angle threshold may be set to 0.95. As an example, the maximum tilt angle threshold may be set to 0.95. As an example, the minimum tilt angle deviation threshold may be set to 0. As an example, the maximum tilt angle deviation threshold may be set to 10.

[0096] At step 1002, the method includes, in response to determining that the maximum tilt angle and the average tilt angle deviation of the image frame sequence do not meet the second threshold condition, performing a target tilt correction with a minimum pixel shift and loss (e.g., a minimum tilt correction). At step 1003, the method includes, in response to determining that the maximum tilt angle and the average tilt angle deviation of the image frame sequence meet the second threshold condition, performing a target tilt correction with a maximum tilt correction (e.g., a maximum pixel shift and loss).

[0097] Fig.11 is a flowchart (1100) illustrating a method for selecting cropping and / or external padding of an image frame for tilt correction according to an embodiment as disclosed herein. In an embodiment, the method allows a tilt correction controller (110) to perform steps 1101-1104 of the flowchart (1100). In step 1101, the method includes determining a cropped area on an image frame as a percentage of a total area of ​​the image frame. In step 1102, the method includes determining whether the cropped area on each image frame satisfies a third threshold condition. The third threshold condition is a minimum cropped area threshold ≤ cropped area ≤ maximum cropped area threshold. The cropped area threshold can be calculated as a combination of a cropped area percentage of the complete image and an importance of the cropped area.

[0098] The cropped area percentage can be calculated as a function of the tilt angle. The importance of the cropped area can be calculated based on a lookup table that can assign scores based on the presence of people, documents, and whether the cropped area is part of a salient area of ​​the complete image. Assuming the lookup table has a minimum score of 0 and a maximum score of 100, the minimum cropped area threshold can be set to 5 and the maximum cropped area threshold can be set to 35. In step 1103, the method includes cropping the image frame sequence in response to determining that the area to be cropped on each image does not meet the third threshold condition. In step 1104, the method includes cropping and externally padding the image frame sequence in response to determining that the area to be cropped on each image meets the third threshold condition.

[0099] Fig.12 is a flow chart (1200) illustrating a method for tilt correction of an image frame according to an embodiment as disclosed herein. In an embodiment, the method allows a tilt correction controller (110) to perform steps 1201-1206 of the flow chart (1200). At step 1201, the method includes receiving an image frame having a tilt. At step 1202, the method includes detecting the tilt in the image frame. At step 1203, the method includes determining a required tilt correction in the image frame and applying the tilt correction to the image frame. At step 1204, the method includes detecting black patches (or black fill) formed on a peripheral area of ​​the image frame due to the application of the tilt correction. At step 1205, the method includes determining a required external fill on the black patches and applying the external fill on the black patches. At step 1206, the method includes providing an image frame without tilt and maintaining a resolution of the image frame based on the received image frame.

[0100] The various actions, behaviors, blocks, steps, etc. in the flowcharts (300, 800, 900, 1000, 1100, 1200) may be performed in the order presented, in a different order, or simultaneously. Furthermore, in some embodiments, some of the actions, behaviors, blocks, steps, etc. may be omitted, added, modified, skipped, etc. without departing from the scope.

[0101] Fig.13 An image frame with tilt and image frame with tilt correction according to an embodiment as disclosed herein is shown, wherein the tilt is corrected by minimizing pixel shift and loss from the image frame. 1301A-1304A are tilted image frames with an example scene span. Consider that the electronic device (100) selects tilt correction by minimizing pixel shift and loss from the image frame (1301A-1304A). Then, the electronic device (100) reduces the tilt in the image frame (1301A-1304A) by minimizing pixel shift and loss, and provides image frames (1301B-1304B) for the scene span. The electronic device (100) ensures minimal data loss due to cropping outside the boundary area of ​​the image frame (1301A-1304A). However, the image frame (1301B-1304B) still has some tilt relative to the image frame with minimal tilt distortion.

[0102] Fig.14 An image frame with tilt and tilt correction according to an embodiment disclosed herein is shown, wherein the tilt is corrected by maximizing the tilt correction of the image frame. 1401A-1403A are tilted image frames with example scene spans. Consider that the electronic device (100) selects the tilt correction by maximizing the tilt correction of the image frame (1401A-1403A). The electronic device (100) then reduces the tilt in the image frame (1401A-1403A) by maximizing the tilt correction without worrying about minimizing pixel shift and loss, which results in higher pixel shift and data loss outside the boundary area when calculating the cropping area based on the highest angle correction applied.

[0103] Fig.15 An example scene of correcting tilt in an image according to an embodiment as disclosed herein is shown. 1501 is an image frame containing tilt, and 1502 is an image frame generated by the electronic device (100) after correcting the tilt in the image (1501). 1501A represents the horizon behind the person in the image (1501), which appears tilted to the left. 1502A represents the horizon behind the person in the image (1502), where the horizon is orthogonal to the gravity vector, and after the tilt correction, no tilt appears in the image (1502), which improves the aesthetics of the image frame and brings a professional film effect to the recorded video.

[0104] Fig.16 is a flowchart illustrating a method for controlling an electronic device according to an embodiment.

[0105] A method for controlling an electronic device includes: receiving a user selection (or user input) for selecting a video (or content) (S1605), obtaining a plurality of image frames of a video representing a scene in the video (S1610), obtaining an object in each of the plurality of image frames (S1615), obtaining a tilt angle of the object (S1620) and performing tilt correction (or a tilt correction function) on the plurality of image frames by analyzing the tilt angle of the object using a trained artificial intelligence (AI) model based on the tilt angle corresponding to a first threshold condition (S1625).

[0106] According to one or more embodiments, even if there are multiple objects in each image frame, the tilt angle of each image frame itself can be determined by one object. This is because the tilt angle of each image frame is usually caused by the rotation of the camera (or image sensor). There is a high probability that the tilt angle of each of the multiple objects is the same.

[0107] The object may be a target object capable of representing a horizontal line. Therefore, the object may be described as a target object.

[0108] The tilt angle of the object may mean a tilt angle between a horizontal line of the object and a horizontal line of each image frame.

[0109] The horizontal line of the object may be different according to the type of the object. In addition, the method of obtaining the tilt angle of the object may vary according to the type of the object.

[0110] For example, the tilt of each image frame may be obtained based on an angle difference between a measured horizontal line of the building object and a reference horizontal line of each image frame.

[0111] For example, when there are multiple objects in each image frame, the tilt angle of each image frame may be determined based on the tilt angles of the multiple objects.

[0112] Each image frame may include a plurality of objects. Based on each image frame including a plurality of objects, the method includes obtaining a representative tilt angle of each image frame. The method includes obtaining a representative tilt angle of each image frame by using an average tilt angle.

[0113] For example, based on a first image frame including a first object and a second object, the method includes obtaining a representative tilt angle of the first image frame. The method includes obtaining a first tilt angle of the first object and a second tilt angle of the second object. The method includes obtaining an average tilt angle based on the first tilt angle and the second tilt angle. The method includes obtaining a representative tilt angle of the first image frame based on the average tilt angle.

[0114] The method also includes using the trained AI model to determine whether the tilt angle corresponds to a first threshold condition, and based on the tilt angle being greater than the first threshold angle, determining that the tilt angle corresponds to the first threshold condition.

[0115] Each embodiment herein may be used in combination with any other embodiment described herein.

[0116] Performing tilt correction on a plurality of image frames includes obtaining a tilt angle of each image frame based on an object in each image frame. The tilt angle of each image frame may be written as a representative tilt angle of each image frame.

[0117] Performing tilt correction on multiple image frames includes obtaining a tilt angle deviation of each image frame, wherein the tilt angle deviation of an image frame in the multiple image frames is an absolute value of a difference between a tilt angle of the image frame and an average value of tilt angles of the image frame and adjacent image frames in the multiple image frames.

[0118] Performing tilt correction on the plurality of image frames includes obtaining pixel shift and loss in each image frame and an area to be cropped on each image frame based on the tilt angle deviation and the tilt angle.

[0119] Performing tilt correction on the plurality of image frames includes obtaining a target tilt correction required for each image frame based on pixel offsets and losses.

[0120] Performing the deflection correction on the plurality of image frames includes applying a target deflection correction to the obtained region of each image frame, and at least one of cropping and external padding.

[0121] Performing deskew on the plurality of image frames includes stitching each of the deskewed image frames to reconstruct the scene.

[0122] Performing the tilt correction on the plurality of image frames includes generating a video by sequentially splicing the stitched image frames.

[0123] Obtaining the tilt angle of each image frame includes obtaining a range of possible tilt angles for each image frame using a trained self-attention model, and obtaining the tilt angle by obtaining a weighted average of predicted tilt angles having a standard deviation of zero or greater values.

[0124] A self-attention model is trained for tilt angle prediction in the following manner: a range of possible tilt angles of a training image frame is obtained based on the relative positions of objects in the training image frame, the correlation of objects, and the relationship between objects, training data is generated using the range of possible tilt angles, an output vector with distribution probability is generated by performing multi-label training on the training data, and a self-attention model is trained using the output vector using an S-shaped activation for tilt angle prediction.

[0125] Obtaining pixel offset and loss in each image frame and an area to be cropped on each image frame includes obtaining a minimum tilt angle of multiple image frames based on the tilt angle of each image frame, obtaining an average tilt angle deviation of multiple image frames based on the tilt angle deviation of each image frame, and determining whether the minimum tilt angle is greater than the average tilt angle deviation.

[0126] Obtaining pixel shift and loss in each image frame and an area to be cropped on each image frame includes performing one of the following: (a) setting an average tilt angle deviation as a minimum tilt angle, and obtaining an angle for correcting the tilt of each image frame based on the minimum tilt angle being greater than the average tilt angle deviation, and (b) obtaining an angle for correcting the tilt of each image frame based on the minimum tilt angle being not greater than the average tilt angle deviation, and (c) obtaining an area to be cropped on each image frame based on the average tilt angle deviation, wherein the target tilt correction has minimum pixel shift and loss.

[0127] Obtaining an angle for correcting the tilt of each image frame includes determining whether the tilt angle of an image frame among the plurality of image frames is one of equal to zero, less than or equal to an average tilt angle deviation, and greater than an average tilt angle deviation.

[0128] Obtaining an angle for correcting the tilt of each image frame includes performing one of the following: (a) based on the tilt angle of the image frame being equal to zero, obtaining the angle for correcting the tilt of the image frame as zero, (b) based on the tilt angle of the image frame being less than or equal to an average tilt angle deviation, obtaining the angle for correcting the tilt of the image frame as the tilt angle, and (c) based on the tilt angle of the image frame being greater than an average tilt angle deviation, obtaining the angle for correcting the tilt of the image frame as the average tilt angle deviation.

[0129] Obtaining pixel offset and loss in each image frame and an area to be cropped on each image frame based on the tilt angle deviation and the tilt angle includes obtaining a maximum tilt angle of multiple image frames based on the tilt angle of each image frame, obtaining an average tilt angle deviation of multiple image frames based on the tilt angle deviation of each image frame, and determining whether the tilt angle of an image frame among the multiple image frames is one of equal to zero and less than or equal to the maximum tilt angle.

[0130] Obtaining a pixel shift and loss in each image frame and an area to be cropped on each image frame based on a tilt angle deviation and a tilt angle includes performing one of the following: (a) obtaining an angle for correcting the tilt of the image frame as zero based on the tilt angle of the image frame being equal to zero, (b) obtaining an angle for correcting the tilt of the image frame as a tilt angle based on the tilt angle of the image frame being less than or equal to a maximum tilt angle, and (c) obtaining an area to be cropped on each image frame based on an average tilt angle deviation, wherein a target tilt correction has a maximum pixel shift and loss.

[0131] Obtaining a desired target tilt correction for each image frame based on the pixel offset and the loss includes determining whether a maximum tilt angle and an average tilt angle deviation of the plurality of image frames correspond to a second threshold condition based on the pixel offset and the loss.

[0132] Obtaining the target tilt correction required for each image frame based on the pixel offset and loss includes performing one of the following: (a) performing a target tilt correction with maximum pixel offset and loss based on the maximum tilt angle and average tilt angle deviation of multiple image frames corresponding to a second threshold condition, and (b) performing a target tilt correction with minimum pixel offset and loss based on the maximum tilt angle and average tilt angle deviation of multiple image frames not corresponding to the second threshold condition.

[0133] Applying target tilt correction and cropping to each image frame on the obtained area of ​​each image frame includes rotating each image frame based on the obtained angle to correct the tilt of the image frame, and cropping the obtained area on each image frame based on the average tilt angle deviation.

[0134] Applying at least one of cropping and outer padding to the obtained area of ​​each image frame includes determining whether the area to be cropped on each image frame corresponds to a third threshold condition.

[0135] Applying at least one of cropping and external padding to the obtained area of ​​each image frame includes performing one of: (a) cropping the plurality of image frames based on the area to be cropped on each image not corresponding to a third threshold condition, and (b) cropping and external padding the plurality of image frames based on the area to be cropped on each image corresponding to the third threshold condition. As used herein, "based on" encompasses at least based on.

[0136] The embodiments disclosed herein may be implemented using at least one hardware device and performing a network management function to control elements.

[0137] The foregoing description of the specific embodiments will so fully reveal the general nature of the embodiments herein that others can easily modify and / or adapt the various applications of these specific embodiments by applying current knowledge without departing from the general concept, and therefore, these adaptations and modifications should and are intended to be understood within the meaning and scope of the equivalents of the disclosed embodiments. It should be understood that the wording or terminology employed herein is for the purpose of description rather than limitation. Therefore, although the embodiments herein have been described according to preferred embodiments, those skilled in the art will recognize that the embodiments herein can be practiced by modification within the scope of the embodiments as described herein. Although the present disclosure has been shown and described with reference to various embodiments, it should be understood that the various embodiments are intended to be illustrative rather than restrictive. Those skilled in the art will further understand that various changes can be made in form and detail without departing from the true spirit and full scope of the present disclosure (including the appended claims and their equivalents). It should also be understood that any embodiment described herein may be used in combination with any other embodiment described herein.

Claims

1. A method for tilt correction of a video by an electronic device, the method comprising: receiving a user selection of the video; identifying a sequence of image frames of the video representing a scene in the video; determining a tilt in each image frame of the sequence of image frames; determining whether a cause of the tilt in the sequence of image frames is unintentional by at least analyzing the tilt in each image frame using a trained artificial intelligence (AI) model; and Based on the cause of the tilt being unintentional, the tilt correction is performed on the sequence of image frames.

2. The method according to claim 1, wherein: Determining whether a cause of the tilt in the sequence of image frames is unintentional comprises: Determining whether the tilt in each image frame satisfies a first threshold condition using the trained AI model; and Based on the tilt in each image frame satisfying the first threshold condition, it is determined that the cause of the tilt in the image frame sequence is unintentional.

3. The method according to claim 1, wherein: Performing the tilt correction on the image frame sequence includes: estimating a tilt angle for each image frame for a maximum tilt correction at each image frame; Determining a tilt angle deviation of each image frame, wherein the tilt angle deviation of an image frame in the sequence of image frames is an absolute value of a difference between a tilt angle of the image frame and an average value of tilt angles of the image frame and adjacent image frames in the sequence of image frames; estimating pixel shift and loss in each image frame and an area to be cropped on each image frame based on the tilt angle deviation and the tilt angle; determining a required target tilt correction for each image frame based on the pixel shifts and losses; applying object tilt correction, and at least one of cropping and external padding to the determined region of each image frame; stitching each of the tilt-corrected image frames to reconstruct the scene; and A video is generated by at least sequentially joining the stitched image frames.

4. The method according to claim 3, wherein: Estimating the tilt angle of each image frame for maximum tilt correction at each image frame includes: Use the trained self-attention model to predict the range of possible tilt angles for each image frame; and The tilt angle is determined by at least determining a weighted average of the predicted tilt angles having a standard deviation of zero or greater.

5. The method according to claim 4, wherein: The self-attention model for tilt angle prediction is trained at least in the following way: determining a range of possible tilt angles of the training image frame based on relative positions of objects in the training image frame, correlation of the objects, and relationships between the objects; generating training data using the range of possible tilt angles; generating an output vector having a distribution probability by at least performing multi-label training on the training data; and The self-attention model is trained with the output vector using a sigmoid activation for tilt angle prediction.

6. The method according to claim 3, wherein: Estimating pixel shift and loss in each image frame and an area to be cropped on each image frame based on the tilt angle deviation and the tilt angle comprises: identifying a minimum tilt angle of the sequence of image frames based on the tilt angle of each image frame; Determining an average tilt angle deviation of the image frame sequence based on the tilt angle deviation of each image frame; determining whether the minimum tilt angle is greater than the average tilt angle deviation; Do at least one of the following: setting the average tilt angle deviation as the minimum tilt angle, and estimating an angle for correcting the tilt of each image frame based on the minimum tilt angle being greater than the average tilt angle deviation, and estimating an angle for correcting the tilt of each image frame based on the minimum tilt angle being no greater than the average tilt angle deviation; and The region to be cropped on each image frame is determined based on the average tilt angle deviation, wherein the target tilt correction has a minimum pixel shift and loss.

7. The method according to claim 6, wherein: Estimating the angle used to correct the tilt of each image frame involves: Determining whether the tilt angle of an image frame in the image frame sequence is one of equal to zero, less than or equal to an average tilt angle deviation, and greater than an average tilt angle deviation; and Do at least one of the following: estimating the angle for correcting the tilt of the image frame to be zero based on the tilt angle of the image frame being equal to zero, estimating the angle used to correct the tilt of the image frame as the tilt angle based on the tilt angle of the image frame being less than or equal to the average tilt angle deviation, and Based on the tilt angle of the image frame being greater than the average tilt angle deviation, the angle for correcting the tilt of the image frame is estimated to be the average tilt angle deviation.

8. The method according to claim 3, wherein: Estimating pixel shift and loss in each image frame and an area to be cropped on each image frame based on the tilt angle deviation and the tilt angle comprises: identifying a maximum tilt angle of the sequence of image frames based on the tilt angle of each image frame; Determining an average tilt angle deviation of the image frame sequence based on the tilt angle deviation of each image frame; Determining whether the tilt angle of an image frame in the image frame sequence is one of equal to zero and less than or equal to the maximum tilt angle; Do at least one of the following: estimating the angle for correcting the tilt of the image frame to be zero based on the tilt angle of the image frame being equal to zero, and estimating the angle used to correct the tilt of the image frame as the tilt angle based on the tilt angle of the image frame being less than or equal to the maximum tilt angle; and The region to be cropped on each image frame is determined based on the average tilt angle deviation, wherein the target tilt correction has a maximum pixel shift and loss.

9. The method according to claim 3, wherein: Determining the target tilt correction required for each image frame based on the pixel shift and loss includes: Determining whether a maximum tilt angle and an average tilt angle deviation of the image frame sequence meet a second threshold condition based on the pixel shift and loss; and Do at least one of the following: performing target tilt correction with maximum pixel shift and loss based on the maximum tilt angle and the average tilt angle deviation of the image frame sequence satisfying the second threshold condition, and Based on the maximum tilt angle and the average tilt angle deviation of the image frame sequence not satisfying the second threshold condition, the target tilt correction with minimum pixel shift and loss is performed.

10. The method according to claim 3, wherein: Applying the target tilt correction and cropping to each image frame over the determined area of ​​each image frame comprises: rotating each image frame based on the estimated angle to correct the tilt of the image frame; and The determined region is cropped on each image frame based on the average tilt angle deviation.

11. The method according to claim 3, wherein: Applying at least one of cropping and external padding to the determined area of ​​each image frame includes: Determining whether the area to be cropped on each image frame satisfies a third threshold condition; and Do at least one of the following: Based on the region to be cropped on each image not satisfying the third threshold condition, cropping the sequence of image frames, and The sequence of image frames is cropped and externally padded based on the region to be cropped on each image satisfying the third threshold condition.

12. An electronic device for video tilt correction, comprising: Memory; processor; a tilt correction controller, the tilt correction controller comprising a processing circuit, the processing circuit coupled to the memory and the processor, the tilt correction controller being configured to: receiving a user selection of the video, identifying a sequence of image frames of the video representing a scene in the video, determining a tilt in each image frame of the sequence of image frames, determining whether the cause of the tilt in the sequence of image frames is unintentional by at least analyzing the tilt in each image frame using a trained artificial intelligence (AI) model, and Based on the cause of the tilt being unintentional, the tilt correction is performed on the sequence of image frames.

13. The electronic device according to claim 12, wherein: Determining whether the cause of the tilt in the image frame sequence is unintentional by analyzing the tilt in each image frame using the trained AI model, including: Determining whether the tilt in each image frame satisfies a first threshold condition using the trained AI model; and Based on the tilt in each image frame satisfying the first threshold condition, it is determined that the cause of the tilt in the image frame sequence is unintentional.

14. The electronic device according to claim 12, wherein: Performing the tilt correction on the image frame sequence includes: estimating a tilt angle for each image frame for a maximum tilt correction at each image frame; Determining a tilt angle deviation of each image frame, wherein the tilt angle deviation of an image frame in the sequence of image frames is an absolute value of a difference between a tilt angle of the image frame and an average value of tilt angles of the image frame and adjacent image frames in the sequence of image frames; estimating pixel shift and loss in each image frame and an area to be cropped on each image frame based on the tilt angle deviation and the tilt angle; determining a required target tilt correction for each image frame based on the pixel shifts and losses; applying object tilt correction, and at least one of cropping and external padding to the determined region of each image frame; stitching each of the tilt-corrected image frames to reconstruct the scene; and A video is generated by at least sequentially joining the stitched image frames.

15. The electronic device according to claim 14, wherein: Estimating the tilt angle of each image frame for maximum tilt correction at each image frame includes: Use the trained self-attention model to predict the range of possible tilt angles for each image frame; and The tilt angle is determined by at least determining a weighted average of the predicted tilt angles having a standard deviation of zero or greater.

Images