Video playback device and control method thereof

Abstract

The present disclosure relates to a video playback device and a control method thereof, the device comprising: a display; a memory storing one or more instructions; and at least one processor configured to display a first video on at least a portion of the display by executing the one or more instructions to perform the following operations: comparing the aspect ratio of the first video with the aspect ratio of a display area of ​​the display, when the aspect ratio of the first video is different from the aspect ratio of the display area, determining a category of the first video based on at least one frame included in the first video, selecting a trained artificial intelligence (AI) model related to the category of the first video, obtaining an extended video by applying at least one frame of the first video to the trained AI model, wherein the extended video has an aspect ratio corresponding to the aspect ratio of the display area, and displaying the extended video in the display area.

Description

[0001] This application is a divisional application of application No. 201980007493.2, entitled “Video playback device and control method thereof”, filed with the China Intellectual Property Office on January 3, 2019. Technical Field

[0002] The present disclosure relates to a video playback device and a control method thereof. Background Art

[0003] A device including a display may output a video on the display upon receiving a signal from the outside, and also output a video in the display using pre-stored data.

[0004] When a device plays a video using a display, when the aspect ratio of the video is different from the aspect ratio of the display area of ​​the display, a portion of the display area of ​​the display may be displayed in black. Recently, various methods for playing a video in a device without a region displayed in black have been proposed.

[0005] In addition, artificial intelligence (AI) technology has recently been used to play or generate videos in devices.

[0006] AI systems refer to computer systems that mimic human-level intelligence. Unlike traditional rule-based intelligent systems, AI systems are systems that are trained, determined, and become more intelligent. The more an AI system is used, the more the recognition rate of the AI ​​system can be improved, and the AI ​​system can understand user preferences more accurately. Therefore, traditional rule-based intelligent systems are gradually being replaced by deep learning-based AI systems.

[0007] AI technology consists of machine learning (deep learning) and basic technologies that utilize machine learning.

[0008] Machine learning is an algorithmic technology that classifies and learns the features of input data, and the underlying technology is a technology that uses a machine learning algorithm such as deep learning and consists of technical fields including language understanding, visual understanding, reasoning / prediction, knowledge representation, operation control, and the like.

[0009] AI technology is applied to various fields as follows. Language understanding is a technology for recognizing and applying / processing human language / characters, and includes natural language processing, machine translation, conversational systems, question and answer, speech recognition / synthesis, etc. Visual understanding is a technology for recognizing and processing objects as in human vision, and includes object recognition, object tracking, image search, human recognition, scene understanding, spatial understanding, image enhancement, etc. Reasoning / prediction is a technology for judging information and logically reasoning and predicting results, and includes knowledge / probability-based reasoning, optimization prediction, preference-based planning, recommendation, etc. Knowledge representation is a technology for automatically processing human experience information into knowledge data, and includes knowledge construction (data generation / classification), knowledge management (data utilization), etc. Operation control is a technology for controlling the autonomous driving of vehicles and the movement of robots, and includes motion control (navigation, collision avoidance and driving), operation control (behavior control), etc. Summary of the invention

[0010] Technical issues

[0011] Provided is a device for displaying a video having an undistorted aspect ratio so that when the aspect ratio of the video is different from the aspect ratio of the screen, black edges are not displayed on the screen of the device.

[0012] The technical problems to be solved are not limited to those described above, and there may be other technical problems.

[0013] Solution

[0014] According to a first aspect of the present disclosure, a device includes: a memory storing at least one program and a first video; a display; and at least one processor configured to display the first video on at least a portion of the display by executing the at least one program, wherein the at least one program includes instructions for the following operations: comparing the aspect ratio of the first video with the aspect ratio of an area where the first video is to be displayed; when the aspect ratio of the first video is different from the aspect ratio of the area, generating a second video corresponding to the aspect ratio of the area by using the first video; and displaying the second video in the area, wherein the generation of the second video is performed by inputting at least one frame of the first video into an AI neural network.

[0015] In addition, the operation of generating a second video may also include: extracting frames included in the first video; generating training data to be input into the AI ​​neural network based on the extracted frames; and training the AI ​​neural network to generate a second video by inputting the training data into the AI ​​neural network.

[0016] In addition, the operation of generating the second video may further include generating the training data by assigning weights to preset pixels located at an edge of a reference frame included in the first video.

[0017] In addition, the device may also include: a communication interface configured to send the first video to a server and receive from the server the AI ​​neural network generated by the server using the first video, and the second video is generated by inputting at least one frame of the first video into the AI ​​neural network received from the server.

[0018] In addition, the operation of generating the second video may further include: when the aspect ratio of the first video is different from the aspect ratio of the area, identifying a black border to be displayed in the area; and generating the second video including an extended video to be displayed in the black border.

[0019] In addition, the operation of generating the second video may also include: training the AI ​​neural network to generate a frame of the second video corresponding to the reference frame by inputting a reference frame included in the first video, at least one previous frame of the reference frame, and at least one subsequent frame of the reference frame into the AI ​​neural network.

[0020] In addition, the operation of generating the second video can be performed by training the AI ​​neural network to generate a frame corresponding to the reference frame of the second video by inputting a reference frame included in the first video and a resized frame obtained by adjusting the size of the reference frame into the AI ​​neural network.

[0021] In addition, the device may also include: a communication interface configured to perform communication via the Internet, and the operation of generating a second video may include: detecting at least one of a pattern and a color constituting a reference frame included in the first video; searching for an image related to the at least one of the detected pattern and color by using the communication interface; and training the AI ​​neural network to generate a frame of the second video corresponding to the reference frame by inputting the reference frame and the searched image into the AI ​​neural network.

[0022] In addition, the operation of generating a second video may also include: training the AI ​​neural network to generate a frame of the second video corresponding to the reference frame by inputting a reference frame, a searched image, and a resized image obtained by adjusting the size of the reference frame into the AI ​​neural network, wherein the aspect ratio of the resized image is adjusted to correspond to the aspect ratio of the area by adjusting the size of the searched image.

[0023] According to a second aspect of the present disclosure, a device includes: a memory storing at least one program; a communication interface configured to receive a video list from a server; a user input interface configured to receive user input for selecting a first video to be downloaded from the video list; a display including an area where the first video will be displayed; and at least one processor configured to play the first video by executing the at least one program, wherein the at least one program includes instructions for the following operations: receiving the video list from the server, receiving user input for selecting the first video, sending identification information of the first video and information about the aspect ratio of the area to the server, comparing the aspect ratio of the first video with the aspect ratio of the area, when the aspect ratio of the first video is different from the aspect ratio of the area, obtaining a second video corresponding to the aspect ratio of the area by using the first video, and displaying the second video in the area, wherein the second video is generated by inputting at least one frame of the first video into an AI neural network.

[0024] According to a third aspect of the present disclosure, a method for displaying a video includes: comparing an aspect ratio of a first video with an aspect ratio of an area where the first video will be displayed; when the aspect ratio of the first video is different from the aspect ratio of the area, generating a second video corresponding to the aspect ratio of the area by using the first video; and displaying the second video in the area, wherein the generation of the second video is performed by inputting at least one frame of the first video into an AI neural network.

[0025] In addition, the step of generating a second video may also include: extracting frames included in the first video; generating training data to be input into the AI ​​neural network based on the extracted frames; and training the AI ​​neural network to generate a second video by inputting the training data into the AI ​​neural network.

[0026] In addition, the step of generating the second video may further include generating training data by assigning weights to preset pixels located at an edge of a reference frame included in the first video.

[0027] In addition, the step of generating a second video may also include: sending the first video to a server; and receiving from the server the AI ​​neural network generated by the server using the first video, and the second video is generated by inputting at least one frame of the first video into the AI ​​neural network received from the server.

[0028] In addition, the step of generating the second video may further include: when the aspect ratio of the first video is different from the aspect ratio of the area, identifying a black border to be displayed in the area; and generating the second video including an extended video to be displayed in the black border.

[0029] In addition, the step of generating the second video may further include: training the AI ​​neural network to generate a frame of the second video corresponding to the reference frame by inputting a reference frame included in the first video, at least one previous frame of the reference frame, and at least one subsequent frame of the reference frame into the AI ​​neural network.

[0030] In addition, the step of generating the second video may further include: training the AI ​​neural network to generate a frame of the second video corresponding to the reference frame by inputting a reference frame included in the first video and a resized frame obtained by adjusting the size of the reference frame into the AI ​​neural network.

[0031] In addition, the step of generating a second video may further include: detecting at least one of a pattern and a color constituting a reference frame included in the first video; searching for an image related to the at least one of the detected patterns and colors; and training the AI ​​neural network to generate a frame of the second video corresponding to the reference frame by inputting the reference frame and the searched image into the AI ​​neural network.

[0032] In addition, the step of generating a second video may also include: training the AI ​​neural network to generate a frame of the second video corresponding to the reference frame by inputting a reference frame, a searched image, and a resized image obtained by adjusting the size of the reference frame into the AI ​​neural network, wherein the aspect ratio of the resized image is adjusted to correspond to the aspect ratio of the area by adjusting the size of the searched image.

[0033] According to a fourth aspect of the present disclosure, a method for displaying a video includes: receiving a video list from a server; receiving user input for selecting a first video to be downloaded from the video list; sending identification information of the first video and information about the aspect ratio of an area where the first video will be displayed to the server; comparing the aspect ratio of the first video with the aspect ratio of an area where the first video will be displayed, and when the aspect ratio of the first video is different from the aspect ratio of the area, obtaining a second video corresponding to the aspect ratio of the area by using the first video, and displaying the second video in the area, wherein the second video is generated by inputting at least one frame of the first video into an AI neural network.

[0034] According to a fifth aspect of the present disclosure, a method for displaying a video includes: comparing an aspect ratio of a first video with an aspect ratio of a screen included in a display that will display the first video; identifying the existence of an extended video generation module corresponding to the first video based on the comparison result; generating a first extended video generation module corresponding to the first video by updating a pre-stored training model for generating an extended video using at least a portion of the first video based on the identification result; inputting the first video into the first extended video generation module to generate an extended video to be displayed in a black border; and displaying a second video generated by combining the first video with the extended video on the screen, wherein the black border is an area displayed due to the difference between the aspect ratio of the first video and the aspect ratio of the screen when the first video is played on the screen, and the aspect ratio of the second video is the same as the aspect ratio of the screen.

[0035] According to a sixth aspect of the present disclosure, a device includes: a display including a screen that will display a first video; at least one memory storing a training model for generating an extended video and the first video; and at least one processor including at least one of a graphics processing unit (GPU) and an NPU, wherein the at least one processor compares the aspect ratio of the first video with the aspect ratio of the screen, identifies whether an extended video generation module corresponding to the first video is stored in the at least one memory based on the comparison result, generates a first extended video generation module corresponding to the first video by updating a pre-stored training model for generating an extended video using at least a portion of the first video via at least one of the GPU and the NPU based on the identification result, and controls the at least one memory to store the generated first extended video generation module, generates an extended video to be displayed in a black border by inputting the first video into the first extended video generation module via at least one of the GPU and the NPU, and controls the display to display a second video generated by combining the first video with the extended video on the screen, wherein the black border is an area displayed due to the difference between the aspect ratio of the first video and the aspect ratio of the screen when the first video is played on the screen. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 is a diagram exemplarily illustrating a screen of an apparatus for displaying an extended video on a black border area according to an embodiment.

[0037] Figure 2 2 is a diagram illustrating various shapes of black borders displayed on a screen of a device according to an embodiment.

[0038] Figure 3 is a diagram illustrating a user interface (UI) for displaying an extended video on a screen of a device according to an embodiment.

[0039] Figure 4 is a flowchart illustrating a method of displaying a video including an extended video on a screen of a display according to an embodiment.

[0040] Figure 5 is a flowchart illustrating a method of generating an extended video according to an embodiment.

[0041] Figure 6 is a flowchart illustrating a method of generating an extended video by using a server according to an embodiment.

[0042] Figure 7 is a flowchart illustrating a method of controlling an apparatus for displaying a video including an extended video using a server according to an embodiment.

[0043] Figure 8 is a flowchart illustrating a method of generating or selecting an extended video according to an embodiment.

[0044] Fig. 9 and Fig.10 is a diagram illustrating an example of inputting frames included in a video to a training model according to an embodiment.

[0045] Fig.11 A diagram illustrating an example of inputting a resized image to a training model according to an embodiment is shown.

[0046] Fig.12 A diagram illustrating an example of inputting a cropped image to a training model according to an embodiment is shown.

[0047] Fig.13 A diagram illustrating an example of applying weights to frames included in a video according to an embodiment is shown.

[0048] Fig.14 is a flowchart illustrating a method of inputting found images as training data into a training model according to an embodiment.

[0049] Fig.15 A diagram illustrating an example of inputting a found image as training data into a training model according to an embodiment is shown.

[0050] Fig.16 and Fig.17 is a block diagram of an apparatus 1000 according to an embodiment.

[0051] Fig.18 is a block diagram of a server 2000 according to an embodiment.

[0052] Fig.19 is a block diagram of a processor 1300 according to an embodiment.

[0053] Fig. 20is a block diagram of a data learner 1310 according to an embodiment.

[0054] Fig.21 is a block diagram of the data identifier 1320 according to an embodiment.

[0055] Fig. 22 is a diagram illustrating an example in which the device 1000 and the server 2000 interact to learn and recognize data according to an embodiment.

[0056] Fig.23 is a schematic block diagram of an apparatus according to an embodiment.

[0057] Fig.24 is a diagram for describing a process of generating an extended video generating module performed by a device according to an embodiment.

[0058] Fig.25 A diagram for describing a process of generating an extended video according to a method of downloading a first video from a server, performed by a device according to an embodiment, is shown.

[0059] Fig.26 is a diagram for describing a sequence of generating an extended video of a frame performed by a device according to an embodiment.

[0060] Fig. 27 is a diagram for describing a case where resizing a video and generating an expanded video are performed by a device according to an embodiment. DETAILED DESCRIPTION

[0061] The present disclosure will now be described more fully with reference to the accompanying drawings so that those of ordinary skill in the art can perform the present disclosure without any difficulty. However, the present disclosure can be implemented in many different forms and should not be construed as being limited to the embodiments set forth herein. In the accompanying drawings, for the sake of clarity, parts not related to the description are omitted to clearly describe the present disclosure, and throughout the specification, the same or similar elements are represented by the same reference numerals.

[0062] Some embodiments of the present disclosure may be described according to function block components and various processing steps. Some or all of the function blocks may be implemented by any number of hardware and / or software components configured to perform a specified function. For example, the function blocks of the present disclosure may be implemented by one or more microprocessors, or may be implemented by circuit components for preset functions. In addition, for example, the function blocks of the present disclosure may be implemented with any programming or various scripting languages. The function blocks may be implemented with algorithms executed on one or more processors. In addition, the present disclosure may adopt any number of technologies according to the prior art for electronic configuration, signal processing and / or data processing, etc. The terms "mechanism", "element", "component", "configuration", etc. may be used extensively and are not limited to mechanical or physical components.

[0063] Throughout the specification, it will also be understood that when an element is referred to as being “connected to” another element, it may be “directly connected to” the other element, or the element may be “electrically connected to” the other element by interposing an intermediate element therebetween. Throughout the specification, unless otherwise specified, the term “comprising” an element does not exclude other elements but further includes an element.

[0064] In addition, the connecting lines or connectors between the elements shown in the drawings are intended to represent exemplary functional connections and / or physical connections or circuit connections between the elements. In actual devices, the connections between the elements can be represented by various functional connections, physical connections or circuit connections that can be replaced or added.

[0065] It will be understood that although the terms "first", "second", etc. may be used herein to describe various elements, these elements should not be limited by these terms. The above terms are only used to distinguish one component from another.

[0066] Hereinafter, for the convenience of description, at least a portion of the screen of the device 1000 displaying the first video 3000 is referred to as a first region. In addition, hereinafter, for the convenience of describing the first region, the first region will be described as the entire region of the screen of the device 1000, but the embodiment is not limited thereto.

[0067] Hereinafter, the present disclosure will be described in detail with reference to the accompanying drawings.

[0068] Figure 1 is a diagram exemplarily illustrating a screen of the device 1000 displaying an extended video 3020a and an extended video 3020b in a black border 3010a and a black border 3010b according to an embodiment.

[0069] Reference Figure 1 , the device 1000 may play and display the first video 3000 in the first area.

[0070] The aspect ratio of the first video 3000 displayed in the first region may be different from the aspect ratio of the first region in which the first video 3000 is displayed. When the aspect ratio of the first video 3000 is different from the aspect ratio of the first region, when the video is displayed in the first region, black borders 3010a and 3010b may be displayed in the first region.

[0071] When the black border 3010a and the black border 3010b are displayed in the first region, the user will feel that the first video 3000 is smaller than the actual size. Usually, the first video 3000 is displayed on the first region after the top and bottom portions of the first video 3000 or the left and right portions of the first video 3000 are cropped so that the black border 3010a and the black border 3010b are not displayed in the first region. Optionally, the size of the first video 3000 is adjusted so that the aspect ratio of the first video 3000 corresponds to the aspect ratio of the first region.

[0072] When a portion of the first video 3000 is cropped, all information included in the first video 3000 may not be displayed in the first region. Since the user only views a portion of the first video 3000, there is a problem that the user may miss the intention of the producer of the video.

[0073] In addition, when the size of the first video 3000 is adjusted, the aspect ratio of the first video 3000 is modified, causing the user to watch a distorted video. There is a problem that the user feels uncomfortable when watching the distorted video.

[0074] In order to solve the above problems, the present disclosure provides an apparatus and a method for displaying video by using AI neural network technology to generate extended video 3020a and extended video 3020b so that the first video 3000 is not distorted, does not display black edges, and displays all information included in the first video 3000.

[0075] The present disclosure also provides an apparatus for generating a second video including the generated extended video 3020a and the extended video 3020b and the first video 3000, and a method for displaying the video. The present disclosure also provides an apparatus for generating a second video whose aspect ratio corresponds to the aspect ratio of the first region, and a method for displaying the video. The present disclosure also provides an apparatus for displaying the generated second video in the first region, and a method for displaying the video.

[0076] The extended video 3020a and the extended video 3020b refer to images generated using the AI ​​neural network technology and displayed in the black border 3010a and the black border 3010b without being inconsistent with the first video 3000.

[0077] The first video 3000 may include at least one scene, and the scene may include at least one segment. A segment may include at least one frame.

[0078] According to an embodiment of the present disclosure, the extended video 3020a and the extended video 3020b may be generated by inputting at least one frame included in the first video 3000 to the AI ​​neural network. In addition, the second video may be generated by inputting at least one frame included in the first video 3000 to the AI ​​neural network.

[0079] According to an embodiment of the present disclosure, the AI ​​neural network may include at least one of a learner, a training model, and an extended video generation module. The extended video generation module may be generated by inputting training data into the training model. By inputting at least one frame included in the first video 3000 into the extended video generation module, at least one of the extended video and the second video may be generated. The training model and the extended video generation module may be composed of one AI neural network or may be composed of independent AI neural networks. The learner may be an algorithm configured to update the training model by inputting training data into the training model to generate the extended video generation module 2440.

[0080] According to an embodiment of the present disclosure, the device 1000 may be manufactured to have a pre-generated extended video generation module built therein. The device 1000 may generate an extended video by inputting a video to the internally built extended video generation module.

[0081] According to an embodiment of the present disclosure, the device 1000 may be manufactured to have a training model built therein. The device 1000 may generate an extended video generation module by applying training data to the training model. The device 1000 may refine the extended video generation module by continuously applying training data to the training model. For example, the device 1000 may input training data to the training model, and control the learner to generate (or refine) the extended video generation module by updating the training model using the training data. Fig.24 A process of generating an extended video generating module using a learner is described. The device 1000 may generate an extended video 3020a and an extended video 3020b by inputting the first video 3000 to the extended video generating module.

[0082] According to an embodiment of the present disclosure, the device 1000 may receive an extended video generation module generated by a training model built in the server 2000 from the server 2000. The device 1000 may receive a refined extended video generation module by applying training data to the training model built in the server 2000. For example, the device 1000 may receive an extended video generation module generated by inputting the training model and the training data to a learner included in the server 2000. The device 1000 may generate an extended video 3020a and an extended video 3020b by inputting the first video 3000 to the extended video generation module.

[0083] According to an embodiment of the present disclosure, the device 1000 may receive a second video generated in the server 2000 and including a first video and an extended video corresponding to the first video. The device 1000 may display the second video on the screen. In this case, the extended video 3020a and the extended video 3020b corresponding to the first video may be a video generated by inputting at least one frame of the first video to an extended video generation module generated by a training model built in the server 2000.

[0084] Figure 2 2 is a diagram illustrating various shapes of black borders displayed on a screen of a device according to an embodiment.

[0085] Reference Figure 2 , black borders can be displayed in various shapes in the first area. For example, black borders 3011a and black borders 3011b can be displayed at the portion near the left and right sides of the video 3000a displayed in the first area of ​​the device 1000a. In another example, black borders 3012a and black borders 3012b can be displayed at the portion near the top side and bottom side of the video 3000b displayed in the first area of ​​the device 1000b. In another example, black borders 3013 can be displayed at the portion near all four sides of the video 3000c displayed in the first area of ​​the device 1000c. In another example, black borders 3014a and black borders 3014b can be displayed at the portion near at least one pair of the left and right sides and the top side and bottom side of the video 3000d displayed in the first area of ​​the device 1000d.

[0086] According to an embodiment of the present disclosure, an extended video to be displayed in a black border can be generated regardless of the shape of the black border displayed in the first area. That is, according to the present disclosure, when a black border is displayed on at least one of a portion near the left and right sides of a video, a portion near the top and bottom sides of a video, a portion near all four sides of a video, a portion near the left and right sides, and at least one pair of portions near the top and bottom sides, an extended video to be displayed in a black border area can be generated. In the following, although the case of displaying black borders on the left and right sides of a video is described by way of example, it is obvious to those skilled in the art that the present disclosure may also be applied to other situations.

[0087] Figure 3 is a diagram illustrating a user interface UI for displaying an extended video on a screen of a device according to an embodiment.

[0088] Reference Figure 3 , the first area may include an icon 3030 allowing a user to select a shape in which the first video 3000 is displayed.

[0089] According to an embodiment, the icons 3030 may include an icon for displaying the first video 3000 in the original aspect ratio, an icon for adjusting the size of the first video 3000 according to the aspect ratio of the first region and displaying the resized video, and an icon for cropping a portion of the first video 3000 according to the aspect ratio of the first region and displaying the cropped video. In addition, the icons 3030 may include an icon 3031 for displaying an expanded video.

[0090] According to an embodiment, the device 1000 may receive a user input for modifying the icon 3030. The device 1000 may modify the icon 3030 according to the user input, and display the first video 3000 in the first area to correspond to the modified icon 3030. For example, the device 1000 may modify the icon 3030 to an icon 3031 for displaying an extended video in response to the user input. The device 1000 may display the first video 3000 and the extended videos 3020a and 3020b in the first area to correspond to the modified icon 3031.

[0091] Figure 4 is a flowchart illustrating a method of displaying a video including an extended video on a screen of a display according to an embodiment.

[0092] Reference Figure 4 In operation S410, the device 1000 may compare an aspect ratio (eg, 4:3, 16:9, and 2.35:1) of a first video to be displayed in a first region with an aspect ratio (eg, 3:2, 4:3, 16:9, 16:10, and 18:9) of the first region.

[0093] That is, the aspect ratio of the first video may be determined by the intention of the video producer, and the first video may be generated according to the determined aspect ratio. For example, the first video may be generated with an aspect ratio of 16:9.

[0094] The aspect ratio of the first region may be determined when manufacturing the device 1000. In addition, the aspect ratio of the first region may be determined by a user input received from a user. In addition, the aspect ratio of the first region may be determined by a ratio set by a program running in the device 1000. The aspect ratio of the first region may be 18:9.

[0095] Referring to operation S430, the device 1000 may determine whether the aspect ratio of the first video is different from the aspect ratio of the first region by comparing the aspect ratio of the first video with the aspect ratio of the first region. For example, when the aspect ratio of the first video is 16:9 and the aspect ratio of the first region is 18:9, the device 1000 may determine that the aspect ratio of the first video is different from the aspect ratio of the first region. In another example, when the aspect ratio of the first video is 16:9 and the aspect ratio of the first region is 16:9, the device 1000 may determine that the aspect ratio of the first video is the same as the aspect ratio of the first region.

[0096] Referring to operation S450, the device 1000 may generate an extended video using the first video.

[0097] According to an embodiment, the device 1000 may generate an extended video to be displayed in a black border region to be generated due to an aspect ratio of the first video being different from an aspect ratio of the first region.

[0098] The device 1000 may identify the black border region and generate an extended video to be displayed in the black border region. For example, when the aspect ratio of the first video is 16:9 and the aspect ratio of the first region is 18:9, the device 1000 may identify the black border region at a portion near the left and right sides of the first video. In another example, when the aspect ratio of the first video is 2.35:1 and the aspect ratio of the first region is 16:9, the device 1000 may identify the black border region at a portion near the top and bottom sides of the first video.

[0099] According to an embodiment, the device 1000 may generate an extended video and a second video by inputting the first video to an AI neural network built in the device 1000. In this case, the second video may include at least a portion of the first video and the extended video. For example, the device 1000 may generate an extended video by inputting the first video to an extended video generation module built in the device 1000. In this case, when the device 1000 is manufactured, the extended video generation module built in the device 1000 may be built in the device 1000. Alternatively, the extended video generation module built in the device 1000 may be generated from a training model built in the device 1000. In addition, the extended video generation module built in the device 1000 may be generated in a server and may be received from the server.

[0100] According to an embodiment, the device 1000 may generate an extended video generation module by inputting at least one frame included in the first video to the AI ​​neural network. In this case, the second video may include at least a portion of the first video and the extended video. For example, the device 1000 may generate an extended video by inputting at least one frame included in the first video to the extended video generation module.

[0101] That is, by inputting all frames included in the first video into the extended video generation module, the device 1000 can generate an extended video and a second video respectively corresponding to all frames of the first video. Alternatively, by inputting some frames (e.g., reference frames) included in the first video into the extended video generation module, the device 1000 can generate an extended video and a second video respectively corresponding to all frames of the first video.

[0102] According to an embodiment, the device 1000 can generate an extended video and a second video corresponding to a reference frame and a frame related to the reference frame by inputting the reference frame into an extended video generation module. In this regard, the reference frame is a frame selected as needed to generate an extended video. The reference frame may include a frame substantially input to the extended video generation module. The reference frame may be one or more. The frame related to the reference frame may include a frame similar to the reference frame (e.g., a frame decoded by a decoder with reference to the reference frame), a frame having the same pixels as the pixels of the reference frame reaching a preset ratio or a greater ratio, and a frame played before and after a preset time from the reference frame.

[0103] For example, the reference frame may be an intra frame. An intra frame is a frame that is decoded by a decoder in the absence of other reference frames.

[0104] In another example, the reference frame may be a frame that is most suitable for generating an extended video among the multiple frames of the segment constituting the first video segment. The reference frame may be a frame that is most similar to multiple frames of the multiple frames constituting the segment of the first video. Specifically, the reference frame may be the first frame of the multiple frames constituting the segment. Alternatively, the reference frame may be an intermediate frame of the multiple frames constituting the segment. Alternatively, the reference frame may be a frame of the multiple frames constituting the segment that includes the largest number of pixels that are identical to pixels of other frames.

[0105] When the device 1000 generates an extended video, a large amount of computing resources of the device 1000 are used. Therefore, the device 1000 can reduce the use of computing resources for generating the extended video by inputting only some frames (e.g., reference frames) of the first video to the extended video generation module. That is, the utilization rate of the processor, graphics processor (GPU or neural processing unit (NPU)) or memory (e.g., buffer memory) of the device 1000 can be reduced. In addition, the power consumption of the device 1000 for generating the extended video can be reduced.

[0106] The first video may be composed of a plurality of reference frames. The device 1000 may generate a plurality of extended videos and a second video respectively corresponding to the plurality of reference frames by inputting the respective reference frames into the extended video generation module. For example, the extended video generation module may generate a plurality of extended videos respectively corresponding to the plurality of reference frames in response to the respective inputs of the plurality of reference frames. The device 1000 may generate a second video by combining the plurality of reference frames and the extended videos respectively corresponding to the plurality of reference frames. In another example, the extended video generation module may generate a second video in which the plurality of reference frames are respectively combined with a plurality of extended videos corresponding to the plurality of reference frames in response to the respective inputs of the plurality of reference frames.

[0107] According to various embodiments, the AI ​​neural network (or training model) may include a generative adversarial network (GAN). The GAN model may include, for example, a generator and a discriminator. In this case, the device 1000 may generate an extended video by inputting the first video to the generator.

[0108] Referring to operation S470, the device 1000 may display the second video including at least a portion of the first video and the extended video in the first region. For example, the device 1000 may display the second video including the reference frame and the extended video corresponding to the reference frame in the first region.

[0109] According to an embodiment, the reference frame may be one or more in number, and the first video may be composed of a plurality of reference frames. In addition, a plurality of extended videos may be provided in number. In this case, a plurality of extended videos may be generated by inputting a plurality of reference frames into an extended video generation module to correspond to the plurality of reference frames respectively. The second video may be a video in which the plurality of reference frames are respectively combined with the extended videos corresponding to the plurality of reference frames respectively.

[0110] Figure 5 is a flowchart illustrating a method of generating an extended video according to an embodiment.

[0111] Reference Figure 5 , the device 1000 may generate an extended video by using an AI neural network built in the device 1000. Specifically, the device 1000 may generate training data and train the AI ​​neural network by inputting the generated training data into the AI ​​neural network. The device 1000 may generate an extended video by inputting the first video into the trained AI neural network.

[0112] For example, the extended video generation module may be generated by using a training model built in the device 1000. The device 1000 may generate the extended video by using the generated extended video generation module.

[0113] According to an embodiment, the device 1000 may generate an extended video generation module and an extended video corresponding to a video stored in the device 1000 during a time when the user does not use the device 1000 .

[0114] Referring to operation S510, the device 1000 may retrieve a first video stored in the device 1000. The device 1000 may retrieve a video including the first video stored in the device 1000 during a time when the device 1000 is not used by a user.

[0115] According to an embodiment, the device 1000 may store the search result of the video in the device 1000. In this case, the device 1000 may store information related to the video stored in the device 1000. For example, the device 1000 may store in the device 1000 a path where the video is stored in the memory 1700, a playback time of the video, an aspect ratio of the video, whether an extended video has been generated, and the like.

[0116] Referring to operation S530, the device 1000 may extract at least one frame included in the first video. That is, the device 1000 may generate an extended video generation module corresponding to the first video by inputting all frames included in the first video. Alternatively, the device 1000 may generate an extended video generation module corresponding to the first video by inputting some frames (e.g., reference frames) among the frames included in the first video.

[0117] The device 1000 may extract at least one frame included in the first video during a time when the user does not use the device 1000 .

[0118] According to an embodiment, the device 1000 may extract a reference frame included in the first video. In this case, the reference frame is a frame selected to generate an extended video as needed. The reference frame may include a frame required to generate an extended video corresponding to a plurality of frames constituting a segment of the first video. The reference frame may be one or more.

[0119] For example, the reference frame may be an intra frame. The reference frame may be a frame to be decoded by a decoder by referring to the intra frame.

[0120] In another example, the reference frame may be a frame that is most similar to other frames among a plurality of frames constituting a segment of the first video. Specifically, the reference frame may be the first frame among the plurality of frames constituting the segment. Alternatively, the reference frame may be an intermediate frame among the plurality of frames. Alternatively, the reference frame may be a frame among the plurality of frames that includes the largest number of pixels that are identical to pixels of other frames.

[0121] When the device 1000 updates the training model by applying the training data to the training model, a large amount of computing resources of the device 1000 are used. Therefore, by inputting only some frames (e.g., reference frames) of the first video as training data to the training model, the device 1000 can reduce the use of computing resources for updating the training model. That is, the utilization rate of the processor, GPU (or NPU), or memory (e.g., buffer memory) of the device 1000 can be reduced. In addition, the power consumption of the device 1000 for updating the training model can be reduced.

[0122] Referring to operation S550, the device 1000 may generate training data based on the extracted frame. The device 1000 may generate the training data during a time when the user does not use the device 1000.

[0123] For example, the device 1000 may generate all frames of the first video as training data.

[0124] In another example, the device 1000 may generate the extracted reference frame and at least one previous frame played before the reference frame as training data. In another example, the device 1000 may generate the reference frame and at least one subsequent frame played after the reference frame as training data. In another example, the device 1000 may generate the reference frame and the resized frame obtained by adjusting the size of the reference frame as training data. In another example, the device 1000 may generate the reference frame as training data by applying weights to preset pixels located at the edge of the reference frame. In another example, the device 1000 may generate the reference frame and the cropped frame obtained by cropping a part of the reference frame as training data. In another example, the device 1000 may generate the reference frame and the image related to the reference frame as training data. In this case, the image related to the reference frame may be an image related to at least one of a pattern and a color constituting the reference frame. The image related to the reference frame may be an image obtained by searching the Internet. In another example, the device 1000 may generate the reference frame, the image related to the reference frame, and the resized image obtained by adjusting the size of the image related to the reference frame as training data. The training data used to train the model is not limited thereto.

[0125] The following will refer to Figures 9 to 15 Describe the training data.

[0126] Referring to operation S570, the device 1000 may generate an extended video generating module by inputting training data into a training model. The device 1000 may generate an extended video generating module during a time when a user does not use the device 1000. The device 1000 may refine the generated extended video generating module by applying the training data to the training model.

[0127] According to an embodiment, the device 1000 may generate an extended video generation module by applying the training data described above in operation S550 to a training model constructed in the device 1000. For example, the device 1000 may input the training model and the training data into a learner, and control the learner to generate (or refine) the extended video generation module by updating the training model using the training data. In this case, the generated extended video generation module may be a dedicated module for displaying the first video in the first area. According to an embodiment, the manner in which the training data is applied to the training model may be determined based on training performed based on a preset standard. For example, supervised learning, unsupervised learning, and reinforcement learning may be used to apply the training data to the training model, and deep neural network technology may also be used to apply the training data to the training model.

[0128] Referring to operation S590, the device 1000 may generate an extended video using the extended video generating module. The device 1000 may generate the extended video by inputting the first video to the extended video generating module during a time when the user does not use the device 1000.

[0129] According to an embodiment, the device 1000 may extract a reference frame included in the first video, and input the extracted reference frame to an extended video generation module, thereby generating an extended video corresponding to the reference frame. In this case, the reference frame is a frame selected as needed to generate an extended video. The reference frame may be one or more in number, and the first video may include multiple reference frames. In addition, the extended video may be multiple in number. In this case, multiple extended videos may be generated by inputting the multiple reference frames to the extended video generation module to correspond to the multiple reference frames, respectively. Figure 4 A method of generating an extended video by inputting a reference frame to an extended video generating module is described, and thus a repeated description will be omitted.

[0130] According to an embodiment, the device 1000 may display in the first area a second video generated by combining a reference frame with an extended video corresponding to the reference frame. In this case, the second video may be a video in which a plurality of reference frames are respectively combined with a plurality of extended videos respectively corresponding to the plurality of reference frames. Figure 4 The method of displaying the second video has been described, and thus a repeated description will be omitted.

[0131] Figure 6 is a flowchart illustrating a method of generating an extended video by using the server 2000 according to an embodiment.

[0132] Reference Figure 6, the device 1000 may generate an extended video using an AI neural network built in the server 2000. The device 1000 may receive the AI ​​neural network from the server 2000, and generate an extended video by using the received AI neural network. In addition, the device 1000 may receive a refined AI neural network from the server 2000.

[0133] For example, the device 1000 may receive an extended video generating module generated by a training model built in the server 2000 from the server 2000. In addition, the device 1000 may receive a refined extended video generating module by applying training data to the training model built in the server 2000.

[0134] Referring to operation S610 , the device 1000 may transmit a first video to the server 2000 .

[0135] According to an embodiment, the device 1000 may transmit at least a portion of the first video stored in the device 1000 to the server 2000 .

[0136] For example, the device 1000 may transmit at least one frame extracted from the first video to the server 2000. That is, the device 1000 may transmit all frames stored in the first video to the server 2000. Alternatively, the device 1000 may transmit some frames (e.g., reference frames) stored in the first video to the server 2000. The reference frames are frames selected as needed to generate the extended video generation module. The reference frames may include frames required to generate an extended video corresponding to a plurality of frames constituting a segment of the first video. Figure 5 The reference frame is described, so a repeated description will be omitted.

[0137] A large amount of network resources are used when the device 1000 transmits the first video to the server 2000. Therefore, by transmitting only some frames (e.g., reference frames) of the first video to the server 2000, the device 1000 can reduce the amount of data transmission. In addition, the power consumption of the device 1000 for transmitting the first video to the server 2000 can be reduced.

[0138] In another example, the device 1000 may transmit a cropped image obtained by cropping a portion of the first video to the server 2000 .

[0139] According to an embodiment, the device 1000 may transmit information about the first video posted on the Internet to the server 2000. For example, the device 1000 may transmit an identifier (e.g., URL) of the first video to the server 2000. In another example, the device 1000 may transmit at least one frame extracted from the first video to the server 2000. In another example, the device 1000 may transmit a video composed of a plurality of reference frames extracted from the first video.

[0140] Referring to operation S620, the server 2000 may extract a reference frame included in the received first video.

[0141] According to an embodiment, the server 2000 may extract a reference frame included in the first video. In this case, the reference frame is a frame selected as needed to generate an extended video generation module. The reference frame may include a frame required to generate an extended video corresponding to a plurality of frames constituting a segment of the first video. Figure 5 The reference frame is described, and thus a repeated description will be omitted. In addition, the server 2000 may extract at least one previous frame played before the reference frame. In addition, the server 2000 may extract at least one subsequent frame played after the reference frame.

[0142] According to an embodiment, when a reference frame is received from the device 1000, the server 2000 may omit an operation of extracting the reference frame from the first video.

[0143] Referring to operation S630, the server 2000 may generate training data based on the extracted frames.

[0144] For example, the server 2000 may generate the extracted reference frame as training data. In another example, the server 2000 may generate the extracted reference frame and at least one previous frame played before the reference frame as training data. In another example, the server 2000 may generate the reference frame and at least one subsequent frame played after the reference frame as training data. In another example, the server 2000 may generate the reference frame and the resized frame obtained by adjusting the size of the reference frame as training data. In another example, the server 2000 may generate the reference frame as training data by applying weights to preset pixels located at the edge of the reference frame. In another example, the server 2000 may generate the reference frame and the cropped frame obtained by cropping a part of the reference frame as training data. In another example, the server 2000 may generate the reference frame and the image related to the reference frame as training data. In this case, the image related to the reference frame may be an image related to at least one of the pattern and color constituting the reference frame. The image related to the reference frame may be an image obtained by searching the Internet. In another example, the device 1000 may generate a reference frame, an image related to the reference frame, and a resized image obtained by resizing the image related to the reference frame as training data. The training data applied to the training model is not limited thereto.

[0145] The following will refer to Figures 9 to 15 Describe the training data.

[0146] Referring to operation S640, the server 2000 may generate an extended video generating module by applying the training data to the training model. In addition, the server 2000 may refine the generated extended video generating module by applying the training data to the training model.

[0147] According to an embodiment, the server 2000 may generate an extended video generation module by applying the training data described in operation S630 to the training model. For example, the server 2000 may input the training model and the training data into the learning model, and control the learning model to generate (or refine) the extended video generation module by updating the training model using the training data. In this case, the generated extended video generation module may be a dedicated module for displaying the first video in the first area. The manner in which the training data is applied to the training model may be determined based on the training performed based on a preset standard. For example, supervised learning, unsupervised learning, and reinforcement learning may be used to apply the training data to the training model, and deep neural network technology may also be used to apply the training data to the training model.

[0148] Referring to operation S650, the server 2000 may transmit the generated extended video generating module to the device 1000. In this case, the transmitted extended video generating module may be a dedicated module for displaying the first video in the first region.

[0149] According to an embodiment, the server 2000 may transmit the extended video generating module to the device 1000 during a time when the user does not use the device 1000 .

[0150] According to an embodiment, in case that a user requests to transmit the extended video generating module to the device 1000 , the server 2000 may transmit the extended video generating module to the device 1000 .

[0151] According to an embodiment, in a case where the device 1000 displays the first video, the server 2000 may transmit the extended video generating module to the device 1000 .

[0152] Referring to operation S660, the device 1000 may generate an extended video by using the extended video generating module received from the server. The device 1000 may generate an extended video by inputting the first video to the extended video generating module during a time when the user does not use the device 1000.

[0153] That is, by inputting all frames included in the first video into the extended video generation module, the device 1000 can generate an extended video and a second video respectively corresponding to all frames of the first video. Alternatively, by inputting some frames (e.g., reference frames) included in the first video into the extended video generation module, the device 1000 can generate an extended video and a second video respectively corresponding to all frames of the first video.

[0154] According to an embodiment, the device 1000 may extract a reference frame included in the first video, and input the extracted reference frame to the extended video generation module, thereby generating an extended video corresponding to the reference frame. In this case, the reference frame may be one or more in number, and the first video may be composed of a plurality of reference frames. In addition, the extended video may be plural in number. In this case, a plurality of extended videos may be generated by inputting the plurality of reference frames to the extended video generation module to correspond to the plurality of reference frames, respectively. Figure 4 A method of generating an extended video by inputting a reference frame to an extended video generating module is described, and thus a repeated description will be omitted.

[0155] According to an embodiment, the device 1000 may display a second video generated by combining a reference frame with an extended video corresponding to the reference frame in the first region. In this case, the second video may be a video combining a plurality of reference frames with a plurality of extended videos respectively corresponding to the plurality of reference frames.

[0156] Figure 7 is a flowchart illustrating a method of controlling a device for displaying a video including an extended video using the server 2000 according to an embodiment.

[0157] Reference Figure 7 , the device 1000 may receive a second video generated by the server 2000 and including a first video and an extended video corresponding to the first video. The device 1000 may display the received second video on the screen. In this case, the extended video 3020a and the extended video 3020b corresponding to the first video may be videos generated by an AI neural network built in the server 2000.

[0158] For example, extended videos 3020 a and 3020 b corresponding to the first video may be generated by inputting at least one frame of the first video to an extended video generation module generated by a training model built in the server 2000 .

[0159] Referring to operation S710, the server 2000 may transmit a video list to the device 1000. The video list refers to a list of videos that may be transmitted by the server 2000 to the device 1000.

[0160] According to an embodiment, the server 2000 may be a server that provides a VOD service. In another example, the server 2000 may be a server that provides a service that allows users to upload videos and share the uploaded videos.

[0161] In addition, the device 1000 may display the received video list on the screen of the device 1000. The video list displayed on the screen of the device 1000 may be a list including at least one frame included in each video as a thumbnail.

[0162] Referring to operation S720, the device 1000 may receive a user input for selecting a first video included in the video list from the user. For example, the device 1000 may receive a user input for selecting the first video from the user using a screen including a touch screen. In another example, the device 1000 may receive a user input for selecting the first video from the user by using an input device such as a remote control, a keyboard, and a mouse.

[0163] Referring to operation S730, the device 1000 transmits identification information (eg, an identifier) ​​of the selected first video and information about an aspect ratio (eg, 18:9) of the first region to the server.

[0164] Referring to operation S740, the server 2000 may compare the aspect ratio of the first video (e.g., 16:9) with the aspect ratio of the first region (e.g., 18:9) based on the aspect ratio of the first region received from the device 1000. The server 2000 may identify whether the aspect ratio of the first video is different from the aspect ratio of the first region. For example, when the aspect ratio of the first video is 16:9 and the aspect ratio of the first region is 18:9, the server 2000 may determine that the aspect ratio of the first video is different from the aspect ratio of the first region. In another example, when the aspect ratio of the first video is 16:9 and the aspect ratio of the first region is 16:9, the server 2000 may determine that the aspect ratio of the first video is the same as the aspect ratio of the first region.

[0165] Referring to operation S750, the server 2000 may generate an extended video by using the first video.

[0166] According to an embodiment, because the aspect ratio of the first video is different from that of the first region, the server 2000 may generate an extended video to be displayed in the black border region when the first video is displayed in the first region.

[0167] The server 2000 may identify the black border region and generate an extended video to be displayed in the black border region. For example, when the aspect ratio of the first video is 16:9 and the aspect ratio of the first region is 18:9, the device 1000 may identify the black border region at the portion near the left and right sides of the first video. In another example, when the aspect ratio of the first video is 2.35:1 and the aspect ratio of the first region is 16:9, the device 1000 may identify the black border region at the portion near the top and bottom sides of the first video.

[0168] According to an embodiment, the server 2000 may generate an extended video and a second video by inputting at least one frame included in the first video to an AI neural network. In this case, the second video may include at least a portion of the first video and the extended video.

[0169] For example, the server 2000 may generate an extended video by inputting at least one frame included in the first video into the extended video generation module. Specifically, the server 2000 may generate an extended video corresponding to the reference frame by inputting the reference frame included in the first video into the extended video generation module. In this case, the reference frame may be one or more. The first video may be composed of a plurality of reference frames.

[0170] The server 2000 can generate a plurality of extended videos respectively corresponding to a plurality of reference frames by inputting each reference frame into the extended video generation module. Figure 4 A method of generating an extended video by inputting a reference frame to an extended video generating module is described, and thus a repeated description will be omitted.

[0171] Referring to operation S760 , the server 2000 may transmit the second video including at least a portion of the first video and the extended video to the device 1000 .

[0172] According to an embodiment, the server 2000 may transmit the second video to the device 1000 during a time when the user does not use the device 1000 .

[0173] According to an embodiment, the server 2000 may transmit the second video to the device 1000 if the user requests to transmit the second video to the device 1000 .

[0174] According to an embodiment, the server 2000 may transmit the second video to the device 1000 if the device 1000 displays the second video.

[0175] Referring to operation S770, the device 1000 may display the second video including at least a portion of the first video and the extended video in the first region. For example, the device 1000 may display the second video including the reference frame and the extended video corresponding to the reference frame in the first region.

[0176] According to an embodiment, the reference frame may be one or more in number, and the first video may be composed of a plurality of reference frames. In addition, the extended video may be plural in number. In this case, a plurality of extended videos may be generated by inputting the plurality of reference frames into an extended video generation module to correspond to the plurality of reference frames respectively. The second video may be a video composed of a plurality of reference frames and a plurality of extended videos corresponding to the plurality of reference frames respectively.

[0177] Figure 8 is a flow chart showing a method of generating or selecting an extended video according to an embodiment. Figure 8 The disclosed embodiments may be performed by a processor of the device 1000 or a processor of the server 2000. That is, the processor to be described below may be at least one of the processor of the device 1000 and the processor of the server 2000. The processor may include a graphics processing unit (GPU) or a neural network processing unit (NPU).

[0178] Reference Figure 8, the processor may compare the aspect ratio of the first video with the aspect ratio of the screen included in the display on which the first video will be played. The display may include a flexible display having a variable aspect ratio of the screen, wherein the first video is output on the screen by folding at least a portion of the screen or unfolding to at least a portion of the screen. The processor may compare the aspect ratio of the first video with the aspect ratio of the screen modified by folding or unfolding at least a portion of the screen. Whenever the aspect ratio of the screen is changed, the processor may compare the aspect ratio of the first video with the aspect ratio of the modified screen.

[0179] The processor may determine whether to generate a new extended video generation module corresponding to the first video based on the comparison result (S810). The processor may input the first video to the training model 4000 so that the training model 4000 generates a new extended video generation module by using the first video (S820). Alternatively, the processor determines the category of the first video (S830) and selects one of the previously trained extended video generation modules related to the determined category (S840). The processor may generate an extended video by using the extended video generation module (S850).

[0180] Referring to operation S810, the processor may determine whether to generate a new extended video generation module corresponding to the first video. The processor may compare the aspect ratio of the first video with the aspect ratio of the screen on which the first video will be displayed. When the aspect ratio of the first video is different from the aspect ratio of the screen, the processor may identify whether there is an extended video generation module corresponding to the first video. The previously generated extended video generation module may be multiple in number. The previously generated extended video generation module may be stored in the memory of the device 1000 or in the database DB of the server 2000. The processor may identify whether there is an extended video generation module corresponding to the first video among the multiple extended video generation modules.

[0181] According to an embodiment, the processor may determine whether to generate a new extended video generation module based on the existence of an extended video generation module corresponding to the first video. For example, when there is a previously generated extended video generation module corresponding to the first video, the processor may determine not to generate a new extended video generation module. In another example, when a preset time has passed since the previously generated extended video generation module corresponding to the first video, the processor may determine to generate a new extended video generation module. In another example, when the previously generated extended video generation module corresponding to the first video is not configured to match the aspect ratio of the area where the first video will be displayed, the processor may determine to generate a new extended video generation module. The previously generated extended video generation module may be stored in the memory of the device 1000 and the DB of the server 2000. The processor may determine whether to generate a new extended video generation module based on the header information of the previously generated extended video generation module. The header information may include information about the video corresponding to the extended video generation module (for example, the path where the video is stored, the playback time of the video, the aspect ratio of the video, and information about whether the extended video has been generated), information about the generation time of the extended video generation module, and information about the category of the first video.

[0182] According to an embodiment, the processor may determine whether to generate a new extended video generating module based on the existence of an extended video generating module corresponding to a video similar to the first video.

[0183] For example, when there is an extended video generation module corresponding to the category of the first video (e.g., SF movie, documentary about nature, live performance, 2D animation, 3D animation, augmented reality (AR) video, holographic video, etc.), the processor may not generate a new extended video generation module.

[0184] In another example, in a case where an extended video generating module corresponding to a video having a name similar to that of the first video exists, the processor may not generate a new extended video generating module.

[0185] In another example, in the case where there is an extended video generation module corresponding to a video including at least one frame similar to the first video, the processor may not generate a new extended video generation module. Referring to operation S820, the processor may input the first video into the training model 4000 so that the training model 4000 generates a new extended video generation module corresponding to the first video by using the first video. The training model 4000 may generate a new extended video generation module by learning at least one training data related to the first video. The processor may control the training model 4000 to learn via at least one of the GPU and the NPU. For example, the device 1000 may input the training model 4000 and the first video into the learner, and control the learner to update the training model 4000 using the first video and generate a new extended video generation module corresponding to the first video. The following will refer to Figures 9 to 15 A training data for generating an extended video generating module corresponding to a first video is described in detail.

[0186] Referring to operation S830, the processor may determine a category of the first video (eg, SF movie, documentary about nature, live show, 2D animation, 3D animation, augmented reality (AR) video, holographic video, etc.) to select an extended video generating module corresponding to the first video.

[0187] According to an embodiment, the processor may determine the category of the first video based on metadata of the first video. For example, the processor may determine the category of the first video based on header information of the first video. In another example, the processor may determine the category of the first video based on the name of the first video.

[0188] According to an embodiment, the processor may determine the category of the first video based on at least one frame included in the first video. Specifically, since a SF movie may include at least one frame related to the universe, the processor may determine the category of the first video as a SF movie by determining at least one frame included in the first video as a frame related to the universe. In addition, since a documentary about nature may include at least one frame related to nature such as a forest, a desert, and an ocean, the processor may determine the category of the first video as a documentary about nature by determining at least one frame included in the first video as a frame related to nature.

[0189] According to an embodiment, the processor may determine the category of the first video by identifying an object included in at least one frame of the first video. Specifically, when the object of the first video is a 2D animation character, the processor may determine the category of the first video as a 2D animation.

[0190] Referring to operation S840, the processor may select one of previously generated extended video generating modules related to the category of the first video.

[0191] According to an embodiment, the processor may store information about a previously generated extended video generating module based on categories. The information about the previously generated extended video generating module may include information about the category of the video and information about a scene of the video used to generate the extended video generating module.

[0192] For example, the processor may determine the category of the first video as a documentary about nature. The processor may select an extended video generation module generated to correspond to a documentary video about nature among the previously generated extended video generation modules. In addition, the processor may select an extended video generation module generated to correspond to a scene of fishing in the sea.

[0193] In another example, the processor may determine the category as SF movie. The processor may select an extended video generation module generated to correspond to SF movie among previously generated extended video generation modules. In addition, the processor may select an extended video generation module generated to correspond to a scene of a spaceship moving in space.

[0194] According to an embodiment, the processor may select one of the previously generated extended video generation modules based on at least one frame of the first video. For example, at least one frame of the first video may be a frame constituting a scene of a fishing boat moving in the sea. The processor may select an extended video generation module generated to correspond to a scene of fishing in the sea. The processor may select an extended video generation module generated to correspond to a scene of a boat moving in the sea.

[0195] Referring to operation S850, the processor may generate at least one of the extended video and the second video by inputting the first video to the extended video generation module newly generated in operation S820. The processor may generate at least one of the extended video and the second video by inputting the first video to the extended video generation module selected in operation S840. The processor may generate the extended video from the extended video generation module to which the first video is input via at least one of the GPU and the NPU. The processor may generate the second video by combining the first video with the extended video via at least one of the GPU and the NPU. The method of generating at least one of the extended video and the second video is similar to operation S450, operation S590, and operation S660, and therefore repeated descriptions will be omitted.

[0196] Fig. 9 and Fig.10 4000 is a diagram illustrating an example of inputting a frame included in a video to a training model 4000 according to an embodiment.

[0197] At least one frame included in the first video may be input to the AI ​​neural network as training data. In addition, the AI ​​neural network may be trained by the input training data to generate an extended video corresponding to the first video and a second video. In this regard, the second video may include the first video and the extended video corresponding to the first video.

[0198] Reference Fig. 9 , a reference frame 3110 included in the video and at least one of previous frames 3120 - 1 , 3120 - 2 , and 3120 - n played temporally before the reference frame 3110 may be input to the training model 4000 as training data.

[0199] Reference Fig.10 , at least one of a reference frame 3210 included in a video and subsequent frames 3220 - 1 , 3220 - 2 , and 3220 - n played temporally after the reference frame 3210 may be input to the training model 4000 as training data.

[0200] In addition, it is apparent that the reference frame 3110 and the reference frame 3210, at least one of the previous frames 3120-1, 3120-2 and 3120-n, and at least one of the subsequent frames 3220-1, 3220-2 and 3220-n may be input to the training model 4000 as training data.

[0201] Reference Fig. 9 and Fig.10 , the person in the video may move from right to left when the video is played. As the person in the video moves from right to left, a background that may be located near the right side of the reference frame 3110 may be included in the previous frames 3120-1, 3120-2, and 3120-n. In addition, as the person in the video moves from right to left, a background that may be located near the left side of the reference frame 3210 may be included in the subsequent frames 3220-1, 3220-2, and 3220-n. Therefore, at least one of the previous frames 3120-1, 3120-2, and 3120-n and the subsequent frames 3220-1, 3220-2, and 3220-n may be input as training data to the training model 4000 to generate an extended video generation module.

[0202] According to an embodiment, the previous frames 3120-1, 3210-2, and 3120-n and the subsequent frames 3220-1, 3210-2, and 3220-n may be frames played before and after a preset time period from the reference frames 3110 and 3210, respectively. In addition, the previous frames 3120-1, 3210-2, and 3120-n and the subsequent frames 3220-1, 3210-2, and 3220-n may be frames having similarities with the reference frames 3110 and 3210, respectively. In this case, the similarity may be based on the amount of data change between the reference frame 3110 and the previous frames 3120-1, 3120-2, and 3120-n or the amount of data change between the reference frame 3210 and the subsequent frames 3220-1, 3220-2, and 3220-n.

[0203] The training model 4000 may recognize features (such as patterns and colors) of the reference frames 3110 and 3210 based on the reference frames 3110 and 3210, the previous frames 3120-1, 3120-2, and 3120-n, and the subsequent frames 3220-1, 3200-2, and 3220-n. The training model 4000 may generate an extended video generation module by using the recognized features to generate an extended video to be displayed in a black border at a portion near both sides of the reference frame.

[0204] Although the above only describes the case where black borders are displayed at the parts near both sides of the video, it is obvious that the present embodiment is also applicable to the case where black borders are displayed at the parts near both sides of the video. Figure 2 The black borders of various shapes are shown in .

[0205] Fig.11 A diagram illustrating an example of inputting a resized image 3320 to a training model according to an embodiment is shown.

[0206] Reference Fig.11 , a reference frame 3310 included in a video and a resized image 3320 obtained by resizing the reference frame 3310 may be input to the training model 4000 as training data.

[0207] Reference Fig.11 The resized image 3320 is an image obtained by resizing the reference frame 3310 so that the aspect ratio of the reference frame 3310 corresponds to the aspect ratio of the first region. The resized image 3320 may have an increased number of pixels 3321 compared to the reference frame 3310.

[0208] According to an embodiment, the training model 4000 may recognize features such as patterns and colors of the reference frame 3310. The training model 4000 may generate an extended video generation module using the recognized features to generate an extended video to be displayed in black bars at portions near both sides of the reference frame 3310.

[0209] The AI ​​neural network can learn the input reference frame 3310 and the resized image 3320, thereby generating an extended video generation module to generate an extended video that will not be inconsistent with the reference frame 3310.

[0210] Fig.12 A diagram illustrating an example of inputting a cropped image to a training model according to an embodiment is shown.

[0211] Reference Fig.12 , the reference frame 3410 included in the video and the cropped image obtained by cropping the reference frame 3410 may be input as training data to the training model 4000. In this case, the cropped image 3420 may be an image obtained by cropping preset pixels 3411a and 3411b on both sides of the reference frame 3410.

[0212] According to an embodiment, the aspect ratio of the video may be 2.35: 1, and the aspect ratio of the first area may be 18: 9. The cropped image 3420 may be an image obtained by cropping preset pixels 3411a and 3411b at both sides of the reference frame 3410 to correspond to the aspect ratio of the screen.

[0213] According to an embodiment, the aspect ratio of the video may be 16:9, and the aspect ratio of the first region may be 18:9. The cropped image may be an image obtained by cropping preset pixels at both sides of the reference frame to correspond to the aspect ratio of the first region.

[0214] According to an embodiment, the training model 4000 may recognize features (such as patterns and colors) of the reference frame 3410 based on the reference frame 3410 and the cropped image 3420. The training model 4000 may generate an extended video generation module by using the recognized features to generate an extended video to be displayed in black borders at portions near both sides of the reference frame 3410.

[0215] The AI ​​neural network can learn the input reference frame 3410 and the cropped image 3420, thereby generating an extended video generation module to generate an extended video that will not be inconsistent with the reference frame 3410.

[0216] Fig.13 A diagram illustrating an example of applying weights to frames included in a video according to an embodiment is shown.

[0217] Reference Fig.13 , weights may be applied to preset pixels 3511a and 3511b of a reference frame 3510 in a video. The reference frame 3510 generated by applying the weights to the preset pixels 3511a and 3511b may be input to the training model 4000 as training data.

[0218] According to an embodiment, the preset pixels 3511a and 3511b to which the weights are applied may be pixels located near the black border area. For example, the aspect ratio of the video may be 16:9, and the aspect ratio of the first area may be 18:9. In this case, the black border area may be a portion near both sides of the video. Therefore, weights may be applied to the preset pixels 3511a and 3511b located at both sides of the reference frame 3510.

[0219] According to an embodiment, the training model 4000 may learn the reference frame 3510 by applying weights to the preset pixels 3511a and 3511b. The training model 4000 may recognize features (such as patterns and colors) of the preset pixels 3511a and 3511b to which the weights are applied. The training model 4000 may generate an extended video generation module by using the recognized features to generate an extended video to be displayed in a black border at a portion near both sides of the reference frame 3510.

[0220] Fig.14 is a flowchart illustrating a method of inputting found images as training data into a training model according to an embodiment. Fig.15 A diagram illustrating an example of inputting a found image as training data into a training model according to an embodiment is shown.

[0221] Reference Fig.14 and Fig.15 , the device 1000 or the server 2000 retrieves or searches for an image related to the reference frame, and inputs the searched image into the AI ​​neural network as training data. The device 1000 or the server 2000 may input the searched image into the training model 4000. For example, the device 1000 or the server 2000 may control a learner included in the device 1000 or the server 2000 to update the training model 4000 by inputting the searched image into the training model 4000.

[0222] In operation S1410, the device 1000 or the server 2000 may detect at least one of a pattern and a color constituting a reference frame. Fig.15 , the device 1000 or the server 2000 may detect at least one of a pattern (eg, vertical lines of a background, horizontal lines of a railing, etc.) and a color (eg, brown of a background, gray of a railing, etc.) constituting a reference frame.

[0223] Referring to operation S1430, the device 1000 or the server 2000 may search for an image related to at least one of the detected pattern and color. In this case, the device 1000 or the server 2000 may retrieve an image stored in an internal storage device (e.g., a memory or a database). In addition, the device 1000 or the server 2000 may search for an image through the Internet.

[0224] Reference Fig.15 , the device 1000 or the server 2000 may retrieve or search for an image 3621 related to a horizontal line of a railing as a pattern detected from the reference frame 3610. In addition, the device 1000 or the server 2000 may retrieve or search for an image 3622 related to a vertical line as a background of the pattern detected from the reference frame 3610.

[0225] Referring to operation S1450, the device 1000 or the server 2000 may input the searched image as training data to the training model 4000. Fig.15 , the device 1000 or the server 2000 may input the searched images 3621 and 3622 together with the reference frame 3610 as training data to the training model 4000 .

[0226] According to an embodiment, the device 1000 or the server 2000 may generate a resized image by resizing the searched image, and input the generated resized image to the training model 4000 as training data.

[0227] According to an embodiment, the training model 4000 may learn the reference frame 3610 and the searched images 3621 and 3622. The training model 4000 may recognize features (such as a pattern or color) of the reference frame 3610. In addition, the training model 4000 may recognize features (such as a pattern and a color) of the searched images 3621 and 3622. In addition, the training model 4000 may recognize features (such as a pattern or a color) of a resized image generated by resizing the searched images 3621 and 3622. The training model 4000 may generate an extended video generation module by using the recognized features to generate an extended video to be displayed in a black border at a portion located near both sides of the reference frame 3610.

[0228] Fig.16 and Fig.17 is a block diagram of an apparatus 1000 according to an embodiment.

[0229] like Fig.16 As shown in , the device 1000 according to the embodiment may include a user input interface 1100, an output interface 1200, a processor 1300 and a communication interface 1500. However, Fig.16 Not all of the elements shown are necessary elements of the device 1000. The device 1000 may include Fig.16 Optionally, the device 1000 may include more than the elements shown in FIG. Fig.16 Fewer components are shown.

[0230] For example, Fig.17As shown in , the device 1000 according to an embodiment may further include a sensor 1400 , an A / V input interface 1600 , and a memory 1700 in addition to the user input interface 1100 , the output interface 1200 , the processor 1300 , and the communication interface 1500 .

[0231] The user input interface 1100 refers to a device for inputting data that allows the user to control the device 1000. For example, the user input interface 1100 may be, but is not limited to, a keyboard, a dome switch, a touch pad (e.g., a touch capacitive touch pad, a pressure resistive touch pad, an infrared beam sensing touch pad, a surface acoustic wave touch pad, an integrated strain gauge touch pad, and a piezoelectric touch pad), a scroll wheel, or a scroll wheel switch.

[0232] The user input interface 1100 may receive a user input selecting a first video from a video list.

[0233] The output interface 1200 may output an audio signal, a video signal, or a vibration signal. The output interface 1200 may include a display 1210 , a speaker 1220 , and a vibration motor 1230 .

[0234] The display 1210 includes a screen for displaying information processed by the device 1000. In addition, the screen can display a video. For example, at least a portion of the screen can display a second video including at least a portion of a first video and an extended video corresponding to the first video. The display can be, for example, a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display (e.g., an active matrix organic light emitting diode (AMOLED) and a passive matrix OLED (PMOLED)), a microelectromechanical system (MEMS) display, or a flexible display that changes the aspect ratio of a screen that outputs information by folding or unfolding.

[0235] The speaker 1220 outputs audio data received from the communication interface 1500 or stored in the memory 1700. In addition, the speaker 1220 may output audio signals related to functions processed in the device 1000 (eg, a call signal reception sound, a message reception sound, and a notification sound).

[0236] The processor 1300 generally controls the overall operation of the device 1000. For example, the processor 1300 may control the overall operation of the user input interface 1100, the output interface 1200, the sensor 1400, the communication interface 1500, the A / V input interface 1600, etc. by executing the program stored in the memory 1700. In addition, the processor 1300 may execute the above-referenced program by executing the program stored in the memory 1700. Figures 1 to 14 The functionality of the apparatus 1000 is described.

[0237] Specifically, the processor 1300 may control the user input interface 1100 to receive input of text, images, and videos from the user. The processor 1300 may control the microphone 1620 to receive the user's voice input. The processor 1300 may execute an application configured to perform the operation of the device 1000 based on the user's input, and control the application to receive the user's input. For example, the processor 1300 may control the microphone 1620 to receive the user's voice input by executing a voice assistant application and controlling the executed application.

[0238] The processor 1300 may control the output interface 1200 and the memory 1700 of the device 1000 to display the first video.

[0239] When the aspect ratio of the first video is different from the aspect ratio of the first region, the processor 1300 may generate an extended video to be displayed in a black border displayed in the first region.

[0240] The processor 1300 may train the AI ​​neural network by inputting the training data into the AI ​​neural network. For example, the processor 1300 may generate an extended video generation module by inputting the training data into the training model 4000 stored in the memory 1700 or the server 2000 to generate an extended video. In this case, it should be understood that the processor 1300 includes a graphics processing unit (GPU) or a neural network processor (NPU).

[0241] In addition, the processor 1300 may generate an extended video corresponding to a reference frame included in the video by using the generated extended video generation module, Figures 19 to 22 This is described in more detail. In this case, the reference frame is a frame selected as needed to generate the extended video. Figure 4 The reference frame used to generate the extended video has been described, and thus a repeated description will be omitted.

[0242] The processor 1300 may extract at least one frame included in the first video.

[0243] The processor 1300 may control the output interface 1200 and the memory 1700 of the device 1000 to display the second video including at least a portion of the first video and an extended video corresponding to the first video.

[0244] In addition, the processor 1300 can learn a standard for generating an extended video corresponding to a reference frame included in a video by using a training model 4000 and a learner stored in the memory 1700 or the server 2000, and can provide a user with an extended video generation module for generating an extended video matching the user's intention based on the learning result.

[0245] The sensor 1400 may detect the state of the device 1000 and the state of the environment of the device 1000, and transmit the obtained information to the processor 1300. The sensor 1400 may be used to generate a portion of information about product specifications of the device 1000, information about the state of the device 1000, information about the surrounding environment of the device 1000, information about the state of the user, and information about the history of the user using the device 1000.

[0246] The sensor 1400 may include at least one sensor selected from the following sensors: a magnetic sensor 1410, an acceleration sensor 1420, a temperature / humidity sensor 1430, an infrared sensor 1440, a gyro sensor 1450, a position sensor (e.g., GPS) 1460, an atmospheric pressure sensor 1470, a proximity sensor 1480, and an RGB sensor (illuminance sensor) 1490, but is not limited thereto. Since a person of ordinary skill in the art can intuitively infer the functions of each sensor, a detailed description thereof will be omitted.

[0247] The communication interface 1500 may include at least one element that enables communication between the device 1000 and another external device (not shown) and the server 2000. The external device (not shown) may be a computing device or a sensing device like the device 1000, but is not limited thereto. For example, the communication interface 1500 may include a short-range wireless communication interface 151, a mobile communication interface 1520, and a broadcast receiver 1530.

[0248] The short-range wireless communication interface 1510 may include a Bluetooth communication interface, a Bluetooth low energy (BLE) communication interface, a near field communication interface, a wireless local area network (WLAN) or Wi-Fi communication interface, a Zigbee communication interface, an infrared data association (IrDA) communication interface, a Wi-Fi Direct (WFD) communication interface, an ultra-wideband (UWB) communication interface, and an Ant+ communication interface, but is not limited thereto.

[0249] The mobile communication interface 1520 may transmit and receive wireless signals to and from at least one of a base station, an external terminal, and a server on a mobile communication network. The wireless signals may include various types of data for transceiving audio call signals, video communication call signals, or text / multimedia messages.

[0250] The broadcast receiver 1530 may receive a broadcast signal and / or broadcast-related information from the outside via a broadcast channel. The broadcast channel may include a satellite channel and a terrestrial channel. According to an embodiment of the present disclosure, the device 1000 may not include the broadcast receiver 1530.

[0251] In addition, the communication interface 1500 may transmit the video to the server 2000 .

[0252] According to an embodiment, the communication interface 1500 may transmit at least a portion of the first video stored in the memory 1700 to the server 2000. For example, the communication interface 1500 may transmit at least one frame extracted from the first video to the server 2000. In another example, the communication interface 1500 may transmit a video consisting of a plurality of frames extracted from the first video to the server 2000. In another example, the communication interface 1500 may transmit a video obtained by cropping a portion of the first video to the server 2000.

[0253] According to an embodiment, the communication interface 1500 may transmit information about the first video posted on the Internet to the server 2000. For example, the communication interface 1500 may transmit an identifier (e.g., URL) of the first video to the server 2000. In another example, the communication interface 1500 may transmit at least one frame extracted from the first video to the server 2000. In another example, the communication interface 1500 may transmit a video consisting of a plurality of frames extracted from the first video to the server 2000.

[0254] According to an embodiment, the communication interface 1500 may receive an extended video generating module generated by a server.

[0255] According to an embodiment, the communication interface 1500 may receive a video list from a server.

[0256] According to an embodiment, the communication interface 1500 may receive a video from a server.

[0257] The audio / video (A / V) input interface 1600 is used to input an audio signal or a video signal, and may include a camera 1610, a microphone 1620, etc. The camera 1610 may obtain an image frame of a still image or a moving image via an image sensor in a video call mode or an image capture mode. The image captured by the image sensor may be processed by the processor 1300 or a separate image processor (not shown). The image captured by the camera 1610 may be used as context information of the user.

[0258] Microphone 1620 can receive an audio signal from the outside and process the received signal into electrical sound data. For example, microphone 1620 can receive a sound signal from an external device or a user. Microphone 1620 can receive a user's voice input. Microphone 1620 can use various algorithms to remove noise generated during the process of receiving the input of the external sound signal.

[0259] The memory 1700 may store a program for processing and controlling the processor 1300, and store data input to or output from the device 1000. In addition, the memory 1700 may store videos and results of searching for videos stored in the memory 1700. The memory 1700 may store information related to the videos stored in the device 1000 in the device 1000. For example, the memory 1700 may store a path where the video is stored, a playback time of the video, an aspect ratio of the video, whether an extended video has been generated, and the like.

[0260] The memory 1700 may store at least one frame extracted from the video.

[0261] The memory 1700 may store an AI neural network. For example, the memory 1700 may store a learner, a training model, and an extended video generation module generated by the training model. In addition, the memory 1700 may store an extended video generated by inputting a frame included in the video to the extended video generation module.

[0262] The memory 1700 may include at least one type of storage medium selected from the following items: flash memory type, hard disk type, micro multimedia card type, card type memory (e.g., SD or XD memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk or optical disk.

[0263] Programs stored in the memory 1700 may be classified into a plurality of modules according to their functions, for example, a user interface (UI) module 1710 , a touch screen module 1720 , and a notification module 1730 .

[0264] The UI module 1710 may provide a dedicated UI, GUI, etc. for interacting with the device 1000 according to an application. The touch screen module 1720 may detect a touch gesture of a user on the touch screen and send information about the touch gesture to the processor 1300. The touch screen module 1720 according to an embodiment of the present disclosure may recognize and analyze a touch code. The touch screen module 1720 may be configured as a separate hardware component including a controller.

[0265] The notification module 1730 may generate a signal for notifying the occurrence of an event of the device 1000. Examples of events occurring in the device 1000 may include reception of a call signal, reception of a message, input of a key signal, and notification of a schedule. The notification module 1730 may output the notification signal in the form of a video signal through the display 1210, in the form of an audio signal through the speaker 1220, and in the form of a vibration signal through the vibration motor 1230.

[0266] Fig.18 is a block diagram of a server 2000 according to an embodiment.

[0267] Reference Fig.18 , the server 2000 according to the embodiment may include a communication interface 2500 , a DB 2700 , and a processor 2300 .

[0268] The communication interface 2500 may include at least one component that enables communication with the device 1000 .

[0269] The communication interface 2500 may receive a video from the device 1000 or transmit a video to the device 1000 .

[0270] DB 2700 may store an AI neural network or training data to be input to the AI ​​neural network. For example, DB 2700 may store a learner, a training model, and training data to be input to the training model. DB 2700 may store an extended video generation module generated by the training model.

[0271] The DB 2700 may store at least one video. The DB 2700 may store an extended video corresponding to the at least one video generated by the extended video generating module. The DB 2700 may store a video in which the at least one video is combined with the extended video corresponding to the at least one video.

[0272] The processor 2300 generally controls the overall operation of the server 2000. For example, the processor 2300 may control the overall operation of the DB 2700 and the communication interface 2500 by executing a program stored in the DB 2700 of the server 2000. The processor 2300 may execute a program stored in the DB 2700 of the server 2000. Figures 1 to 15 1. A portion of the operation of apparatus 1000 is shown in FIG.

[0273] Processor 2300 may perform at least one of the following functions: a function of extracting frames from a video, a function of generating training data based on the extracted frames, a function of generating an extended video generation module by inputting the training data into a training model, and a function of generating an extended video by inputting a video into the extended video generation module.

[0274] Processor 2300 may control at least one of the following data: data required to extract frames from a video, data required to generate training data based on the extracted frames, data required to generate an extended video generation module by inputting the training data into a training model, and data required to generate an extended video by inputting a video into the extended video generation module.

[0275] Fig.19 is a block diagram of a processor 1300 according to an embodiment.

[0276] Reference Fig.19 , the processor 1300 according to an embodiment may include a data learner 1310 and a data identifier 1320 .

[0277] The data learner 1310 may learn a standard for generating an extended video corresponding to a reference frame included in a video. The reference frame is a frame selected as needed to generate an extended video. Figure 5 A reference frame for generating an extended video is described, and a repeated description will be omitted.

[0278] The data learner 1310 may learn a criterion for selecting data for generating an extended video corresponding to a reference frame included in a video. The data learner 1310 may learn a criterion for generating an extended video corresponding to a reference frame included in a video by obtaining data for learning and applying the obtained data to a data recognition model to be described below.

[0279] For example, the data learner 1310 may learn the following operations as criteria for obtaining training data: obtaining a previous frame and a subsequent frame that can be obtained before and after a preset time period from a reference frame as training data, obtaining a reference frame and a resized image obtained by resizing the reference frame, obtaining a reference frame and a cropped image obtained by cropping the reference frame as training data, and obtaining an image related to the reference frame as training data. Figures 9 to 15 The data used for learning and the method of obtaining the same have been described, so repeated descriptions will be omitted.

[0280] For example, the data learner 1310 may learn the following operations as a criterion for learning training data: applying weights to pixels of training data located near a black border region and applying weights to pixels of training data located near the same pixel as a pixel of a reference frame.

[0281] The data identifier 1320 may generate an extended video corresponding to a reference frame included in the video based on the data. The data identifier 1320 may generate an extended video corresponding to a reference frame included in the video from preset data by using an updated extended video generation module. The data identifier 1320 may obtain preset data according to a standard preset by learning, and use an extended video generation module using the obtained data as an input value. In addition, the data identifier 1320 may generate an extended video corresponding to a reference frame included in the video based on the preset data. In addition, a result value output from the extended video generation module using the obtained data as an input value may be used to refine the extended video generation module.

[0282] Any one or both of the data learner 1310 and the data identifier 1320 may be manufactured as at least one hardware chip and built into the device. For example, at least one of the data learner 1310 and the data identifier 1320 may be manufactured as a dedicated hardware chip for artificial intelligence (AI), or may be manufactured as a part of an existing general-purpose processor (e.g., a central processing unit (CPU) or an application processor) or a graphics processor (e.g., a graphics processing unit (GPU)).

[0283] In this case, the data learner 1310 and the data identifier 1320 may be constructed in one device or may be separately constructed in different devices. For example, one of the data learner 1310 and the data identifier 1320 may be included in the device 1000, and the other may be included in the server 2000. In addition, the data learner 1310 may provide the model information constructed by the data learner 1310 to the data identifier 1320 by wire or wirelessly, or the data identifier 1320 may provide the data input to the data identifier 1320 as additional training data to the data learner 1310 by wire or wirelessly.

[0284] In addition, any one or both of the data learner 1310 and the data identifier 1320 may be implemented as a software module. When any one or both of the data learner 1310 and the data identifier 1320 are implemented as a software module (or a program module including instructions), the software module may be stored in a non-transitory computer-readable recording medium. In addition, in this case, at least one software module may be provided by an operating system (OS) or a preset application. Alternatively, a portion of the at least one software module may be provided by the OS, and the remaining portion may be provided by a preset application.

[0285] Fig. 20 is a block diagram of a data learner 1310 according to an embodiment.

[0286] Reference Fig. 20 , the data learner 1310 according to an embodiment may include a data obtainer 1310 - 1 , a preprocessor 1310 - 2 , a training data selector 1310 - 3 , a model learner 1310 - 4 , and a model evaluator 1310 - 5 .

[0287] The data obtainer 1310-1 may obtain data required to generate an extended video corresponding to a video. The data obtainer 1310-1 may obtain, for example, a reference frame included in the video, at least one previous frame played before the reference frame, at least one subsequent frame played after the reference frame, a resized frame obtained by resizing the reference frame, a cropped image generated by cropping a portion of the reference frame, and an image related to at least one of a pattern and a color constituting the reference frame.

[0288] The preprocessor 1310-2 may preprocess the obtained data so that the data is used to generate an extended video corresponding to the video. The preprocessor 1310-2 may process the obtained data into a preset format so that the model learner 1310-4 described below may use the obtained data for learning to generate an extended video corresponding to the video.

[0289] The training data selector 1310-3 may select data required for learning from a plurality of pieces of preprocessed data. The selected data may be provided to the model learner 1310-4. The training data selector 1310-3 may select data required for learning from a plurality of pieces of preprocessed data according to a preset standard for generating an extended video corresponding to a video. In addition, the training data selector 1310-3 may select data according to a standard preset by learning of the model learner 1310-4 to be described below.

[0290] The model learner 1310-4 may learn a standard on how to generate an extended video corresponding to a video based on training data. In addition, the model learner 1310-4 may learn a standard on how to select training data to be used to generate an extended video corresponding to a video.

[0291] For example, criteria for how to select training data to be learned by a model learning model may include: criteria for selecting a reference frame to be used as training data, criteria for selecting at least one frame from multiple frames that can be played before or after a preset time from a reference frame as training data, criteria for selecting a reference frame and a resized image of the reference frame as training data, criteria for selecting a reference frame and a cropped image of the reference frame as training data, and criteria for selecting an image related to a category of the video as training data.

[0292] In addition, the model learner 1310-4 can update the training model for generating an extended video corresponding to the video by using the training data. In this case, the training model can be a pre-established model. For example, the training model can be pre-established by receiving input of basic training data (e.g., sample data).

[0293] The training model may be pre-established in consideration of the field to which the recognition model is applied, the purpose of learning, or the computing performance of the device 1000. The training model may be a model based on, for example, a neural network. For example, models such as a deep neural network (DNN), a recurrent neural network (RNN), a bidirectional recurrent deep neural network (BRDNN), and a generative adversarial network (GAN) may be used as the training model, without limitation thereto.

[0294] According to various embodiments, when there are a plurality of pre-established training models, the model learner 1310-4 may determine a training model having a high correlation between the input training data and the basic training data as a training model to be updated. In this case, the basic training data may be classified in advance according to the type of data, and the training model may be pre-established according to the type of data. For example, the basic training data may be pre-classified according to various criteria such as: the region where the training data is generated, the time when the training data is generated, the size of the training data, the type of the training data, the generator of the training data, and the type of the object in the training data.

[0295] In addition, the model learner 1310-4 can update the training model by using a learning algorithm including, for example, error back propagation or gradient descent.

[0296] In addition, the model learner 1310-4 may update the training model through supervised learning, for example, by using the training data as an input value. In addition, the model learner 1310-4 may update the training model through unsupervised learning to find a standard for determining a standard for generating an extended video corresponding to a video by selecting a reference frame of the video and learning the type of data required to generate an extended video corresponding to the selected reference frame by itself without supervision. In addition, the model learner 1310-4 may update the training model through reinforcement learning using feedback on whether the result of generating an extended video corresponding to the video by learning is correct.

[0297] In addition, when the training model is updated, the model learner 1310-4 may store an extended video generation module generated using the updated training model. In this case, the model learner 1310-4 may store the extended video generation module in a memory of the device including the data identifier 1320. Alternatively, the model learner 1310-4 may store the extended video generation module in a DB of a server connected to the device via a wired or wireless network.

[0298] In this case, the memory storing the extended video generation module may also store, for example, commands or data associated with at least one other element of the device. In addition, the memory may store software and / or programs. The program may include, for example, a kernel, middleware, an application programming interface (API), and / or an application program (or "application").

[0299] When the model evaluator 1310-5 inputs the evaluation data into the training model, and the recognition result output according to the evaluation data does not meet the preset standard, the model evaluator 1310-5 can control the model learner 1310-4 to learn again. In this case, the evaluation data can be preset data for evaluating the training model.

[0300] For example, when the number or ratio of inaccurate recognition results in the recognition results of the updated training model output according to the evaluation data exceeds a preset threshold, the model evaluator 1310-5 may evaluate that the preset standard is not met. For example, when the preset threshold is defined as 2%, and the updated training model outputs more than 20 inaccurate recognition results in a total of 1000 evaluation data, the model evaluator 1310-5 may evaluate that the updated training model is not suitable.

[0301] In addition, when there are multiple updated training models, the model evaluator 1310-5 may evaluate whether each of the updated training models meets the preset criteria, and may determine the training model that meets the preset criteria as the final training model. In this case, when there are multiple training models that meet the preset criteria, the model evaluator 1310-5 may determine one model or a preset number of models among the preset models as the final training model in descending order of evaluation scores.

[0302] In addition, at least one of the data acquirer 1310-1, the preprocessor 1310-2, the training data selector 1310-3, the model learner 1310-4, and the model evaluator 1310-5 of the data learner 1310 may be manufactured as at least one hardware chip and built into the device. For example, at least one of the data acquirer 1310-1, the preprocessor 1310-2, the training data selector 1310-3, the model learner 1310-4, and the model evaluator 1310-5 may be manufactured as a dedicated hardware chip for artificial intelligence (AI) or may be manufactured as part of an existing general-purpose processor (e.g., a central processing unit (CPU) or an application processor) or a graphics processor (e.g., a GPU) and built into the device.

[0303] In addition, the data obtainer 1310-1, the preprocessor 1310-2, the training data selector 1310-3, the model learner 1310-4, and the model evaluator 1310-5 may be constructed in one device 1000, or may be constructed in separate devices, respectively. For example, some of the data obtainer 1310-1, the preprocessor 1310-2, the training data selector 1310-3, the model learner 1310-4, and the model evaluator 1310-5 may be included in the device, and the others may be included in the server.

[0304] In addition, at least one of the data obtainer 1310-1, the preprocessor 1310-2, the training data selector 1310-3, the model learner 1310-4, and the model evaluator 1310-5 may be implemented as a software module. When at least one of the data obtainer 1310-1, the preprocessor 1310-2, the training data selector 1310-3, the model learner 1310-4, and the model evaluator 1310-5 is implemented as a software module (or a program module including instructions), the software module may be stored in a non-transitory computer-readable recording medium. In addition, in this case, at least one software module may be provided by an operating system (OS) or a preset application. Optionally, a portion of the at least one software module may be provided by the OS, and the rest may be provided by a preset application.

[0305] The processor 1300 may use various training models and effectively learn a standard for generating an extended video corresponding to a video through various methods using the training models.

[0306] Fig.21 is a block diagram of the data identifier 1320 according to an embodiment.

[0307] Reference Fig.21 , the data identifier 1320 according to an embodiment may include a data obtainer 1320 - 1 , a preprocessor 1320 - 2 , a recognition data selector 1320 - 3 , a recognition result provider 1320 - 4 , and a model refiner 1320 - 5 .

[0308] The data obtainer 1320-1 may obtain data for generating an extended video corresponding to a video. For example, the data obtainer 1320-1 may obtain a reference frame included in the video, at least one previous frame played before the reference frame, at least one subsequent frame played after the reference frame, a resized frame obtained by resizing the reference frame, a cropped image generated by cropping a portion of the reference frame, and an image related to at least one of a pattern and a color constituting the reference frame.

[0309] The preprocessor 1320-2 may preprocess the obtained data so that the data is used to generate an extended video corresponding to the video. The preprocessor 1320-2 may process the obtained data into a preset format so that the recognition result provider 1320-4 described below may use the obtained data for learning to generate an extended video corresponding to the video.

[0310] The recognition data selector 1320-3 may select data required to generate an extended video corresponding to the video from a plurality of pre-processed data. The selected data may be provided to the recognition result provider 1320-4. The recognition data selector 1320-3 may select some or all data from the pre-processed plurality of data according to a preset standard for generating an extended video corresponding to the video. In addition, the recognition data selector 1320-3 may select data according to a standard pre-set by learning of the model learner 1310-4 described above.

[0311] The recognition result provider 1320-4 can generate an extended video corresponding to the video by applying the selected data to the extended video generation module. The recognition result provider 1320-4 can provide a recognition result according to the recognition purpose of the data. The recognition result provider 1320-4 can apply the selected data to the extended video generation module by using the data selected by the recognition data selector 1320-3 as an input value. In addition, the recognition result can be determined by the extended video generation module.

[0312] The model refiner 1320-5 may refine the extended video generation module based on an evaluation of the recognition result provided by the recognition result provider 1320-4. For example, the model refiner 1320-5 may provide the recognition result provided by the recognition result provider 1320-4 to the model learner 1310-4, so that the model learner 1310-4 refines the training model and refines the extended video generation module by using the refined training model.

[0313] In addition, at least one of the data obtainer 1320-1, the preprocessor 1320-2, the recognition data selector 1320-3, the recognition result provider 1320-4, and the model refiner 1320-5 may be manufactured as at least one hardware chip and built into the device. For example, at least one of the data obtainer 1320-1, the preprocessor 1320-2, the recognition data selector 1320-3, the recognition result provider 1320-4, and the model refiner 1320-5 may be manufactured as a dedicated hardware chip for artificial intelligence (AI) or may be manufactured as part of an existing general-purpose processor (e.g., a central processing unit (CPU) or an application processor) or a graphics processor (e.g., a GPU) and built into the device.

[0314] In addition, the data obtainer 1320-1, the preprocessor 1320-2, the recognition data selector 1320-3, the recognition result provider 1320-4, and the model refiner 1320-5 may be constructed in one device 1000, or may be constructed in separate devices. For example, some of the data obtainer 1320-1, the preprocessor 1320-2, the recognition data selector 1320-3, the recognition result provider 1320-4, and the model refiner 1320-5 may be included in the device 1000, and the others may be included in the server 2000.

[0315] In addition, at least one of the data obtainer 1320-1, the preprocessor 1320-2, the recognition data selector 1320-3, the recognition result provider 1320-4, and the model refiner 1320-5 may be implemented as a software module. When at least one of the data obtainer 1320-1, the preprocessor 1320-2, the recognition data selector 1320-3, the recognition result provider 1320-4, and the model refiner 1320-5 is implemented as a software module (or a program module including instructions), the software module may be stored in a non-transitory computer-readable recording medium. In addition, in this case, at least one software module may be provided by an operating system (OS) or a preset application. Optionally, a portion of the at least one software module may be provided by the OS, and the remaining portion may be provided by a preset application.

[0316] In addition, the device 1000 may provide the user with an extended video matching the user's intention by using the extended video generating module to which the training result is applied.

[0317] Fig. 22 is a diagram illustrating an example in which the device 1000 and the server 2000 according to an embodiment interact to learn and recognize data according to an embodiment.

[0318] Reference Fig. 22 , the server 2000 may learn a standard for generating an extended video corresponding to the video, and the device 1000 may generate the extended video corresponding to the video based on the learning result of the server 2000.

[0319] In this case, the model learner 2340 of the server 2000 may execute Fig.19The model learner 2340 of the server 2000 may learn a standard for selecting data for generating an extended video corresponding to a video. In addition, the model learner 2340 of the server may learn a standard for generating an extended video corresponding to a video using data. The model learner 2340 may learn a standard for generating an extended video generation module by obtaining data to be used for learning and applying the obtained data to a training model to be described below.

[0320] In addition, the recognition result provider 1320-4 of the device 1000 can generate an extended video corresponding to the video by applying the data selected by the recognition data selector 1320-3 to the extended video generation module generated by the server 2000. For example, the recognition result provider 1320-4 can send the data selected by the recognition data selector 1320-3 to the server 2000, and request the server 2000 to generate an extended video corresponding to the video by applying the data selected by the recognition data selector 1320-3 to the extended video generation module. In addition, the recognition result provider 1320-4 can receive information about a method of generating an extended video corresponding to the video obtained by the server 2000 from the server 2000.

[0321] Alternatively, the recognition result provider 1320-4 of the device 1000 may receive an extended video generation module generated by the server 2000 from the server 2000, and generate an extended video corresponding to the video by using the received extended video generation module. In this case, the recognition result provider 1320-4 of the device 1000 may generate an extended video corresponding to the video by applying the data selected by the recognition data selector 1320-3 to the extended video generation module received from the server 2000.

[0322] In addition, the device 1000 and the server 2000 can effectively divide and execute the operation for updating the training model, the operation for generating the extended video generation module, and the operation for recognizing the data. Therefore, data processing can be effectively performed and the privacy of the user can be effectively protected so as to provide a service that meets the user's intention.

[0323] Fig.23 is a schematic block diagram of an apparatus according to an embodiment.

[0324] Reference Fig.23 , the device 1000 may include a processor 2360, a memory 2362, a graphics processing unit (GPU) (or a neural network processing unit (NPU)) 2364, and a display 2366. The processor 2360 may include Fig.16 The processor 1300. The display 2366 may include Fig.16 Display 1210.

[0325] According to an embodiment, the device 1000 may further include a decoder 2360a and at least one buffer memory 2360b. For example, the decoder 2360a and the buffer memory 2360b may be included in the processor 2360.

[0326] According to an embodiment, the memory 2362 may store the first video and / or the extended video generation module. The first video may be, for example, a video decoded by the decoder 2360 a included in the processor 2360 and stored in the memory 2362 .

[0327] According to an embodiment, the processor 2360 may obtain a first video stored in the memory 2362 and send the first video to the buffer memory 2360b. The processor 2360 may generate a copy of the first video stored in the buffer memory 2360b and send the copy to the GPU 2364. According to an embodiment, the GPU (or NPU) 2364 may be a processor optimized for artificial intelligence. The GPU (or NPU) 2364 may be a processor that is more integrated, has less latency, has higher performance, or has more resources than a general-purpose processor. Optionally, the GPU (or NPU) 2364 may be a processor optimized for matrix calculations of artificial intelligence.

[0328] According to an embodiment, GPU 2364 may generate an extended video using the first video based on an instruction of processor 2360. Processor 2360 may identify an area to be black-bordered when displaying the first video on display 2366, and control GPU 2364 to generate an extended video to be displayed in the black-bordered area.

[0329] For example, when the aspect ratio of the first video is 16:9 and the aspect ratio of the first area of ​​the display 2366 playing the video is 18:9, the processor 2360 of the device 1000 may identify the black border area at the portion near the left and right sides of the first video. The processor 2360 may obtain the extended video generation module stored in the memory 2362, store the extended video generation module in the buffer memory 2360b, and send a copy of the extended video generation module to the GPU 2364. The GPU 2364 may generate an extended video corresponding to the size of the black border area by using the extended video generation module and the first video.

[0330] The processor 2360 may obtain an extended video corresponding to the generated edge area and store the extended video in the buffer memory 2360b. The processor 2360 may generate a second video by using the extended video stored in the buffer memory 2360b and the first video. The second video may refer to, for example, a video including at least a portion of the first video and the extended video.

[0331] According to various embodiments, the GPU 2364 may generate a second video by using the first video. For example, the GPU 2364 may generate an extended video based on the size of the identified black border area, and generate the second video by using the generated extended video and at least a portion of the first video.

[0332] According to an embodiment, the processor 2360 may post-process the boundary area between the first video and the extended video in the generated second video. The post-processing may be, for example, processing of the boundary area that is not easily distinguished by the user. For example, the post-processing may be performed by smoothing or coloring the boundary area between the first video and the extended video.

[0333] According to various embodiments, the processor 2360 may perform post-processing by applying a deblocking filter to a boundary area between the first video and the extended video in the second video.

[0334] The apparatus 1000 may obtain a second video, wherein the second video has a reduced pixel value difference between pixels located at a boundary area between the first video and the extended video.

[0335] The processor 2360 may transmit the second video stored in the buffer memory 2360 b to the display 2366 , and control the display 2366 to output the second video.

[0336] According to various embodiments, when a user downloads a first video from a server and stores the first video, the device 1000 may perform the above-mentioned process of generating an extended video before the user watches the first video. However, the embodiment is not limited thereto. The device 1000 may perform the above-mentioned process of generating an extended video when the first video is downloaded from the server in a streaming form.

[0337] Fig.24 is a diagram for describing a process of generating an extended video generating module performed by a device according to an embodiment.

[0338] According to an embodiment, the device 1000 may input the training model 2410 stored in the memory 2362 or the GPU 2364 and the training data 2420 corresponding to the first category stored in the memory 2362 to the learner 2430. The learner 2430 may be, for example, an algorithm configured to update the training model 2410 by inputting the training data 2420 to the training model 2410 to generate the extended video generation module 2440.

[0339] The training model 2410 may be a training model updated to generate an extended video corresponding to an edge region of a video. When the device 1000 is manufactured, the training model 2410 may be built in the device 1000. Alternatively, the training model 2410 may be received from the server 2000.

[0340] The first category may be, for example, a SF movie, a documentary about nature, a live performance, a 2D animation, a 3D animation, an augmented reality (AR) video, or a holographic video. That is, the training data corresponding to the first category may be an image classified into at least one of the categories (e.g., a still image and a video). For example, the device 1000 may identify a SF movie as the first category, a documentary about nature as the second category, or a live performance as the third category.

[0341] According to an embodiment, the learner 2430 may perform training by applying the training data 2420 corresponding to the input first category to the training model 2410. In this case, the learner 2430 may update the training model 2410 by using the training data corresponding to the plurality of first categories.

[0342] According to an embodiment, when the training model 2410 is updated during a preset time or when the training data of a preset amount is used to complete, the learner 2430 may generate a training model updated using the training data corresponding to the first category (e.g., an extended video generation module 2440 corresponding to the first category). The device 1000 may store the extended video generation module 2440 corresponding to the first category in the memory 2362. However, the embodiment is not limited thereto. The device 1000 may also store the extended video generation module corresponding to the first category in the memory 2362. Figure 1 GPU 2364 described.

[0343] According to various embodiments, the device 1000 may repeat the above process using training data for different categories (e.g., training data for the second category and training data for the third category). The device 1000 may store a plurality of extended video generation modules corresponding to different categories and generated as a result of the repeated process in the GPU 2364. When a first video is input, the device 1000 may generate an extended video of the first video using an extended video generation module corresponding to the category of the first video, and generate a second video.

[0344] According to various embodiments, the device 1000 may guide efficient use of the memory 2362 by providing a user interface for deleting the generated extended video generating module at the time of a user's request.

[0345] Fig.25A diagram for describing a process of generating an extended video performed by a device according to an embodiment according to a method of downloading a first video from a server is shown.

[0346] Reference Fig.25 Referring to figure mark 25-a, the device 1000 can download the first video from the server 2000 at one time.

[0347] According to an embodiment, the device 1000 may decode the downloaded first video by using the decoder 2510 .

[0348] The device 1000 may generate an extended video or a second video by using the decoded first video 2520. The device 1000 may identify a black border region formed when each of a plurality of frames included in the decoded first video 2520 is displayed, and generate an extended video to be displayed in the black border region by using an extended video generation module 2530. The device 1000 may generate a second video 2540 by using the generated extended video and at least a portion of the decoded first video.

[0349] The device 1000 may display the generated second video 2540 .

[0350] Reference Fig.25 Referring to figure 25-b, the device 1000 may download the first video from the server 2000 in a streaming form.

[0351] According to an embodiment, the device 1000 may decode frames downloaded in a streaming form by using the decoder 2510 .

[0352] The device 1000 may generate an extended video of a decoded frame by using a plurality of extended video generating modules according to characteristics of the decoded frame.

[0353] According to an embodiment, the device 1000 may identify whether a decoded frame is an intra frame or an inter frame.

[0354] An intra frame may refer to a frame that is decoded without information from other reference frames, for example, during a decoding process of the frame.

[0355] An inter-frame may refer to, for example, a frame that is decoded using other frames that have already been decoded in a decoding process of the frame.

[0356] When the decoded frame is an intra frame, the device 1000 may control the extended video generation module 2550 to generate an extended video of the frame by using the central area of ​​the frame, wherein the extended video generation module 2550 is configured to generate an extended video of the video by using the input video. The device 1000 may generate an extended frame including the generated extended video and the decoded intra frame.

[0357] When the decoded frame is an inter-frame, the device 1000 may control the extended video generation module 2560 to generate an extended video of the frame by using the central area of ​​the frame and another frame that has been decoded and used to generate the extended video, wherein the extended video generation module 2560 is configured to perform an operation of generating an extended video by using the input video and an operation of generating an extended video by using another reference frame. In this case, the device 1000 may store the previously extended frame in the buffer memory 2562 or the decoder 2510 included in the extended video generation module 2560, and use the frame as a reference frame to extend another inter-frame. The device 1000 may generate an extended frame including the generated extended video and the decoded inter-frame.

[0358] According to an embodiment, the device 1000 may generate and display the second video by using the extended frame 2571 and the extended frame 2572 .

[0359] Fig.26 is a diagram for describing a sequence of generating an extended video of a frame performed by a device according to an embodiment.

[0360] According to an embodiment, the device 1000 may determine the decoding order according to characteristics of the frames. The device 1000 may continue to generate the extended video for the decoded frames.

[0361] According to an embodiment, the device 1000 may decode the first frame 2611 to the ninth frame 2619 and generate an extended video for the decoded frames. The device 1000 may use different decoding methods according to characteristics of the frames.

[0362] According to an embodiment, a type of frame may have a frame characteristic of being decoded without other reference frames in the decoding process. According to various embodiments, when decoding a frame, 50% or more of the frames are decoded without other reference frames. According to an embodiment, a frame having such a characteristic may be referred to as an I frame or an intra frame.

[0363] Alternatively, another type of frame may have a frame characteristic of being decoded using a temporally earlier frame when the frame is decoded. According to an embodiment, a frame having such a characteristic may be referred to as a P frame.

[0364] Alternatively, another type of frame may have a frame characteristic of being decoded using temporally earlier and later frames when the frame is decoded. According to an embodiment, a frame having such a characteristic may be referred to as a B frame.

[0365] Reference Fig.26 , the first frame 2611 may be an I frame, the second frame 2612 to the eighth frame 2618 may be B frames, and the ninth frame 2619 may be a P frame.

[0366] The device 1000 according to an embodiment may decode the first frame 2611 according to the characteristics of the frame, and then decode the ninth frame 2619. The device 1000 may generate an extended video of the first frame 2611 by applying the decoded first frame 2611 to an extended video generation module, wherein the extended video generation module is configured to generate an edge region of the video by using an input video. The device 1000 may generate an extended video of the ninth frame 2619 by applying the decoded ninth frame 2619 to the extended video generation module, wherein the extended video generation module is configured to perform an operation of generating an edge region of the video by using the input video and an operation of generating an edge region of the video by using another frame that may be used as a reference.

[0367] According to an embodiment, the device 1000 may decode the fifth frame 2615 by using the first frame 2611 and the ninth frame 2619. The device 1000 may generate an extended video of the fifth frame 2615 by applying the decoded fifth frame 2615 to an extended video generation module, wherein the extended video generation module is configured to perform an operation of generating an edge area of ​​a video using an input video and an operation of generating an edge area of ​​a video by using other frames (e.g., the first frame 2611 and the ninth frame 2619) that may be used as references.

[0368] According to an embodiment, the device 1000 may decode the frames in the order of the third frame 2613, the second frame 2612, and the fourth frame 2614, and generate the edge region by using the above method. In addition, the device 1000 may decode the frames in the order of the seventh frame 2617, the sixth frame 2616, and the eighth frame 2618, and generate the extended video by using the above method.

[0369] According to an embodiment, the device 1000 may display sequentially from the first frame 2611 to the ninth frame 2619 when the decoding and generation of the extended video are completed. However, the embodiment is not limited thereto. For example, when the extended video up to the fourth frame 2614 is generated, the device 1000 decodes the seventh frame 2617, the sixth frame 2616, and the eighth frame 2618 while sequentially displaying from the first frame 2611 to the fifth frame 2615, and generates the extended video of the seventh frame 2617, the sixth frame 2616, and the eighth frame 2618.

[0370] Fig. 27 is a diagram for describing a case where resizing a video and generating an expanded video are performed by a device according to an embodiment.

[0371] Reference Fig. 27, the device 1000 may enlarge the size of the decoded video by using a resizer or scaler 2730. For example, the device 1000 may modify the resolution of the first video 2710 from 1280×720 (aspect ratio of 16:9) to 2560×1440 (aspect ratio of 16:9) by using the resizer 2730.

[0372] According to an embodiment, the aspect ratio (eg, 16:9) of the first video 2710 to be displayed in the first region may be different from the aspect ratio (eg, 18:9) of the first region to display the first video 2710. In this case, the device 1000 may display the black border 2711a and the black border 2711b.

[0373] According to an embodiment, the device 1000 may generate an extended video including each frame in the first video 2710 by inputting the first video 2710 to the extended video generating module 2740. The device 1000 may generate a second video 2720 including the generated extended video and at least a portion of the first video 2710. The second video 2720 may have a resolution of 2960×1440 and an aspect ratio of 18:9, which is the same as the aspect ratio of the first region.

[0374] The device 1000 may display the generated second video 2720 .

[0375] According to various embodiments, the device 1000 may enable the extended video generating module to generate edge regions of other frames by inputting frames included in the second video 2720 to the extended video generating module 2740 .

[0376] Embodiments of the present disclosure may be implemented as a computer-readable recording medium, for example, a program module including computer-readable instructions to be executed in a computer. Computer-readable recording media may include any available media, volatile and non-volatile media, and removable and non-removable media that can be accessed by a computer. In addition, computer-readable recording media may include computer storage media and communication media. Computer storage media include all volatile and non-volatile media and removable and non-removable media that are technically implemented to store information (including computer-readable instructions, data structures, program modules, or other data). Communication media include computer-readable instructions, data structures, or program modules, and include other information transmission media.

[0377] In addition, throughout the specification, a 'unit' may be a hardware component such as a processor or a circuit and / or a software component executed by hardware such as a processor.

[0378] The above description of the present disclosure is provided for the purpose of illustration, and it will be understood by those skilled in the art that various changes and modifications may be made without changing the technical concept and essential features of the present disclosure. Therefore, it is clear that the above embodiments of the present disclosure are illustrative in all aspects and do not limit the present disclosure. For example, each component described as a single type can be implemented in a distributed manner. Similarly, the components described as distributed can be implemented in a combined manner.

[0379] The scope of the present disclosure is defined by the claims and their equivalents rather than by the detailed description of the illustrative embodiments. It should be understood that all modifications and embodiments conceived from the meaning and scope of the claims and their equivalents are included in the scope of the present disclosure.

Claims

1. A device comprising: monitor; a memory storing one or more instructions; as well as At least one processor configured to display a first video on at least a portion of the display by executing the one or more instructions to: comparing the aspect ratio of the first video with the aspect ratio of the display area of ​​the display, When an aspect ratio of the first video is different from an aspect ratio of the display area, obtaining an extended video corresponding to all frames of the first video by applying a reference frame included in the first video to a trained artificial intelligence (AI) model, wherein the extended video has an aspect ratio corresponding to the aspect ratio of the display area, and displaying the extended video in the display area, and The at least one processor is configured to execute the one or more instructions to obtain the extended video in the following manner: The trained AI model is generated by using the reference frame and at least one related frame associated with the reference frame as training data.

2. The device according to claim 1, wherein: The at least one processor is further configured to execute the one or more instructions to perform the following operations: when an aspect ratio of the first video is different from an aspect ratio of the display area, identifying a black border to be displayed in the display area.

3. The device according to claim 1, wherein: The at least one processor is further configured to execute the one or more instructions to perform the following operations: extracting frames included in the first video, generating training data to be used to generate the trained AI model based on the extracted frames, and The extended video is obtained by inputting the training data into the trained AI model to update the trained AI model.

4. The device according to claim 1, wherein: The at least one processor is further configured to execute the one or more instructions to perform the following operations: generating the trained AI model by using the reference frame and a resized frame obtained by resizing the reference frame as training data.

5. The device according to claim 1, wherein: The at least one processor is further configured to execute the one or more instructions to perform the following operations: extracting the reference frame, at least one previous frame and at least one subsequent frame included in the first video, and The trained AI model is generated by using the reference frame, the at least one previous frame, and the at least one subsequent frame as training data.

6. The device according to claim 1, wherein: The apparatus further includes: a communication interface configured to send the first video to a server, and receive from the server the trained AI model generated by the server using the first video, and Wherein, the at least one processor is further configured to execute the one or more instructions to perform the following operations: obtain the extended video by inputting the reference frame of the first video into the trained AI model received from the server.

7. The device according to claim 1, wherein: The at least one processor is also configured to execute the one or more instructions to perform the following operations: generate frames of the extended video corresponding to all frames of the first video by inputting the reference frame, at least one previous frame, and at least one subsequent frame into the trained AI model to train the trained AI model.

8. The device according to claim 1, wherein: The device also includes a communication interface, and The at least one processor is further configured to execute the one or more instructions to perform the following operations: detecting at least one of a pattern and a color constituting the reference frame included in the first video, searching for an image using the communication interface, wherein the image is associated with the at least one of the detected pattern and color, and The trained AI model is generated by using the reference frame and the searched image as training data.

9. The device according to claim 1, wherein: The at least one processor is further configured to execute the one or more instructions to perform the following operations: identifying an object included in the reference frame of the first video, determining a category of the first video according to the object, and Selecting the trained AI model associated with the category of the first video.

10. The device according to claim 9, wherein: The categories include 2D animation or 3D animation.

11. A method for displaying a video, the method comprising: comparing an aspect ratio of the first video to an aspect ratio of a display area in which the first video will be displayed; When an aspect ratio of the first video is different from an aspect ratio of the display area, obtaining an extended video corresponding to all frames of the first video by applying a reference frame included in the first video to a trained artificial intelligence (AI) model, wherein the extended video has an aspect ratio corresponding to the aspect ratio of the display area; and displaying the extended video in the display area, and The step of obtaining the extended video includes: The trained AI model is generated by using the reference frame and at least one related frame associated with the reference frame as training data.

12. The method according to claim 11, wherein: The step of comparing the aspect ratio of the first video with the aspect ratio of the display area includes identifying a black border to be displayed in the display area when the aspect ratio of the first video is different from the aspect ratio of the display area.

13. The method according to claim 11, wherein: The steps of obtaining the extended video include: extracting frames included in the first video; Generating training data to be used to generate the trained AI model based on the extracted frames; and The extended video is obtained by inputting the training data into the trained AI model to update the trained AI model.

14. The method according to claim 11, wherein: The step of obtaining the extended video includes generating the trained AI model by using the reference frame and a resized frame obtained by resizing the reference frame as training data.

15. The method according to claim 11, wherein: The steps of obtaining the extended video include: extracting the reference frame, at least one previous frame, and at least one subsequent frame included in the first video; and The trained AI model is generated by using the reference frame, the at least one previous frame, and the at least one subsequent frame as training data.

16. The method according to claim 11, wherein: The steps of obtaining the extended video include: sending the first video to a server, and receiving from the server the trained AI model generated by the server using the first video, and The extended video is obtained by inputting the reference frame of the first video into the trained AI model received from the server.

17. The method according to claim 11, wherein: The step of obtaining the extended video includes generating frames of the extended video corresponding to all frames of the first video by inputting the reference frame, at least one previous frame, and at least one subsequent frame into the trained AI model to train the trained AI model.

18. The method according to claim 11, wherein: The steps of obtaining the extended video include: detecting at least one of a pattern and a color constituting the reference frame included in the first video; searching for an image using the communication interface, wherein the image is associated with the at least one of the detected pattern and color; and The trained AI model is generated by using the reference frame and the searched image as training data.

19. The method according to claim 11, further comprising: identifying an object included in the reference frame of the first video, determining a category of the first video according to the object, Selecting the trained AI model associated with the category of the first video.

20. The method according to claim 19, wherein: The categories include 2D animation or 3D animation.

Images