Electronic device for providing video editing function, or operating method thereof

Abstract

According to various embodiments, an electronic device may be set to: acquire a first image including a plurality of first image frames from a user device connected to the electronic device via a communication device; acquire a point set and a bounding box set for at least one object included in the first image by using an object recognition model; identify the contour of the at least one object on the basis of the point set and the bounding box set; acquire a mask for segmenting the at least one object on the basis of the contour of the at least one object identified from the first image; acquire a second image from which an area excluding the at least one object in the first image has been removed by using the mask; and transfer, via the communication device, the second image so that the user device outputs the second image. Various other embodiments identified through the specification are possible.

Description

Electronic device providing video editing function or method of operating the same

[0001] Various embodiments disclosed in this document relate to an electronic device providing a video editing function or a method of operating the same.

[0002] Recently, vision systems that utilize artificial intelligence models to identify specific objects within images have been developed and are being utilized in various fields. For example, they can identify objects within images comprising multiple image frames and provide various services related to the identified objects.

[0003] Meanwhile, deep learning-based semantic segmentation can be considered in relation to artificial intelligence models that identify objects within images. Many methods utilizing convolutional neural networks (CNNs) are being studied for deep learning-based semantic segmentation. CNNs are a type of multilayer, feed-forward artificial neural network used to analyze visual images. CNNs are a deep neural network technique that can be applied to artificial neural networks to effectively process images. It is a technique for classifying images through a process in which each element of a filter expressed as a matrix is ​​automatically learned to be suitable for data processing.

[0004] Fully Convolutional Networks (FCNs) are a representative method utilizing convolutional neural networks for object identification and image processing related to identified objects. FCNs were the first semantic segmentation methods to be proposed, utilizing a classification network for segmentation. However, FCNs suffer from the problem of losing object outlines and detailed information during segmentation map generation, resulting in inaccurate segmentation of object shapes.

[0005] In addition, in the past, in image processing based on object identification, a large number of down-sampling operations were performed in the encoder, which resulted in a significant decrease in processing speed and an increase in the loss of information about the identified object.

[0006] According to various embodiments of the present disclosure, at least one object included in an image can be detected and tracked to identify the outline of the at least one object with higher accuracy.

[0007] Additionally, according to various embodiments of the present disclosure, it is possible to identify the outline of at least one included object with high accuracy and provide a high-quality image corrected for the at least one object.

[0008] According to various embodiments, an electronic device may include a communication device, a storage device for storing an object recognition model learned to generate points and a bounding box for at least one object included in an image, and at least one processor, wherein the at least one processor may be configured to obtain a first image including a plurality of first image frames from a user device connected to the electronic device through the communication device, obtain a point set and a bounding box set for at least one object included in the first image using the object recognition model, identify an outline of the at least one object based on the point set and the bounding box set, obtain a mask for segmenting the at least one object based on the outline of the at least one object identified from the first image, obtain a second image in which an area excluding at least one object of the first image is removed using the mask, and transmit the second image through the communication device so that the user device outputs the second image.

[0009] According to various embodiments, a method of operating an electronic device may include obtaining a first image including a plurality of first image frames from a user device connected to the electronic device, obtaining a first set of points and a first set of bounding boxes for at least one object included in the first image using an object recognition model learned to generate points and bounding boxes for at least one object included in the image, identifying at least one object based on the first set of points and the first set of bounding boxes, obtaining a mask for segmenting at least one object of the first image identified from the first image, obtaining a second image from which an area excluding at least one object of the first image is removed using the mask, and transmitting the second image so that the user device outputs the second image.

[0010] According to various embodiments, a non-volatile computer-readable recording medium having recorded thereon a program for an operation method executable by a processor of an electronic device, the operation method including: obtaining a first image including a plurality of first image frames from a user device connected to the electronic device; obtaining a first set of points and a first set of bounding boxes for at least one object included in the first image using an object recognition model learned to generate points and bounding boxes for at least one object included in the image; identifying at least one object based on the first set of points and the first set of bounding boxes; obtaining a mask for segmenting at least one object of the first image identified from the first image; obtaining a second image in which an area excluding at least one object of the first image is removed using the mask; and transmitting the second image so that the user device outputs the second image.

[0011] According to various embodiments, an electronic device may include at least one camera, a display device, a communication device, a storage device for storing an object recognition model learned to generate points and bounding boxes for at least one object included in an image, and at least one processor, wherein the at least one processor may be configured to: acquire a first image including a plurality of first image frames through the at least one camera; acquire a point set and a bounding box set for at least one object included in the first image using the object recognition model; identify an outline of the at least one object based on the point set and the bounding box set; acquire a mask for segmenting the at least one object based on the outline of the at least one object identified from the first image; acquire a second image in which an area excluding at least one object of the first image is removed using the mask; and control the display device so that the display device outputs the second image.

[0012] Various embodiments disclosed in this document can detect and track at least one object included in an image acquired from a user device, thereby identifying the outline of said at least one object. In this case, the electronic device according to various embodiments can identify the outline of said at least one object with higher accuracy by utilizing both bounding box generation and key-point generation for said at least one object.

[0013] Additionally, various embodiments disclosed in this document can identify at least one object included in an image acquired from a user device and generate an image corrected for said at least one object. For example, at least one object included in an image can be recognized and an image can be generated by removing all but said at least one object, or an image corrected for said at least one object.

[0014] The electronic devices disclosed in this document enable users to obtain images with higher accuracy and correction for at least one object included in the image. Accordingly, the user can obtain images edited as intended.

[0015] Electronic devices according to various embodiments disclosed in this document can provide various functions related to image editing, thereby providing increased user convenience for image editing.

[0016] The electronic device according to various embodiments disclosed in this document can exhibit accurate object identification performance even when the object recognition model is trained with a small number of training data, as it is possible to utilize low-dimensional images and high-dimensional images in a decoder by using residual U-block technology.

[0017] In addition, various effects may be provided, either directly or indirectly, through this document.

[0018] FIG. 1 is a diagram illustrating a system that provides video editing functions according to various embodiments.

[0019] FIG. 2 is a block diagram of an electronic device according to various embodiments.

[0020] FIG. 3 illustrates a concept for controlling functions related to video editing in an electronic device according to various embodiments.

[0021] FIG. 4 is a flowchart illustrating an operation of an electronic device to generate an image from which an area excluding at least one object is removed according to various embodiments.

[0022] FIG. 5 is a flowchart illustrating an operation of an electronic device grouping multiple image frames within an image acquired from a user device to acquire an outline of at least one object according to various embodiments.

[0023] FIG. 6 is a drawing for explaining a mask generated by an electronic device according to various embodiments.

[0024] FIG. 7 is a drawing for explaining an image from which a background has been removed based on a mask generated by an electronic device according to various embodiments.

[0025] FIG. 8 is a flowchart illustrating an operation of an electronic device obtaining user input for an outline of at least one object generated using an object recognition model according to various embodiments.

[0026] FIG. 9 is a flowchart illustrating an operation of an electronic device to generate a corrected image with respect to at least one object based on user input according to various embodiments.

[0027] FIG. 10 is a diagram for explaining an image corrected through a mask generated by an electronic device based on user input according to various embodiments.

[0028] FIG. 11 is a diagram for explaining an image corrected through a mask generated by an electronic device based on user input according to various embodiments.

[0029] Figure 12 illustrates an execution screen output through a user device according to various embodiments.

[0030] FIG. 13a illustrates an execution screen of user input for a bounding box output through a user device according to various embodiments.

[0031] FIG. 13b illustrates an execution screen of user input for a bounding box output through a user device according to various embodiments.

[0032] FIG. 13c illustrates an execution screen output through a user device as user input for a bounding box is obtained according to various embodiments.

[0033] FIG. 14 is a diagram illustrating an electronic device applying user input according to various embodiments.

[0034] FIG. 15 is a diagram illustrating an electronic device generating a mask when a user input for a bounding box is an object addition input according to various embodiments.

[0035] FIG. 16a illustrates an execution screen of user input for key points output through a user device according to various embodiments.

[0036] FIG. 16b illustrates an execution screen of user input for key points output through a user device according to various embodiments.

[0037] FIG. 16c illustrates an execution screen output through a user device as user input for key points is obtained according to various embodiments.

[0038] FIG. 17 is a diagram illustrating an electronic device applying user input according to various embodiments.

[0039] FIG. 18 illustrates an execution screen of a user input input for a mask output through a user device according to various embodiments.

[0040] FIG. 19a illustrates an execution screen of a user input for selecting at least one object among a plurality of objects within an image frame according to various embodiments.

[0041] FIG. 19b illustrates an execution screen of a text-based user input for selecting at least one object among a plurality of objects within an image frame according to various embodiments.

[0042] FIG. 19c illustrates an execution screen of a text-based user input for selecting at least one object among a plurality of objects within an image frame according to various embodiments.

[0043] In connection with the description of the drawings, the same or similar reference numerals may be used for identical or similar components.

[0044] Specific structural or functional descriptions of various embodiments are merely illustrative for the purpose of explaining the various embodiments, and the various embodiments may be implemented in various forms and should not be construed as limited to the embodiments described in this specification or application.

[0045] Since various embodiments may have various modifications and take various forms, various embodiments are illustrated in the drawings and described in detail in this specification or application. However, the matters disclosed in the drawings are not intended to specify or limit the various embodiments, and should be understood to include all modifications, equivalents, and alternatives included within the spirit and technical scope of the various embodiments.

[0046] While terms such as "first" and / or "second" may be used to describe various components, these components should not be limited by these terms. These terms are only intended to distinguish one component from another; for example, without departing from the scope of the present disclosure, a first component may be referred to as a "second component," and similarly, a second component may also be referred to as a "first component."

[0047] When a component is referred to as being "connected" or "connected" to another component, it should be understood that it may be directly connected or connected to that other component, but that there may be other components in between. Conversely, when a component is referred to as being "directly connected" or "directly connected" to another component, it should be understood that there are no other components in between. Other expressions that describe the relationship between components, such as "between" and "directly between" or "adjacent to" and "directly adjacent to", should be interpreted similarly.

[0048] The terminology used herein is for the purpose of describing specific embodiments only and is not intended to limit the various embodiments. The singular expression includes the plural expression unless the context clearly indicates otherwise. In this specification, it should be understood that the terms "comprises" or "has" specify the presence of a described feature, number, step, operation, component, part, or combination thereof, but do not exclude in advance the presence or possibility of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0049] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by those of ordinary skill in the art to which this disclosure pertains. Terms defined in commonly used dictionaries should be interpreted to have a meaning consistent with their meaning in the context of the relevant technology, and will not be interpreted in an idealized or overly formal sense unless explicitly defined herein.

[0050] Hereinafter, the present disclosure will be described in detail by describing preferred embodiments of the present disclosure with reference to the attached drawings. The same reference numerals in each drawing represent the same components.

[0051]

[0052] FIG. 1 is a diagram illustrating a video editing system that provides video editing functions according to various embodiments.

[0053] Referring to FIG. 1, a video editing system (100) may include a user device (102), a network (104), and an electronic device (106).

[0054] Meanwhile, the video in this document can be expressed as a video. Furthermore, the points for at least one object within the video can be combined into a key-point representation.

[0055] According to various embodiments, the user device (102) may be a device including a signage device, such as a mobile phone, a smart phone, a personal digital assistant (PDA), a laptop computer, a television, a wearable device, or a head mounted device (HMD).

[0056] According to various embodiments, the user device (102) may include various output devices capable of providing video content to the user. For example, the user device (102) may include at least one of an audio device, a display device, or at least one camera capable of capturing video.

[0057] According to various embodiments, the user device (102) may include various input devices capable of obtaining input from a user. For example, it may include at least one of a keyboard, a touchpad, keys (e.g., buttons), a mouse, a microphone, and a digital pen (e.g., a stylus pen).

[0058] According to various embodiments, the network (104) may include any of a variety of wireless communication networks suitable for coupling to enable communication with the user device (102). For example, the network may include WLAN, WAN, PAN, cellular, WMN, WiMAX, GAN, 6LowPAN, and the like.

[0059] According to various embodiments, the electronic device (106) may include a standalone host computing system, an on-board computer system integrated with the user device (102), a mobile device, or any other hardware platform capable of providing video editing capabilities and video content to the user device (102). For example, the electronic device (106) may include a cloud-based computing architecture suitable for servicing video editing executed on the user device (102). Accordingly, the electronic device (106) may include one or more servers (110) and data storage (108). For example, the electronic device (106) may include a Software as a Service (SaaS) architecture, a Platform as a Service (PaaS) architecture, an Infrastructure as a Service (IaaS) architecture, or other similar cloud-based computing architecture.

[0060] According to various embodiments, the electronic device (106) and / or the user device (102) may be configured as one device performing each function, without being limited to the illustrated example.

[0061] For example, the electronic device (106) may perform the functions of the user device (102), including the components included in the user device (102). When the electronic device (106) provides the functions of the user device (102), the electronic device (106) may provide an editing function for a video stored in the electronic device (106). For example, the electronic device (106) may acquire and store a video including a plurality of image frames, audio information, and / or subtitle information through a camera and a microphone. In addition, the electronic device (106) may edit the video based on a user input and output the edited video through a display device.

[0062] It will be described later that the electronic device (106) provides a video editing function according to various embodiments.

[0063]

[0064] Figure 2 is a block diagram of an electronic device (200) according to one embodiment.

[0065] Referring to FIG. 2, an electronic device (200) (e.g., the electronic device (106) of FIG. 1) may include a processor (210), a storage device (220) (e.g., the data storage (108) of FIG. 1), and / or a communication device (230). The components listed above may be operatively or electrically connected to one another. The components of the electronic device (200) illustrated in FIG. 2 may be modified, deleted, or added, for example.

[0066] According to various embodiments, the electronic device (200) may include a processor (210). The processor (210) may include hardware for executing instructions, such as instructions constituting a computer program. For example, to execute an instruction, the processor (210) may retrieve (or fetch) the instruction from an internal register, an internal cache, a storage device (220) (including a memory), decode and execute the instruction, and then store the result in the internal register, the internal cache, or the storage device (220).

[0067] In various embodiments, the processor (210) may execute software (e.g., a computer program) to control at least one other component (e.g., a hardware or software component) of the electronic device (200) connected to the processor (210) and perform various data processing or operations. According to various embodiments, as at least a part of the data processing or operations, the processor (210) may store commands or data received from another component (e.g., a communication device (230)) in volatile memory, process the commands or data stored in the volatile memory, and store result data in non-volatile memory.

[0068] According to various embodiments, the processor (210) may include at least one of a central processing unit (CPU), a graphics processing unit (GPU), a micro controller unit (MCU), a sensor hub, a supplementary processor, a communication processor, an application processor, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or a neural processing unit (NPU), and may have multiple cores.

[0069] According to various embodiments, the processor (210) (e.g., a neural network processing device) may include a hardware structure specialized for processing an artificial intelligence model. The artificial intelligence model may be generated through machine learning. The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the above-described examples. The artificial intelligence model may include a plurality of artificial neural network layers. The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), deep Q-networks, or a combination of two or more of the above, but is not limited to the above-described examples. In addition to the hardware structure, the artificial intelligence model may additionally or alternatively include a software structure.

[0070] According to various embodiments, the processor (210) may obtain a first image including a plurality of image frames from a user device (e.g., the user device (102) of FIG. 1) through a communication device (230). For example, the first image may include a plurality of image frames having a series of temporal flows and an image including audio information and / or subtitle information corresponding to the plurality of image frames.

[0071] According to various embodiments, the first image may include an image captured by at least one camera of the user device (102). For example, the user may capture an image using the user device (102) and transmit the image to the electronic device (200) for editing.

[0072] According to various embodiments, the processor (210) may obtain a first set of points and a first set of bounding boxes for at least one object included in the first image by using an object recognition model trained to generate points and / or bounding boxes for at least one object included in the image (e.g., the object recognition model of the object recognition module (305) described with reference to FIG. 3). For example, the processor (210) may input the first image to the object recognition model. In addition, the processor may obtain a first set of points and a first set of bounding boxes for at least one object generated by recognizing at least one object for each of a plurality of image frames of the first image output from the object recognition model.

[0073] According to various embodiments, the processor (210) may recognize a first contour of the at least one object based on the first point set and the first bounding box set. For example, the processor (210) may identify a first contour of the at least one object by applying both the first point set, which extracts skeleton data for the at least one object included in the first image, and the first bounding box set, which includes at least one bounding box of the at least one object.

[0074] An electronic device (200) according to various embodiments can identify the first contour of at least one object with higher accuracy by utilizing both generation of a set of bounding boxes and generation of a set of key-points for at least one object.

[0075] According to various embodiments, the processor (210) may obtain a mask for segmenting the at least one object based on a first outline of the at least one object identified from the first image. For example, the processor (210) may obtain a mask for segmenting the at least one object for each of a plurality of image frames of the first image. According to various embodiments, the mask may include an image for covering, modifying, or editing a specific portion of the image frame.

[0076] According to various embodiments, the processor (210) may obtain a second image from which an area excluding at least one object of the first image is removed by using the mask. For example, the processor (210) may obtain a second image from which an area excluding at least one object is removed for each of a plurality of image frames of the first image by using a mask for each of the plurality of image frames of the first image.

[0077] According to various embodiments, the processor (210) may obtain the second image by encoding image frames from which an area excluding at least one object is removed from each of a plurality of image frames of the first image using the mask and audio information and / or subtitle information of the first image.

[0078] According to various embodiments, when the at least one object is the foreground of the first image, the second image may include an image in which the background excluding the foreground is removed from the first image.

[0079] According to various embodiments, the processor (210) may transmit the second image so that the user device (102) outputs the second image. For example, the processor (210) may transmit the second image to the user device (102) via a communication device (230).

[0080] According to various embodiments, the user device (102) that has acquired the second image can output the second image through a display device (e.g., a display) included in the user device (102).

[0081] According to various embodiments, the processor (210) may generate a mask based on user input obtained through the user device (102).

[0082] According to various embodiments, the processor (210) may transmit to the user device (102) information about the first point set and / or the first bounding box set. For example, the processor (210) may transmit information about the first point set and / or the first bounding box set to the user device (102) via the communication device (230).

[0083] According to various embodiments, the user device (102) that acquired the second image may output information about the first point set and the first bounding box set using visual objects through a display device (e.g., a display) included in the user device (102). For example, the user device (102) may overlap and display visual objects for each of the first point set and the first bounding box set for each of a plurality of image frames included in the first image.

[0084] According to various embodiments, the processor (210) may obtain a first user input for at least one object included in a first image frame among a plurality of image frames for the first image through the user device (102). For example, the processor (210) may obtain a user input for a point set and / or a bounding box set of the first image frame displayed through the user device (102).

[0085] According to various embodiments, the user input may include various inputs related to the at least one object of the first image frame. For example, the user input may include at least one of an input for selecting the at least one object, an input for a point set of the at least one object, an input for a bounding box of the at least one object, a masking input for a point set of the at least one object, and a text input related to selecting an outline of the at least one object.

[0086] According to various embodiments, the processor (210) may identify a second outline of at least one object in the first image frame based on the user input. For example, the processor (210) may identify an outline of at least one object in the first image frame among a plurality of image frames of the first video as the second outline based on the user input.

[0087] According to various embodiments, the processor (210) may obtain a mask for segmenting the at least one object in the first image based on the first contour and / or the second contour. For example, the mask may be obtained based on the second contour for at least one object in the first image frame among a plurality of image frames included in the first image and the first contour for at least one object in the remaining image frames.

[0088] According to various embodiments, the processor (210) may obtain a second image that corrects the first image in relation to the at least one object for each of a plurality of image frames of the first image, using a mask obtained based on the first contour and / or the second contour.

[0089] According to various embodiments, the second image may be obtained by encoding corrected image frames for the at least one object, audio information of the first image, and / or subtitle information of the first image using a mask obtained based on the first contour and the second contour.

[0090] According to various embodiments, the processor (210) may transmit the second image so that the user device (102) outputs the second image. For example, the processor (210) may transmit the second image to the user device (102) via a communication device (230).

[0091] According to various embodiments, the user device (102) that has acquired the second image can output the second image through a display device (e.g., a display) included in the user device (102).

[0092] According to various embodiments, the storage device (220) may include mass storage for data or instructions. For example, the storage device (220) may include a hard disk drive (HDD), a floppy disk drive, flash memory, an optical disk, an optical-magnetic disk, a magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more thereof.

[0093] In various embodiments, the storage device (220) may include nonvolatile, solid-state memory, read-only memory (ROM). Such ROM may be mask-programmed ROM, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), electrically alterable ROM (EAROM), or flash memory, or a combination of two or more thereof.

[0094] Although this disclosure describes and illustrates a particular storage device, this disclosure contemplates any suitable storage device, and according to various embodiments, the storage device (220) may be internal or external to the electronic device (106).

[0095] According to various embodiments, the processor (210) may store a module related to video editing described with reference to FIG. 3 in the storage device (220).

[0096] According to various embodiments, the processor (210) may execute operations or data processing related to control and / or communication of at least one other component of the electronic device (200) using instructions stored in the storage device (220).

[0097] According to various embodiments, the electronic device (200) may include a storage device (220). According to various embodiments, the storage device (220) may store various data used by at least one component (e.g., processor (210)) of the electronic device (200). The data may include, for example, input data or output data for software (e.g., program) and commands related thereto.

[0098] According to various embodiments, the program may be stored as software in the storage device (220) and may include, for example, an operating system, middleware, or an application. According to various embodiments, the storage device (220) may store instructions that cause the processor (210) to process data or control components of the electronic device (200) to perform operations of the electronic device (200) when executed. The instructions may include code generated by a compiler or code that can be executed by an interpreter.

[0099] According to various embodiments, the storage device (220) may store various information obtained through the processor (210). For example, the storage device (220) may store at least one of a plurality of image frames obtained from the processor (210), an image including the plurality of image frames, order information of each of the plurality of image frames, information on image frame groups that group the plurality of image frames, information on at least one object of each of the plurality of image frames, information on a point set group and a bounding box set for the at least one object output through an object recognition model, and user input information obtained from the user device (102). In addition, the storage device (220) may store identification information on the user device (102) connected to the electronic device (200).

[0100] According to various embodiments, the storage device (220) may store an object recognition model trained to extract skeleton data for at least one object in an image to obtain a point set and a bounding box set for the at least one object. For example, the object recognition model may be a deep neural network model trained to identify and track an object in an image and extract a key-point set and a bounding box set for the at least one object in the image.

[0101] According to various embodiments, the electronic device (200) may include a communication device (230). In various embodiments, the communication device (230) may support establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (200) and an external electronic device (e.g., the user device (102) of FIG. 1), and performance of communication through the established communication channel. The communication device (230) may operate independently from the processor (210) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication device (230) may include a wireless communication module (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device via a first network (e.g., network (104) of FIG. 1) (e.g., a short-range communication network such as Bluetooth, WiFi Direct (wireless fidelity direct) or IrDA (infrared data association)) or a second network (e.g., network (104) of FIG. 1) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., network (104) of FIG. 1) (e.g., a LAN or WAN). The various types of communication modules may be integrated into one component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips).

[0102] According to various embodiments, the electronic device (200) can transmit and receive various data with various external devices through the communication device (230). In addition, the electronic device (200) can store the acquired data in the storage device (220). For example, the electronic device (200) can acquire an image including a plurality of image frames from the user device (102) through the communication device (230). For example, the electronic device (200) can acquire a user input for at least one object included in the image through the communication device (230). For example, the electronic device (200) can transmit information about a point set and a bounding box set for at least one object in the image to the user device (102) through the communication device (230). For example, the electronic device (200) can transmit an image in which the image is corrected in relation to the at least one object to the user device (102) through the communication device (230).

[0103] According to various embodiments, the electronic device (200) may include a computer system. For example, the computer system may be at least one of an embedded computer system, a system-on-chip (SOC), a single-board computer system (SBC), a computer-on-module (COM), a system-on-module (SOM), a desktop computer system, a laptop or notebook computer system, a server, a tablet computer system, and a mobile terminal. For example, the electronic device (200) may include one or more computer systems residing in a cloud, which may include one or more cloud components.

[0104] According to various embodiments, the electronic device (200) may perform one or more operations of one or more methods described or presented in the present disclosure without substantial spatial or temporal limitations. Furthermore, the electronic device (200) may perform one or more operations of one or more methods described or presented in the present disclosure in real time or in batch mode. For example, the electronic device (200) may perform one or more operations of one or more methods described or presented in the present disclosure at different times or locations.

[0105] According to various embodiments, a non-transitory computer-readable recording medium storing computer instructions for performing operations of an electronic device (200) described in the present disclosure may be provided. The non-transitory computer-readable recording medium or storage medium may include one or more semiconductor-based or other integrated circuits (ICs) (such as, for example, a field-programmable gate array (FPGA) or an application-specific IC (ASIC)), a hard disk drive (HDD), a hybrid hard drive (HHD), an optical disk, an optical disk drive (ODD), a magneto-optical disk, a magneto-optical drive, a floppy diskette, a floppy disk drive (FDD), a magnetic tape, a solid-state drive (SSD), a RAM-drive, a SECURE DIGITAL card or drive, any other suitable computer-readable non-transitory storage medium, or any suitable combination of two or more thereof, as appropriate.

[0106] According to various embodiments, the computer-readable non-transitory storage medium may be volatile, non-volatile, or a combination of volatile and non-volatile, as appropriate. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, the term "non-transitory storage medium" only means that it is a tangible device and does not contain signals (e.g., electromagnetic waves), and this term does not distinguish between cases where data is stored semi-permanently and cases where data is stored temporarily. For example, a "non-transitory storage medium" may include a buffer in which data is temporarily stored.

[0107] The operating method of an electronic device according to various embodiments disclosed in this document may be provided as a computer program product. The computer program product may be traded between sellers and buyers as a product. The computer program product may be distributed in the form of a device-readable recording medium (e.g., compact disc read-only memory (CDROM)) or may be available through an application store (e.g., Play Store). TM ) or directly between two user devices (e.g., smartphones), online distribution (e.g., downloading or uploading). In the case of online distribution, at least a portion of the computer program product (e.g., downloadable app) may be at least temporarily stored or temporarily created in a device-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0108] According to various embodiments, when the electronic device (200) described in the present disclosure provides the functions of the user device (102), the electronic device (200) may include at least one camera (not shown) and / or a display device (not shown). Hereinafter, a case in which the electronic device (200) includes at least one camera (not shown) and / or a display device (not shown) will be described.

[0109] According to various embodiments, the processor (210) may acquire the first image including a plurality of image frames through at least one camera. For example, the processor (210) may activate the at least one camera based on a shooting start command, and acquire the first image including a plurality of image frames, audio information, and / or subtitle information through the at least one camera.

[0110] According to various embodiments, the processor (210) may identify the outline of at least one object included in the first image using an object recognition model, and obtain a mask for segmenting the at least one object based on the first outline of the at least one object. This is briefly described as it overlaps with the description above.

[0111] According to various embodiments, the processor (210) may obtain a second image from which an area of ​​the first image excluding at least one object is removed using the mask, and control the display device to output the second image. For example, the processor (210) may output the second image through the display device included in the electronic device (200).

[0112] According to various embodiments, when the processor (210) generates a mask based on a user input, the display device may control the display device to display the first point set and the first bounding box set for the at least one object. For example, the display device may, through the control of the processor (210), overlap and display a visual object for each of the first point set and the first bounding box set for each of a plurality of image frames included in the first image.

[0113] According to various embodiments, the processor (210) may obtain a user input for at least one object included in the first image frame among the plurality of image frames. For example, the processor (210) may obtain a user input through an input device (e.g., a keyboard, a touch pad, a key (e.g., a button), a mouse, a microphone, a digital pen (e.g., a stylus pen)) included in the electronic device (200).

[0114] According to various embodiments, the processor (210) may identify a second contour of at least one object in the first image frame based on the user input. Furthermore, the processor (210) may obtain a mask for segmenting the at least one object based on the first contour and the second contour of the at least one object. This overlaps with the description above and is therefore briefly described.

[0115] According to various embodiments, the processor (210) may obtain a second image by correcting the first image in relation to the at least one object using the mask, and control the display device to output the second image.

[0116]

[0117] FIG. 3 illustrates a concept for controlling functions related to video editing in an electronic device according to various embodiments.

[0118] Referring to FIG. 3, the electronic device (200) may utilize hardware and / or software modules (300) to support various video editing-related functions. For example, the processor (210) may drive an image acquisition module (301), an image grouping module (303), an object recognition module (305), a mask generation module (307), an object correction module (309), an object selection module (311), an image generation module (313), an evaluation module (315), and / or a personalization module (317) by executing commands stored in a storage device (220). In various embodiments, software modules other than those illustrated in FIG. 3 may be implemented. For example, at least two modules may be integrated into one module, or one module may be split into two or more modules. In addition, work performance may be improved by having hardware and software modules share a single function. For example, the electronic device (200) may include both an encoder implemented as hardware and an encoder implemented as a software module, and some of the data acquired through at least one camera module may be processed by the hardware encoder and the remaining part may be processed by the software encoder.

[0119] According to various embodiments, the image acquisition module (301) may provide a user interface (UI) / graphical UI (GUI) related to video upload to a user through a user device (102), and may acquire an image (e.g., a first image described with reference to FIG. 2) through the user device. For example, the image may be acquired by controlling a function related to video upload in response to a user input provided through a UI / GUI output through a display device of the user device (102). In addition, the image acquisition module (301) may extract a plurality of image frames, audio information, and / or subtitle information of the image acquired through the user device (102).

[0120] According to various embodiments, the image grouping module (303) may group a plurality of image frames included in an image acquired through the image acquisition module (301) into preset standard units to create image frame groups. For example, the image grouping module (303) may recognize a scene transition of the image using an object recognition model included in the object recognition module (305), and group a plurality of image frames of the image based on the scene transition to create image frame groups. An operation of the image grouping module (303) to create image frame groups based on the scene transition will be described below with reference to FIG. 5.

[0121] According to various embodiments, the object recognition module (object detection model) (305) may include an object recognition model trained to detect a point set and / or a bounding box of at least one object included in an image. For example, the object recognition model may be a model trained to detect a point set and / or a bounding box set of at least one object through various training data.

[0122] According to various embodiments, the training data for learning the object recognition model may include training data that distinguishes at least one object and a background within a plurality of image frames included in an image, and assigns a label corresponding to the background and a label corresponding to each of at least one object.

[0123] According to various embodiments, the object recognition model may be composed of an artificial neural network model. For example, the object recognition model may be a deep neural network model trained to identify and track objects in an image and extract a set of key points and a set of bounding boxes for at least one object in the image. For example, the object recognition model may be implemented using a Region-based Convolutional Neural Network (R-CNN), a Faster Region-based Convolutional Neural Network (Faster R-CNN), a Single Shot Multibox Detector (SSD), YOLO v4, CenterNet, or MobileNet. However, the object recognition model of the present disclosure is not limited to the aforementioned deep neural network model and may be implemented using any other suitable neural network model.

[0124] According to various embodiments, the object recognition model can obtain a point set by extracting skeleton data of at least one object included in an image. For example, the object recognition model can extract at least one object included in an image and obtain a point set by extracting skeleton data of the extracted object. For example, if the object is a human object or an animal, the joint part or specific part of the object can be detected. In addition, if the object is a human object, body parts such as the head, eyes, nose, mouth, ears, neck, shoulders, elbows, wrists, fingertips, torso, hip joints, wrists, knees, ankles, and toes can be extracted. The skeleton data can be expressed as XY coordinate values ​​as coordinates in the image to form a point set.

[0125] According to various embodiments, the object recognition model may utilize a joint detection algorithm, such as the Kinetics dataset or the NTU-RGB-D (Nanyang Technological University's Red Blue Green and Depth information) dataset, to extract skeletal data of joints or specific parts of the body to obtain a point set. In this case, the number of skeletal joints per object may be arbitrarily defined.

[0126] According to various embodiments, the point set may include a skeleton key point composed of XY coordinate values ​​generated based on a model that structures the shape of the body, and a face key point composed of XY coordinate values ​​generated based on a model for individual locations of the eyes, nose, mouth, etc. of the face.

[0127] According to various embodiments, the object recognition model can track at least one object in an image. For example, the object recognition model can track at least one object recognized from multiple image frames included in the image, and track changes in at least one object within the multiple image frames. For example, if multiple objects are included in an image frame, the object recognition model can extract skeleton data for each of the multiple objects to obtain a point set, and create and track a layer for each of the multiple objects. According to various embodiments, if multiple objects included in an image are recognized at specific time intervals, a layer can be created for each time interval in which each object is recognized.

[0128] According to various embodiments, the object recognition model of the electronic device (200) may include an encoder and a decoder for extracting a set of bounding boxes from a plurality of image frames in an image.

[0129] According to various embodiments, the encoder and the decoder may include a plurality of stages, wherein the plurality of stages may include some encoder stages, some decoder stages, and a bridge stage connecting the encoder stages and the decoder stages. In one embodiment, each of the plurality of stages may include a residual U block.

[0130] According to various embodiments, the encoder and decoder may be connected as a network having a nested U-shaped structure. This nested U-shaped structure can effectively extract and combine multi-scale features of intra-stages. The encoder may extract and compress features from image frames of a video to generate context information. The decoder may be configured to expand a feature map including the context information and output a set of bounding boxes based on segmentation.

[0131] According to various embodiments, the U-shaped network may include an encoder stage (including a bridge stage), a decoder stage, and a bridge stage. This U-shaped network has the effect of minimizing information loss by utilizing intermediate feature information from the encoder in the decoder through a concatenation operation. As a result, segmentation accuracy (the accuracy of extracting a set of bounding boxes) can be improved compared to conventional FCNs.

[0132] According to various embodiments, the object recognition model may include a residual U block. Here, the residual U block may be an improved concept of an existing residual block. For example, a residual block, as used in the ResNet algorithm, may include a concept that includes multi-scale features and local feature functions.

[0133] According to various embodiments, the residual block may include a process of adding the scale features of each layer and the input values ​​of the local feature function to the output value. The electronic device (200) may be able to utilize low-dimensional and high-dimensional images in the decoder due to the use of the residual U block. Accordingly, even if the object recognition model performs training with a small number of training data (R), it can exhibit accurate object identification performance (acquisition of a point set and a bounding box set).

[0134] Accordingly, the object recognition model of the electronic device (200) according to one embodiment of the present disclosure can drastically reduce the number of layers of the network of the encoder and decoder by using the residual U block, thereby reducing the amount of computation and increasing the processing speed, while increasing the calculation of the hierarchical downsampling vector in each block, thereby accurately identifying objects of various resolutions.

[0135] According to various embodiments, the object recognition module (305) can identify at least one object with higher accuracy by utilizing both a point set and a bounding box for at least one object in each of the plurality of image frames of the video using the object recognition model.

[0136] Additionally, the object recognition module (305) can generate a point set and a bounding box set for each of the multiple objects when multiple objects exist in the input image.

[0137] According to various embodiments, the mask generation module (307) may identify a first outline of at least one object based on a point set and a bounding box for at least one object in the image, and generate a mask for segmenting the at least one object based on the outline (including a boundary line) of the identified at least one object. According to various embodiments, the mask generation module (307) may generate a mask for segmenting the at least one object for each of a plurality of image frames included in the image.

[0138] The above mask may include an image for covering, modifying, or editing a specific portion of an image frame. Alternatively, the mask may be map data consisting of '0's' and '1's, and may include '0' or '1' data for each of a plurality of pixels of the image frame.

[0139] According to various embodiments, the mask generation module (307) may generate a mask based on order information, the point set and the bounding box set, and / or user information for each of the plurality of image frames of the video. For example, the mask generation module (307) may obtain a user input for the at least one object in a specific image frame among the plurality of image frames from a user device, and additionally use the user inputs to generate a mask. For example, the mask generation module (307) may identify an outline of the at least one object based on the order information and the user input of the specific image frame, and generate a mask based on the identified outline.

[0140] For example, a first contour can be obtained based on a set of points and a set of bounding boxes determined through the object recognition module (305), and a second contour can be obtained based on a user input for the at least one object, thereby identifying the shape of the at least one object and obtaining mask(s) for the identified at least one object.

[0141] According to various embodiments, the mask generation module (307) may apply user input for at least one object of a specific image frame to other image frames other than the specific image frame based on user input obtained through the user device, thereby generating a mask for each of the other image frames.

[0142] According to various embodiments, the user input for at least one object may be obtained via the object correction module (309) and / or the object selection module (311).

[0143] According to various embodiments, the object correction module (309) may provide a user with a UI / GUI for a user input regarding at least one object included in a specific image frame (e.g., the first image frame described with reference to FIG. 2) among a plurality of image frames of the image through the user device (102), and may obtain a user input regarding a point set and / or a bounding box set for the at least one object through the user device. In one example, the user input may include at least one of a point set input or a bounding box set input for the at least one object. Accordingly, the object correction module (309) may identify a correction request for the at least one object based on the point set input or the bounding box set input, and correct the at least one object included in the specific image frame based on the correction request. The object correction module (309) may provide information regarding a user input related to the correction request to the mask generation module (307). An outline of the at least one object corrected based on the user input may be identified.

[0144] In addition, according to various embodiments, the object correction module (309) may correct at least one object of the specific image frame to generate a corrected specific image frame. For example, based on a correction request for the at least one object, the object correction module (309) may correct pixels of the at least one object of the specific image frame to generate a corrected specific image frame. According to various embodiments, the object correction module (309) may utilize various generative neural network models (e.g., generative adversarial networks (GAN)) in generating the corrected specific image frame.

[0145] According to various embodiments, the object selection module (311) may provide a user with a UI / GUI for a user input regarding at least one object included in a specific image frame (e.g., the first image frame described with reference to FIG. 2) among a plurality of image frames of the image, and may acquire a user input regarding the at least one object from a user device. For example, the object selection module (311) may acquire a user input for selecting at least one object to be identified among a plurality of objects included in the image.

[0146] Additionally, according to various embodiments, the object selection module (311) may obtain a user input including any one of an input for a point set for the at least one object included in a specific frame, an input for a bounding box of the at least one object, a masking input for a point set of the at least one object, and a text input related to selecting an outline of the at least one object, and provide information about the user input to the mask generation module (307).

[0147] According to various embodiments, the image generation module (313) may obtain a corrected image by rendering the plurality of image frames using a mask corresponding to each of the plurality of image frames obtained through the mask generation module (307). For example, the corrected image may be obtained by convolving the plurality of image frames of the image with the mask corresponding to each of the plurality of image frames. For example, if the mask is composed of map data, if the mask value for the image frame is '1', the image frame may be output with a specified color or by removing it. In addition, if the mask map data is '0', it may be determined as an object and the image of the corresponding pixel may be output as is.

[0148] According to various embodiments, the image generation module (313) may obtain a corrected image by encoding the corrected image and audio information and / or subtitle information of the original image in relation to at least one object for each of a plurality of image frames of the original image based on data provided from the object correction module (309), the object selection module (311) and / or the mask generation module (307).

[0149] According to various embodiments, the evaluation module (315) may provide a UI / GUI related to feedback of a mask generated from the mask generation module (307) to the user through the user device (102), and obtain user feedback through the user device. Accordingly, according to various embodiments, the evaluation module (315) may obtain feedback information indicating user satisfaction with the generated mask for each of a plurality of image frames, and may retrain the object recognition model included in the object recognition module (305) using the feedback information.

[0150] According to various embodiments, the personalization module (317) may control the mask generation module (307) to be customized for the user by using information about the mask generated from the mask generation module (307), user input information obtained through various modules, and / or feedback information obtained through the evaluation module (315). For example, feedback information indicating user satisfaction with the generated mask for each of a plurality of image frames may be obtained from the evaluation module (315), and the object recognition model included in the object recognition module (305) may be retrained using the feedback information to use a user-customized object recognition model. Alternatively, the mask generation module (307) may be controlled to store user input information related to mask generation and generate a mask (e.g., using a macro method or operation history information) based on the stored input information without the user repeatedly inputting it through the user device.

[0151] In the embodiment of FIG. 3, the functions performed by the image acquisition module (301), the image grouping module (303), the object recognition module (305), the mask generation module (307), the object correction module (309), the object selection module (311), the image generation module (313), the evaluation module (315) and / or the personalization module (317) may be understood as being performed by the processor (210) executing commands stored in the storage device (220). In addition, in various embodiments, the electronic device (200) may utilize one or more hardware processing circuits to perform various functions and operations disclosed in this document.

[0152] Additionally, the hardware / software connection relationship illustrated in FIG. 3 is for convenience of explanation and does not limit the flow / direction of data or commands. Components included in the electronic device (200) may have various electrical / operational connection relationships.

[0153]

[0154] FIG. 4 is a flowchart (400) illustrating an operation of an electronic device to generate an image from which an area excluding at least one object is removed according to various embodiments. FIG. 5 is a flowchart (500) illustrating an operation of an electronic device to group a plurality of image frames within an image acquired from a user device to acquire an outline of at least one object according to various embodiments.

[0155] Each of the operations described below may be performed in combination with one another. In addition, among the operations described below, operations by an electronic device (200) (e.g., electronic device (106) of FIG. 1) may refer to operations by a processor (210) of the electronic device (200).

[0156] In addition, the “information” described below may be interpreted to mean “data” or “signal,” and “data” may be understood as a concept that includes both analog data and digital data.

[0157] According to various embodiments, the operations illustrated in FIGS. 4 and 5 may be performed in various orders, not limited to the order illustrated. Furthermore, according to various embodiments, more operations may be performed than those illustrated in FIGS. 4 and 5, or at least one operation may be performed less than those illustrated in FIGS.

[0158] Descriptions that overlap or are similar to the above-described descriptions of the operations of the electronic device (200) according to various embodiments may be omitted.

[0159] Referring to FIG. 4, the electronic device (200) may obtain a first image including a plurality of image frames from a user device (e.g., the user device (102) of FIG. 1) in operation 401. For example, the first image may include a plurality of image frames having a series of temporal flows and an image including audio information and subtitle information corresponding to the plurality of image frames.

[0160] According to various embodiments, the electronic device (200) may obtain a point set and a bounding box set for at least one object included in a first image by using an object recognition model (e.g., an object recognition model included in the object recognition module (305) of FIG. 3) in operation 403. For example, the electronic device (200) may input the first image into an object recognition model stored in a storage device (e.g., a storage device (220) of FIG. 2) and obtain the first point set and the first bounding box set generated by recognizing at least one object for each of a plurality of image frames of the first image output from the object recognition model. According to various embodiments, the object recognition model may be a model trained to detect and track at least one object included in an image and output a point set and a bounding box set that extract skeleton data for at least one object.

[0161] According to various embodiments, the electronic device (200) may identify an outline of at least one object based on a point set and a bounding box set in operation 405. For example, the electronic device (200) may identify an outline of at least one object by applying both the point set extracted from skeleton data for the at least one object included in the first image and the bounding box set including at least one bounding box of the at least one object.

[0162] According to various embodiments, the electronic device (200) may obtain a mask for segmenting at least one object based on an outline of at least one object identified from the first image in operation 407. For example, the electronic device (200) may obtain a mask for segmenting the at least one object for each of a plurality of image frames of the first image.

[0163] According to various embodiments, the electronic device (200) may obtain a plurality of masks for segmenting at least one object based on an outline of the at least one object, and determine the mask based on the reliability of the plurality of masks. For example, the electronic device (200) may obtain a plurality of masks and a reliability value for each of the plurality of masks, and determine the mask with the highest reliability value as the mask corresponding to the image frame.

[0164] According to various embodiments, the electronic device (200) may, in operation 409, obtain a second image from which an area excluding at least one object of the first image is removed by using a mask. For example, the electronic device (200) may obtain a second image from which an area excluding at least one object is removed for each of a plurality of image frames of the first image by using a mask for each of the plurality of image frames of the first image.

[0165] According to various embodiments, the electronic device (200) may transmit the second image to the user device so that the user device outputs the second image in operation 411. For example, the electronic device (200) may transmit the second image to the user device (102) via the communication device (230).

[0166]

[0167] Referring to FIG. 5, the electronic device (200) can obtain a first image including a plurality of image frames from the user device in operation 501, similar to operation 401.

[0168] According to various embodiments, the electronic device (200) may generate image frame groups by grouping a plurality of image frames into preset reference units in operation 503.

[0169] According to various embodiments, the electronic device (200) may recognize a scene change of the first image using the object recognition model. For example, the electronic device (200) may identify at least one object included in each of a plurality of image frames of the first image using the object recognition model, and recognize a scene change based on at least one of a change of the at least one object, a change in the type of the at least one object, a change in the number of the at least one object, a change in the main color value of each of the plurality of image frames, audio information of the first image, subtitle information of the first image, order information of the plurality of image frames, shooting time information of each of the plurality of image frames, and a user input for image frame grouping. In addition, the electronic device (200) may group the plurality of image frames of the first image based on the scene change to generate the image frame groups.

[0170] According to various embodiments, the electronic device (200) may track at least one object for each image frame group using an object recognition model in operation 505 to obtain a point set and a bounding box set. For example, the electronic device (200) may track at least one object for each image frame group to obtain a point set and a bounding box set more quickly and with high accuracy. According to various embodiments, when the electronic device (200) obtains a point set and a bounding box set for each of a plurality of image frames, the electronic device (200) may map and store identification information and order information of the image frames for the image frame group corresponding to the point set and the bounding box set.

[0171] According to various embodiments, the electronic device (200) may subsequently perform operations 403 to 411 using the point set and bounding box set obtained in operation 505.

[0172]

[0173] FIG. 6 is a diagram for explaining a mask generated by an electronic device according to various embodiments. FIG. 7 is a diagram for explaining an image from which a background has been removed based on a mask generated by an electronic device according to various embodiments.

[0174] Referring to FIG. 6, a drawing is shown relating to an electronic device (200) obtaining a first corrected image frame from which an area other than at least one object is removed from one first original image frame among a plurality of image frames of a first image.

[0175] Referring to FIG. 7, a drawing is shown relating to an electronic device (200) obtaining a mask for a first image including a plurality of image frames, and obtaining a second image by removing an area excluding at least one object based on the mask.

[0176] According to various embodiments, the electronic device (200) can obtain a point set and a bounding box set for at least one object included in a plurality of image frames included in a first image acquired from a user device.

[0177] For example, the electronic device (200) may obtain a bounding box set (610) for a first original image frame among a plurality of image frames included in a first image using an object recognition model. For example, the object recognition model may identify a tie object and a person object, which are objects included in the first original image frame. Accordingly, the bounding box set (610) may include a bounding box (611) for the person object and a bounding box (612) for the tie object.

[0178] For example, the electronic device (200) can obtain a point set (620) for a first original image frame among a plurality of image frames included in a first image using an object recognition model. For example, the object recognition model can extract and track skeleton data for a person included in the first original image frame to obtain key points (621) for each skeleton data.

[0179] According to various embodiments, the electronic device (200) may identify an outline (630) of at least one object (e.g., a person) included in a first original image frame based on a bounding box set (610) and a point set (620) obtained using an object recognition model. According to various embodiments, the electronic device (200) may identify an outline (630) of at least one object (e.g., a person) with higher accuracy by utilizing both the bounding box set (610) and the point set (620) of the at least one object.

[0180] According to various embodiments, the electronic device (200) may obtain a mask (640) for segmenting at least one object (e.g., a person) based on an outline (630) of a first original image frame. Using the mask, a first corrected image frame (650) may be obtained from which an area of ​​the first original image frame excluding the at least one object (e.g., a person) is removed.

[0181] According to various embodiments, the electronic device (200) can obtain a second image from which an area excluding at least one object is removed from each of a plurality of image frames included in the first image through a method similar to that used to generate the first corrected image frame (650).

[0182] For example, the electronic device (200) may obtain masks (720) for each of the first image frames (710) including a plurality of image frames included in the first image. In addition, the electronic device (200) may obtain second image frames (730) including a plurality of image frames from which an area excluding at least one object is removed from each of the first image frames (710) based on the masks (720).

[0183] According to various embodiments, the electronic device (200) may obtain a second image by performing an encoding operation to include audio information for the first image and / or subtitle information for the first image in the second image frames (730).

[0184]

[0185] FIG. 8 is a flowchart (800) illustrating an operation of an electronic device to obtain user input regarding an outline of at least one object generated using an object recognition model according to various embodiments. FIG. 9 is a flowchart (900) illustrating an operation of an electronic device to generate a corrected image related to at least one object based on user input according to various embodiments.

[0186] Each of the operations described below may be performed in combination with one another. In addition, among the operations described below, operations by an electronic device (200) (e.g., electronic device (106) of FIG. 1) may refer to operations by a processor (210) of the electronic device (200).

[0187] In addition, the “information” described below may be interpreted to mean “data” or “signal,” and “data” may be understood as a concept that includes both analog data and digital data.

[0188] According to various embodiments, the operations illustrated in FIGS. 8 and 9 may be performed in various orders, not limited to the order illustrated. Furthermore, according to various embodiments, more operations may be performed than those illustrated in FIGS. 8 and 9, or at least one operation may be performed less than those illustrated in FIGS.

[0189] Descriptions that overlap or are similar to the above-described descriptions of the operations of the electronic device (200) according to various embodiments may be omitted.

[0190] Referring to FIG. 8, the electronic device (200) can obtain a first image including a plurality of image frames from a user device (e.g., the user device (102) of FIG. 1) in operation 801.

[0191] According to various embodiments, the electronic device (200) may obtain a first point set and a first bounding box set for at least one object included in the first image by using an object recognition model learned to extract a point set and a bounding box set of an object included in the image in operation 803.

[0192] According to various embodiments, the electronic device (200) may identify a first contour of at least one object based on the first point set and the first bounding set in operation 805.

[0193] According to various embodiments, the electronic device (200) may, in operation 807, transmit to the user device to display the first point set and the first bounding box set. For example, the user device may display a visual object for each of the first point set and the first bounding box set by overlapping each of the plurality of image frames included in the first image through the display device.

[0194] According to various embodiments, the electronic device (200) may, in operation 809, obtain a first user input for at least one object included in a first image frame among a plurality of image frames from the user device. For example, the electronic device (200) may obtain a user input for at least one object on a screen on which the first point set and the first bounding box set are displayed.

[0195] According to various embodiments, the user input may include various inputs related to the at least one object of the first image frame. For example, the user input may include at least one of an input for selecting the at least one object, an input for a point set of the at least one object, an input for a bounding box of the at least one object, a masking input for a point set of the at least one object, and a text input related to selecting an outline of the at least one object.

[0196] Referring to FIG. 9, the electronic device (200) may identify a second outline of at least one object in a first image frame based on a first user input in operation 901. For example, the processor (210) may identify an outline of at least one object in a first image frame among a plurality of image frames of the first video as the second outline based on the user input.

[0197] According to various embodiments, the electronic device (200) may obtain a mask for segmenting at least one object in the first image based on the first contour and the second contour in operation 903. For example, the mask may be obtained based on a second contour for at least one object in the first image frame among a plurality of image frames included in the first image and a first contour for at least one object in the remaining image frames.

[0198] According to various embodiments, the electronic device (200) may, in operation 905, obtain a second image by correcting the first image with respect to at least one object using a mask. For example, the electronic device (200) may obtain a second image by correcting the first image with respect to at least one object for each of a plurality of image frames of the first image using a mask obtained based on the first contour and the second contour.

[0199] According to various embodiments, the electronic device (200) may transmit the second image to the user device so that the user device outputs the second image at operation 907. For example, the electronic device (200) may transmit the second image to the user device (102) via the communication device (230).

[0200]

[0201] FIG. 10 is a diagram for explaining an image corrected through a mask generated by an electronic device based on user input according to various embodiments.

[0202] Referring to FIG. 10, an electronic device (200) (e.g., the electronic device (106) of FIG. 1) may obtain an original image (1010) (e.g., the first image of FIG. 8) including a plurality of image frames from a user device (102) of FIG. 1. According to various embodiments, the original image (1010) may be composed of a plurality of image frames including a person (e.g., at least one object of FIG. 2).

[0203] According to various embodiments, the electronic device (200) may obtain a first point set and a first bounding box set for at least one object by using an object recognition model (e.g., the object recognition model of FIG. 3) trained to extract a point set and a bounding box set of an object included in an original image (1010).

[0204] According to various embodiments, the electronic device (200) can identify the outline of a person included in each of a plurality of image frames in the original image (1010) based on the first point set and the first bounding box set.

[0205] According to various embodiments, the electronic device (200) may obtain mask(s) for segmenting a person from each of a plurality of image frames within the original image (1010) based on the outline of the person identified from the original image (1010). For example, the electronic device (200) may obtain a first mask set (1020) for segmenting a person from each of a plurality of image frames of the original image (1010).

[0206] According to various embodiments, the electronic device (200) may transmit to the user device (102) the first point set and / or the first bounding box set generated using the object recognition model to be displayed. For example, the user device (102) may display a visual object for each of the first point set and / or the first bounding box set by overlapping each of the plurality of image frames included in the original image (1010) through a display device. For example, an image (1030) in which the visual object is overlapped may be displayed for each of the plurality of image frames of the original image (1010).

[0207] Meanwhile, according to various embodiments, there may be cases where some modifications are required in the outline of a person identified using an object recognition model. For example, if there is a difference between the actual outline of a person in a first image frame and the outline acquired through an object recognition model among the outlines of a person in each of multiple image frames of the original image (1010), the outline of the person in the first image frame may require modification. In addition, modifications may be required for the first mask (1021) generated based on the outline requiring modification.

[0208] According to various embodiments, a user may determine that an outline correction of a person in a first image frame is necessary through an image (1030) overlapping the visual object for each of a plurality of image frames. In this case, the user may perform a user input operation for the person included in a first image frame (1031) among the plurality of image frames of the image (1030) overlapping the visual object. For example, the user device (102) may obtain a bounding box setting input for the person in the first image frame (1031). Accordingly, the user device (102) may display an image in which a bounding box (1032) set by the user overlaps the first image frame (1031) through a display device.

[0209] According to various embodiments, the electronic device (200) may identify a modified outline of at least one object in a first image frame based on a user input. For example, the electronic device (200) may identify a modified outline of a person in the first image frame based on the bounding box setting input.

[0210] According to various embodiments, the electronic device (200) may obtain a mask for segmenting a person from a first image frame based on a correction outline. In addition, the electronic device (200) may obtain a correction mask set based on a correction outline for a person in the first image frame among a plurality of image frames included in the original image (1010) and an outline for at least one object in the remaining image frames.

[0211] According to various embodiments, the electronic device (200) may obtain a corrected image (1040) by correcting the original image (1010) with respect to a person using a correction mask set. For example, the electronic device (200) may obtain a corrected image (1020) by correcting the original image (1010) with respect to a person for each of a plurality of image frames of the original image (1010) using the correction mask. For example, the electronic device (200) may obtain a corrected image (1040) by removing a background excluding a person from the original image (1010).

[0212] According to various embodiments, the electronic device (200) can transmit the corrected image (1040) to the user device (102) so that the user device (102) outputs the corrected image (1040). Accordingly, the user device (102) can display the corrected image (1040) through a display device.

[0213] According to various embodiments, the electronic device (200) can identify at least one outline with greater accuracy by utilizing user input on the outline of at least one object obtained using an object recognition model. Accordingly, an image that more accurately reflects the user's intent can be obtained with respect to the original image.

[0214]

[0215] FIG. 11 is a diagram for explaining an image corrected through a mask generated by an electronic device based on user input according to various embodiments.

[0216] According to various embodiments, the user input for at least one object in the image frame may include various inputs. For example, the input may include at least one of an input for selecting the at least one object, an input for a set of points of the at least one object, an input for a bounding box of the at least one object, a masking input for the set of points of the at least one object, and a text input related to selecting an outline of the at least one object.

[0217] Referring to FIG. 11, the user input is illustrated to select at least one object from among a plurality of objects included within an image frame.

[0218] According to various embodiments, an electronic device (200) (e.g., the electronic device (106) of FIG. 1) may obtain a first image including a plurality of image frames from a user device (102) of FIG. 1, and may obtain a first set of points and a first set of bounding boxes for at least one object by using an object recognition model (e.g., the object recognition model of FIG. 3) trained to extract a set of points and a set of bounding boxes for at least one object included in the first image. For example, the electronic device (200) may obtain a first image including a plurality of people, and may obtain a first set of points and a first set of bounding boxes for the plurality of people included in the first image.

[0219] According to various embodiments, the electronic device (200) may identify the outlines of a plurality of people included in each of a plurality of image frames based on a first set of points and a first set of bounding boxes. For example, the electronic device (200) may identify the outlines of each of the plurality of people included in the first image frame (1110).

[0220] According to various embodiments, the electronic device (200) may obtain mask(s) that segment a plurality of people from each of the plurality of image frames based on the outlines of each of the plurality of people identified from the first image. For example, the electronic device (200) may obtain a first mask (1121) that segments a plurality of people included in the first image frame (1110).

[0221] According to various embodiments, the electronic device (200) may transmit to the user device (102) the first point set and / or the first bounding box set generated using the object recognition model to be displayed. For example, the user device (102) may display a visual object for each of the first point set and / or the first bounding box set by overlapping each of the plurality of image frames included in the first image through a display device. For example, the user device (102) may display an image frame (1131) that overlaps a visual object representing the first point set with each of the plurality of people included in the first image frame (1110) through a display device.

[0222] Meanwhile, according to various embodiments, there are cases where some modifications are required in the outline of a person identified using an object recognition model. For example, if a user wishes to acquire an image in which only a specific person is selected (e.g., to remove the remaining area within the original image except for the specific person), the outlines of the other people among the outlines of each of the multiple people in each of the multiple image frames of the first image may need to be modified. Accordingly, modifications may also be required for the mask area (1122) for the second person in the first mask (1121).

[0223] According to various embodiments, when a user wants to obtain an image in which only a specific person (e.g., a first person) is selected from a first image, the user may perform a user input action for a person other than the selection target in at least one image frame among a plurality of image frames in the first image. For example, the user device (102) may obtain a user input for an area (1132) of a second person among a plurality of people including the first person and the second person in the first image frame (1110). For example, the user device (102) may obtain a user input for deactivating a point set for the second person.

[0224] According to various embodiments, the electronic device (200) may identify a modified outline of at least one object in the first image frame (1110) based on a user input. For example, the electronic device (200) may obtain a modified outline that retains only the outline of a first person among a plurality of people in the first image frame (1110) based on a user input desiring to release a point set for a second person.

[0225] According to various embodiments, the electronic device (200) may obtain a mask for segmenting a first person from a first image frame (1110) based on a correction outline. For example, the correction mask may be obtained based on a first mask (1121) for the first image frame (1110) and a user input for deselecting a second person. According to various embodiments, the electronic device (200) may obtain a mask set for segmenting only the first person by performing an operation similar to the first image frame (1110) on a mask set for each of a plurality of image frames of the first image.

[0226] According to various embodiments, the electronic device (200) may obtain a corrected image frame (1140) by removing the remaining area except for the first person from among a plurality of people in the first image frame (1110) using a correction mask. The electronic device (200) may obtain a second image (e.g., the second image of FIG. 9) by removing the remaining area except for the first person from the first image using a similar method.

[0227] According to various embodiments, the electronic device (200) can transmit a second image to the user device (102) so that the user device (102) can output the second image. Accordingly, the user device (102) can display the second image through the display device.

[0228]

[0229] FIG. 12 is a drawing showing an execution screen (1200) output through a user device according to various embodiments.

[0230] Referring to FIG. 12, the user device (102) may display an execution screen related to a video editing service based on a control signal obtained through the electronic device (200). The execution screen (1200) of the video editing service displayed through the user device may include a first area (1210), a second area (1220), a third area (1230), a fourth area (1240), a fifth area (1250), and / or a sixth area (1260).

[0231] In this document, icons may be replaced with representations of buttons, menus, objects, etc. In addition, the icons shown in the first area (1210), the second area (1220), the third area (1230), the fourth area (1240), the fifth area (1250), and / or the sixth area (1260) in FIG. 12 are exemplary, and other icons may be placed, or any icon may be replaced with another icon or omitted.

[0232] According to various embodiments, the user device (102) may display an execution screen (1200) based on a control signal of the electronic device (200). According to various embodiments, the execution screen (1200) may include a first area (1210) including icons for executing functions related to video playback among functions related to video editing (e.g., rewind, forward, etc.), menu icons for selecting various functions provided in a video editing service, and a save icon for saving video editing contents executed through the electronic device (200). The electronic device (200) may execute a video editing function based on a user input to the first area (1210) obtained through the user device (102).

[0233] According to various embodiments, the execution screen (1200) may include a second area (1220) related to a user input requesting correction of at least one object in a video among functions related to video editing. For example, the user device (102) may obtain the user input described with reference to FIGS. 8 and 9 through a user input for icons displayed in the second area (1220). For example, the execution screen (1200) may include a second area (1210) in which various icons, such as an icon for executing an image movement and / or image rotation function for an image frame in a video, an icon (1211) related to a bounding box input for at least one object in an image frame, and an icon related to a point input for at least one object, are arranged.

[0234] According to various embodiments, when a user input is obtained for an icon (1211) related to bounding box input, a third area (1230) for executing a function related to drawing a bounding box may be additionally displayed. For example, a menu icon (1231) for selecting the type of at least one object within an image frame (i.e., for selecting a label to enhance object recognition), an icon (1232) for drawing a bounding box in a single image frame as an icon for selecting a user input method, an icon (1233) for drawing a bounding box in a multi-image frame, and an icon (1234) for drawing a bounding box as a prompt input and a segmentation input may be placed in the third area (1230). According to various embodiments, when the icon (1234) is selected, both the output of a bounding box set using an object recognition model and the input of a bounding box through a user input, which have been described with reference to FIGS. 2 to 9, may be utilized.

[0235] According to various embodiments, the execution screen (1200) may include a fourth area (1240) that displays a timeline for a plurality of image frames within an uploaded video and a visual object (1231), a fifth area (1250) as an editing area related to image editing, a sixth area (1260) and / or a seventh area (1270) as an area that displays labels of objects within the image.

[0236]

[0237] FIG. 13a illustrates an execution screen of a user input for a bounding box output through a user device according to various embodiments. FIG. 13b illustrates an execution screen of a user input for a bounding box output through a user device according to various embodiments. FIG. 13c illustrates an execution screen output through a user device as user input for a bounding box is acquired according to various embodiments.

[0238] Below, the execution screens shown can be understood as screens displayed through the user device (102) by the electronic device (200) controlling the user device (102).

[0239] Referring to FIG. 13A, as the first image is acquired through the user device (102), the electronic device (200) can control the user device (102) to display visual objects for a plurality of image frames of the first image in the fourth area (1240) described with reference to FIG. 12, and to display visual objects for image frames selected based on user input in the fifth area (1250).

[0240] According to various embodiments, when a first image frame (1323) is selected from among a plurality of image frames in a first video uploaded through a user device (102) and a user input for an icon (1311) related to bounding box input is obtained, the electronic device (200) may display the first image frame (1323) and pixel information (1321) for the first image frame (1323) in a fifth area (1250). At this time, the user device (102) may additionally display a horizontal axis and / or vertical axis straight line (1322) that may assist bounding box input in the fifth area (1250).

[0241] Referring to FIG. 13B, when a user input related to a bounding box input for a first image frame (1323) is obtained through the user device (102), the electronic device (200) may obtain an input related to setting a bounding box through the user device. For example, when the user wishes to select a bounding box of at least one object within the first image frame (1323) displayed in the fifth region (1250), the user may input the bounding box in the form of a start click and a drag-and-drop. Accordingly, the electronic device (200) may identify the area of ​​the bounding box selected by the user through the (x, y) coordinates of the area where the start click is input and the (x, y) coordinates of the area where the drop event occurs. The bounding box area identified through the user input may be displayed as a visual object (1333) for the bounding box in a form overlapping the first image frame (1323) displayed in the fifth region (1250).

[0242] According to various embodiments, as user input is obtained, the user device (102) may display an icon (1331) indicating that bounding box input is complete and an icon (1332) for further bounding box input or performing the same input in the fifth area (1250).

[0243] According to various embodiments, when a user input for a bounding box is obtained, the user device (102) may display a visual object (1342) representing image frames corresponding to the user input in association with a visual object (1341) representing a plurality of image frames of the first image displayed in the fourth area (1240). For example, the electronic device (200) may control to group the plurality of image frames (1341) and display a visual object (1342) representing image frames of a first image frame group including an image frame corresponding to the user input. In addition, the electronic device (200) may control to display an icon (1343) related to the execution of a function that applies the obtained user input to an image frame group subsequent to the first image frame group in the fourth area (1240).

[0244] Referring to FIG. 13c, an execution screen displayed through the user device (102) is illustrated as user input for the icon (1343) is obtained.

[0245] According to various embodiments, the electronic device (200) may obtain a user input for selecting a bounding box for at least one object within an image frame through the user device (1020), and obtain a user input for an icon (1343) related to executing a function of applying the obtained user input to a subsequent image frame group to the first image frame group.

[0246] According to various embodiments, when a user input for the icon (1343) is obtained, the electronic device (200) may control to apply a bounding box input for at least one object to an image frame group subsequent to the first image frame group and display a visual object (1351) representing the image frame group to which the bounding box input is applied. In addition, the electronic device (200) may control to display an icon (1352) performing a similar function to the icon (1343) in the fourth area (1240).

[0247] According to various embodiments, a user may input a bounding box for at least one object within a particular image frame and apply the input to multiple image frames (e.g., subsequent image frames).

[0248]

[0249] FIG. 14 is a diagram illustrating an electronic device applying user input according to various embodiments.

[0250] Referring to FIG. 14, a visual object representing a plurality of image frames displayed in the fourth area (1240) described with reference to FIG. 12 is illustrated.

[0251] According to various embodiments, the electronic device (200) may display a visual object (1401) representing a plurality of image frames within a first image acquired through the user device (102). In addition, the electronic device (200) may identify a first outline of at least one object using an object recognition model described with reference to FIG. 3 for the plurality of image frames. The first outline acquired for each of the plurality of image frames using the object recognition model may be understood as a base segment (1401).

[0252] According to various embodiments, the electronic device (200) may obtain a user input for selecting a bounding box of at least one object for specific image frame(s) among the plurality of image frames. For example, the electronic device (200) may obtain a user input for selecting a bounding box for image frames within a first image group and a second image group among image groups that group the plurality of image frames. The electronic device (200) may identify a second contour for at least one object in the image frames within the first image group and the second image group based on the user input. The second contour obtained for each of the image frames of the first image group and the second image group may be understood as a first user-selected segment (Bbox Human1) (1402).

[0253] According to various embodiments, the electronic device (200) may obtain a second user-selected segment (Bbox Human2) (1403) in a similar manner to the first user-selected segment (1402). For example, the electronic device (200) may obtain a user input for selecting a bounding box for at least one object in a third image group among a plurality of image frames, and identify a third outline for the at least one object based on the user input.

[0254] According to various embodiments, the electronic device (200) may obtain a final segment (1404) that distinguishes at least one object of the first image based on the first contour, the second contour, and / or the third contour. According to various embodiments, the electronic device (200) may generate a mask that distinguishes at least one object for each of a plurality of image frames in the first image based on the final segment.

[0255]

[0256] FIG. 15 is a diagram illustrating an electronic device generating a mask when a user input for a bounding box is an object addition input according to various embodiments.

[0257] Referring to FIG. 15, a user input for selecting a bounding box as described with reference to FIGS. 13a and 13b is illustrated, wherein the user input selects at least one object among a plurality of objects included within an image frame.

[0258] According to various embodiments, the electronic device (200) may obtain a first image including a plurality of image frames from the user device (102), and may obtain a first point set and a first bounding box set for at least one object by using an object recognition model (e.g., the object recognition model of FIG. 3) trained to extract a point set and a bounding box set of at least one object included in the first image. For example, the electronic device (200) may obtain a first image including a plurality of people, and may obtain a first point set and a first bounding box set for a first person among the plurality of people included in the first image.

[0259] According to various embodiments, the electronic device (200) may identify an outline of at least one object included in each of a plurality of image frames based on a first set of points and a first set of bounding boxes. For example, the electronic device (200) may identify an outline of a first person among a plurality of people included in a first image frame (1510).

[0260] According to various embodiments, the electronic device (200) may obtain mask(s) for segmenting the first person from each of a plurality of image frames based on the outline of the first person identified from the first image. For example, the electronic device (200) may obtain a first mask (1521) for segmenting the first person from among a plurality of people included in the first image frame (1510). Since the outline of the second person from among the plurality of people in the first image frame (1510) is not identified, an area (1522) corresponding to the second person in the first mask (1521) may not be distinguished.

[0261] According to various embodiments, the electronic device (200) may transmit to the user device (102) the first point set and / or the first bounding box set generated using the object recognition model to be displayed. For example, the user device (102) may display a visual object for each of the first point set and / or the first bounding box set by overlapping each of the plurality of image frames included in the first image through a display device. For example, the user device (102) may display an image frame that overlaps a visual object representing a bounding box set for a first person among the plurality of people included in the first image frame (1510) through a display device.

[0262] According to various embodiments, the electronic device (200) may obtain a user input to select a first person as well as a second person among a plurality of people included in a first image frame (1510) through the user device (102). For example, the electronic device (200) may obtain a user input to add a bounding box (1532) for the second person.

[0263] According to various embodiments, the electronic device (200) may identify a modified outline of at least one object in the first image frame (1510) based on a user input. For example, the electronic device (200) may obtain a modified outline including an outline of a second person in addition to an outline of the first person based on a user input desiring to add a second person.

[0264] According to various embodiments, the electronic device (200) can obtain a correction mask (1533) that segments a first person and a second person in a first image frame (1510) based on a correction outline.

[0265] According to various embodiments, the electronic device (200) can obtain a corrected image frame (1540) by removing an area other than the first person and the second person from among a plurality of people in the first image frame (1510) using a correction mask.

[0266]

[0267] FIG. 16a illustrates an execution screen for user input for key points output through a user device according to various embodiments. FIG. 16b illustrates an execution screen for user input for key points output through a user device according to various embodiments. FIG. 16c illustrates an execution screen output through a user device as user input for key points is acquired according to various embodiments.

[0268] Referring to FIG. 16A, as the first image is acquired through the user device (102), the electronic device (200) can control the user device (102) to display visual objects for a plurality of image frames of the first image in the fourth area (1240) described with reference to FIG. 12, and to display visual objects for image frames selected based on user input in the fifth area (1250).

[0269] According to various embodiments, when a user input is obtained for an icon (1610) related to a key-point selection input in relation to a first image uploaded through the user device (102), the electronic device (200) may control to additionally display icons for executing a function related to drawing a key-point. For example, a menu icon (1611) for selecting the type of at least one object within an image frame (i.e., for selecting a label to increase object recognition), an icon (1612) for entering a key-point in a single image frame as an icon for selecting a user input method, an icon (1613) for entering a key-point in a multi-image frame, and an icon (1614) for drawing a key-point with a prompt input and a segmentation input may be arranged. According to various embodiments, when the icon (1614) is selected, both a point set using an object recognition model and a point set through user input, as described with reference to FIGS. 2 to 9, may be utilized.

[0270] Referring to FIG. 16B, when a user input related to a key-point input for a first image frame (1621) is obtained through the user device (102), the electronic device (200) may obtain an input related to a key-point setting through the user device. For example, when the user inputs a key-point of at least one object within the first image frame (1621) displayed in the fifth region (1250), the user may input the key-point in the form of a point click, a point click, and a drag-and-drop. Accordingly, the electronic device (200) may identify the key-point selected by the user through the (x, y) coordinates of the part where the click is input when a new point is input, or the (x, y) coordinates of the part where the click input and drag-and-drop event to move an existing point occur. The key-point identified through the user input may be displayed as a visual object for the key-point in a form overlapping the first image frame (1621) displayed in the fifth region (1250).

[0271] According to various embodiments, as a user input for a key-point is acquired, the user device (102) may display a visual object (1631) representing at least one object identification for a plurality of image frames using an object recognition model in the fourth region (1240), a visual object (1632) for first object recognition according to a key-point user input for a first image frame (1621), and a visual object (1633) for second object recognition. For example, the electronic device (200) may control to group the plurality of image frames and display the visual object (1632) for first object recognition and the visual object (1633) for second object recognition of the first image frame group including the image frame corresponding to the user input. Meanwhile, the user device (102) may display an icon related to the execution of a function that applies the acquired user input described with reference to FIGS. 13B and 13C to an image frame group subsequent to the first image frame group.

[0272] Referring to FIG. 16c, an execution screen displayed through the user device (102) is illustrated as user input for the icon is obtained.

[0273] According to various embodiments, the electronic device (200) may obtain a user input for selecting a key-point for at least one object (e.g., a second object) within an image frame through the user device (1020), and obtain a user input for an icon (1652) related to executing a function of applying the obtained user input to a group of image frames subsequent to the first group of image frames.

[0274] According to various embodiments, when a user input for the icon (1652) is obtained, the electronic device (200) may apply a key-point input for at least one object to an image frame group subsequent to the first image frame group, and display a visual object (1652) for recognition of a second object to which the key-point input has been applied. Meanwhile, since only the key-point input for the second object has been expanded in application, the visual object (1651) for recognition of the first object according to the key-point input for the first object in the first image frame group may be maintained as is.

[0275]

[0276] FIG. 17 is a diagram illustrating an electronic device applying user input according to various embodiments.

[0277] Referring to FIG. 17, a plurality of image frames displayed in the fourth region (1240) described with reference to FIG. 12 and a visual object representing object recognition for the plurality of image frames are illustrated.

[0278] According to various embodiments, the electronic device (200) may display a visual object representing a plurality of image frames within a first image acquired through the user device (102). In addition, the electronic device (200) may identify a first outline for at least one object using an object recognition model described with reference to FIG. 3 for the plurality of image frames. The first outline acquired for each of the plurality of image frames using the object recognition model may be understood as a base segment (1701).

[0279] According to various embodiments, the electronic device (200) may obtain a user input for selecting a point (e.g., a joint, skeleton data, etc.) of at least one object for specific image frame(s) among the plurality of image frames. For example, the electronic device (200) may obtain a user input for selecting a point for a first object and a second object in image frames within a first image group among image groups that group the plurality of image frames. Based on the user input, the electronic device (200) may identify an outline for each of at least one object in the image frames within the first image group (an added group when a point input for a specific object is added). For example, the electronic device (200) may identify a second outline for a first object in the image frames within the first image group and a third outline for a second object in the image frames within the first image group and the second image group. The second contour obtained for each of the image frames of the first image group may be understood as a first user-selected segment (Key Human1) (1702). The third contour obtained for each of the image frames of the first image group and the second image group may be understood as a second user-selected segment (key Human2) (1703).

[0280] According to various embodiments, the electronic device (200) may obtain a final segment (1704) that distinguishes at least one object of the first image based on the first contour, the second contour, and / or the third contour. According to various embodiments, the electronic device (200) may generate a mask that distinguishes at least one object for each of a plurality of image frames in the first image based on the final segment.

[0281]

[0282] FIG. 18 illustrates an execution screen of a user input input for a mask output through a user device according to various embodiments.

[0283] Referring to FIG. 18, as the first image is acquired through the user device (102), the electronic device (200) can control the user device (102) to display visual objects for a plurality of image frames of the first image in the fourth area (1240) described with reference to FIG. 12, and to display visual objects for image frames selected based on user input in the fifth area (1250).

[0284] According to various embodiments, the electronic device (200) may display a visual object representing a plurality of image frames within a first image acquired through the user device (102). In addition, the electronic device (200) may identify a first outline of at least one object using an object recognition model described with reference to FIG. 3 for the plurality of image frames. The first outline acquired for each of the plurality of image frames using the object recognition model may be understood as a base segment (1802).

[0285] According to various embodiments, the electronic device (200) may obtain a user input (hereinafter, referred to as a mask input) for directly inputting a mask for at least one object for a specific image frame(s) among the plurality of image frames. For example, the electronic device (200) may obtain a mask input for at least one object in a first image frame (1801) among the plurality of image frames.

[0286] According to various embodiments, the electronic device (200) may obtain a mask input for the first image frame (1801) through the user device (102). For example, the mask input may be obtained as an input for directly selecting an area of ​​at least one object within the first image frame (1801). The electronic device (200) may identify an area of ​​at least one object input by the user through the user's area selection input for the mask input.

[0287] According to various embodiments, as the mask input is acquired, the user device (102) may display, in the fourth area (1240), a visual object (1802) representing at least one object identification for a plurality of image frames using an object recognition model, and a visual object (1803) for at least one object recognition according to the mask input for the first image frame (1801).

[0288]

[0289] FIG. 19a illustrates an execution screen of a user input for selecting at least one object among a plurality of objects within an image frame according to various embodiments. FIG. 19b illustrates an execution screen of a text-based user input for selecting at least one object among a plurality of objects within an image frame according to various embodiments. FIG. 19c illustrates an execution screen of a text-based user input for selecting at least one object among a plurality of objects within an image frame according to various embodiments.

[0290] Referring to FIG. 19A, as the first image is acquired through the user device (102), the electronic device (200) can control the user device (102) to display visual objects for a plurality of image frames of the first image in the fourth area (1240) described with reference to FIG. 12, and to display visual objects for image frames selected based on user input in the fifth area (1250).

[0291] According to various embodiments, when a user input is obtained for an icon (1911) related to a key-point selection input in relation to a first image uploaded through a user device (102), the electronic device (200) may control to additionally display icons for executing a function related to drawing a key-point. In addition, according to various embodiments, when a user input is obtained for an icon (1911) related to a key-point selection input, the electronic device (200) may additionally obtain an input for selecting a specific object among a plurality of objects within an image frame.

[0292] For example, the electronic device (200) may obtain an input for selecting a first object (1912) from among a plurality of objects in a first image through the user device (102). According to various embodiments, the input for selecting the first object (1912) may be obtained through various methods. For example, the various methods may include a mouse click, a touch input, a sound input, a character input, a keyboard input, etc.

[0293] According to various embodiments, the electronic device (200) may identify an outline of the first object (1912) based on a user input obtained when a user input for selecting the first object (1912) is received. Accordingly, the user device (102) may display a visual object (Base Segment) (1913) representing at least one object identification for a plurality of image frames using an object recognition model in the fourth area (1240), and a visual object (Point Human1) (1914) for first object recognition according to the user input for the first object (1912).

[0294] According to various embodiments, when a user input for an icon (1911) related to a key-point selection input is obtained, the electronic device (200) may additionally obtain an input for selecting a specific object among a plurality of objects within an image frame.

[0295] For example, when text input is selected as an input for selecting an object in the first image, the electronic device (200) may display a text input icon (1921) for obtaining text input through the user device (102). At this time, the user may input text (1922) for selecting the object as the text input icon (1921) is displayed.

[0296] According to various embodiments, when a text input (1922) for selecting an object is obtained through the user device (102), the electronic device (200) can identify a user intention according to the text (1922) and select at least one object in the first image. For example, when the text input obtained through the user device (102) is “Add everyone,” the electronic device (200) can understand the text and identify all people objects included in the first image.

[0297] According to various embodiments, the electronic device (200) may identify (1931) the outlines of all human objects (first human object, second human object, third human object) included in the first image based on the user input (e.g., “Add all people”). Accordingly, the user device (102) may display a visual object (Base Segment) (1913) indicating at least one object identification for a plurality of image frames using an object recognition model in the fourth region (1240), a visual object (Point Human1) for the first human object recognition according to the user input, a visual object (Point Human2) (1932) for the second human object recognition, and a visual object (Point Human3) (1933) for the third human object recognition.

[0298] According to various embodiments, the electronic device (200) may input selection of at least one object in a first image, outline identification of at least one object, and correction of at least one object through various methods such as bounding box input, point input, text input, sound input, etc. The electronic device (200) may understand the user's intention for the input obtained through the user device (102) and provide a function appropriate to the user's intention.

[0299] As described above, the electronic device includes a communication device, a storage device for storing an object recognition model learned to generate points and a bounding box for at least one object included in an image, and at least one processor, wherein the at least one processor may be configured to obtain a first image including a plurality of first image frames from a user device connected to the electronic device through the communication device, obtain a point set and a bounding box set for at least one object included in the first image using the object recognition model, identify an outline of the at least one object based on the point set and the bounding box set, obtain a mask for segmenting the at least one object based on the outline of the at least one object identified from the first image, obtain a second image in which an area excluding the at least one object of the first image is removed using the mask, and transmit the second image through the communication device so that the user device outputs the second image.

[0300] According to various embodiments, the at least one processor may be configured to group the plurality of first image frames included in the first video into preset reference units to create image frame groups, and to track the at least one object for each of the image frame groups using the object recognition model to obtain the point set and the bounding box set.

[0301] According to various embodiments, the at least one processor may be configured to recognize a scene transition of the first image using the object recognition model, and group the plurality of first image frames based on the scene transition to generate the image frame groups.

[0302] According to various embodiments, the object recognition model may generate the point set by extracting skeleton data of the at least one object included in each of the plurality of first image frames, and generate a bounding box set by generating a bounding box for the at least one object included in each of the plurality of first image frames.

[0303] According to various embodiments, the at least one processor may be configured to obtain a user input for an object to be identified through the user device, and determine the at least one object to be identified through the object recognition model based on the user input.

[0304] According to various embodiments, the at least one processor may be configured to obtain a plurality of masks for segmenting the at least one object based on an outline of the at least one object, and to obtain the mask based on a reliability of the plurality of masks.

[0305] According to various embodiments, the at least one processor may be configured to obtain second image frames from which an area excluding the at least one object is removed using the mask, and to obtain the second image using the second image frames and audio information for the first image.

[0306] According to various embodiments, the at least one object may be a foreground of the first image, and the second image may be an image in which the background except for the foreground is removed from the first image.

[0307] According to various embodiments, the at least one processor may be configured to obtain user input for mask generation from the user device, and obtain the mask based on the user input for mask generation and an outline of the at least one object.

[0308] According to various embodiments, the user input for generating the mask may include at least one of text input for at least one object among the plurality of objects of the first image, point selection for the at least one object, bounding box selection, and masking area selection.

[0309] As described above, the operating method of the electronic device may include an operation of obtaining a first image including a plurality of first image frames from a user device connected to the electronic device, an operation of obtaining a first set of points and a first set of bounding boxes for at least one object included in the first image using an object recognition model learned to generate points and bounding boxes for at least one object included in the image, an operation of identifying the at least one object based on the first set of points and the first set of bounding boxes, an operation of obtaining a mask for segmenting the at least one object of the first image identified from the first image, an operation of obtaining a second image in which an area of ​​the first image excluding the at least one object is removed using the mask, and an operation of transmitting the second image so that the user device outputs the second image.

[0310] According to various embodiments, the operating method of the electronic device may further include an operation of grouping the plurality of first image frames included in the first image into preset reference units to create image frame groups, and an operation of tracking the at least one object for each of the image frame groups using the object recognition model to obtain the point set and the bounding box set.

[0311] According to various embodiments, the object recognition model may generate the point set by extracting skeleton data of the at least one object included in each of the plurality of first image frames, and generate a bounding box set by generating a bounding box for the at least one object included in each of the plurality of first image frames.

[0312] According to various embodiments, the at least one object may be a foreground of the first image, and the second image may be an image in which the background excluding the foreground is removed from the first image.

[0313] According to various embodiments, a non-volatile computer-readable recording medium having recorded thereon a program for an operation method executable by a processor of an electronic device, the operation method may include: obtaining a first image including a plurality of first image frames from a user device connected to the electronic device; obtaining a first set of points and a first set of bounding boxes for at least one object included in the first image using an object recognition model learned to generate points and bounding boxes for at least one object included in the image; identifying the at least one object based on the first set of points and the first set of bounding boxes; obtaining a mask for segmenting the at least one object of the first image identified from the first image; obtaining a second image in which an area of ​​the first image excluding the at least one object is removed using the mask; and transmitting the second image so that the user device outputs the second image.

[0314] As described above, the electronic device may include at least one camera, a display device, a communication device, a storage device for storing an object recognition model learned to generate points and a bounding box for at least one object included in an image, and at least one processor, wherein the at least one processor may be configured to obtain a first image including a plurality of first image frames through the at least one camera, obtain a point set and a bounding box set for at least one object included in the first image using the object recognition model, identify an outline of the at least one object based on the point set and the bounding box set, obtain a mask for segmenting the at least one object based on the outline of the at least one object identified from the first image, obtain a second image in which an area excluding the at least one object of the first image is removed using the mask, and control the display device so that the display device outputs the second image.

[0315]

[0316] In this disclosure, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" may include any one of the items listed together in the corresponding phrase, or all possible combinations thereof.

[0317] Terms such as "first," "second," or "first" or "second" may be used simply to distinguish one component from another and do not qualify the components in any other respect (e.g., importance or order).

[0318] The term "module" used in various embodiments of the present disclosure may include a unit implemented in hardware, software, or firmware. For example, it may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions.

[0319] Various embodiments of the present disclosure may be implemented as software (e.g., a program) including one or more commands stored in a storage device (220) (e.g., built-in memory or external memory) readable by a device (e.g., an electronic device (200)). The storage device (220) may be represented as a storage medium.

[0320] According to one embodiment, the methods according to the various embodiments disclosed in this document may be provided as a computer program product. The computer program product may be traded as a commodity between a seller and a buyer. The computer program product may be distributed in the form of a device-readable storage medium (e.g., a compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) through an application store or directly between two user devices.

[0321] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and placed in other components. According to various embodiments, one or more components or operations of the above-described components may be omitted, or one or more other components or operations may be added. Additionally or alternatively, a plurality of components (e.g., a module or a program) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each component of the plurality of components in a manner identical to or similar to that performed by the corresponding component among the plurality of components prior to the integration.

[0322] According to various embodiments, the operations performed by a module, program, or other component may be performed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be performed in a different order, omitted, or one or more other operations may be added.

Claims

1. In electronic devices, communication device; A storage device for storing an object recognition model trained to generate points and bounding boxes for at least one object included in an image; and Contains at least one processor, At least one processor of the above: Through the communication device, a first image including a plurality of first image frames is obtained from a user device connected to the electronic device, Using the object recognition model, a point set and a bounding box set for at least one object included in the first image are obtained, Identifying the outline of at least one object based on the set of points and the set of bounding boxes, Obtaining a mask for segmenting at least one object based on the outline of the at least one object identified from the first image, Using the mask, a second image is obtained from which an area of ​​the first image excluding at least one object is removed, and An electronic device configured to transmit the second image so that the user device outputs the second image through the communication device.

2. In claim 1, The at least one processor generates image frame groups by grouping the plurality of first image frames included in the first image into preset reference units, An electronic device configured to track at least one object for each group of image frames using the object recognition model to obtain the point set and the bounding box set.

3. In claim 2, The at least one processor recognizes a scene transition of the first image using the object recognition model, An electronic device configured to group the plurality of first image frames based on the scene transition to generate the image frame groups.

4. In claim 1, An electronic device in which the object recognition model generates the point set by extracting skeleton data of the at least one object included in each of the plurality of first image frames, and generates a bounding box set by generating a bounding box for the at least one object included in each of the plurality of first image frames.

5. In claim 4, At least one processor, Obtaining user input for an object to be identified through the user device; An electronic device configured to determine at least one object to be identified through the object recognition model based on the user input.

6. In claim 1, At least one processor, Obtaining a plurality of masks for segmenting the at least one object based on the outline of the at least one object, An electronic device configured to acquire the mask based on the reliability of the plurality of masks.

7. In claim 1, At least one processor, Obtaining second image frames from which an area excluding at least one object is removed using the mask, An electronic device configured to acquire the second image using the second image frames and audio information for the first image.

8. In claim 1, wherein said at least one object is the foreground of said first image, An electronic device in which the second image is an image from which the background except for the foreground is removed from the first image.

9. In claim 1, The at least one processor obtains user input for mask generation from the user device, An electronic device configured to obtain said mask based on user input for said mask generation and an outline of said at least one object.

10. In claim 9, An electronic device wherein the user input for generating the mask comprises at least one of text input for at least one object among a plurality of objects of the first image, point selection for at least one object, bounding box selection, and masking area selection.

11. In the method of operating an electronic device, An operation of obtaining a first image including a plurality of first image frames from a user device connected to the electronic device; An operation of obtaining a first set of points and a first set of bounding boxes for at least one object included in the first image using an object recognition model trained to generate points and bounding boxes for at least one object included in the image; An operation of identifying at least one object based on the first point set and the first bounding box set; An operation of obtaining a mask that segments at least one object of the first image identified from the first image; An operation of obtaining a second image from which an area excluding at least one object of the first image is removed using the mask; and An operating method of an electronic device, comprising an action of transmitting the second image so that the user device outputs the second image.

12. In claim 11, An operation of generating image frame groups by grouping the plurality of first image frames included in the first image into preset standard units; and An operating method of an electronic device further comprising an operation of tracking at least one object for each group of image frames using the object recognition model to obtain the point set and the bounding box set.

13. In claim 11, An operating method of an electronic device in which the object recognition model extracts skeleton data of the at least one object included in each of the plurality of first image frames to generate the point set, and generates a bounding box set by generating a bounding box for the at least one object included in each of the plurality of first image frames.

14. In claim 11, wherein said at least one object is the foreground of said first image, A method of operating an electronic device in which the second image is an image in which the background except for the foreground is removed from the first image.

15. In a non-volatile recording medium readable by a computer, a program of an operation method executable by a processor of an electronic device is recorded, The above method of operation is, An operation of obtaining a first image including a plurality of first image frames from a user device connected to the electronic device; An operation of obtaining a first set of points and a first set of bounding boxes for at least one object included in the first image using an object recognition model trained to generate points and bounding boxes for at least one object included in the image; An operation of identifying at least one object based on the first point set and the first bounding box set; An operation of obtaining a mask that segments at least one object of the first image identified from the first image; An operation of obtaining a second image from which an area excluding at least one object of the first image is removed using the mask; and A recording medium including an operation for transmitting the second image so that the user device outputs the second image.

Images