Method for utilizing clipping data on basis of object recognition, and electronic device thereof

Abstract

An embodiment of the present invention may comprise a display (160), a memory (130), and a processor (120). Instructions stored in the memory may, when the processor is executed, cause the electronic device (101) to: recognize a plurality of objects from an image or video displayed on the display; separate the plurality of recognized objects into one or more objects on the basis of a user input; generate the separated object as clipping data; display the generated clipping data on the display; and store the displayed clipping data in the memory together with metadata of the image or video including the plurality of objects.

Description

Method for utilizing clipping data based on object recognition and electronic device thereof

[0001] Various embodiments of the present invention disclose a method for utilizing clipping data based on object recognition and an electronic device thereof.

[0002] With the advancement of digital technology, various types of electronic devices, such as mobile terminals, personal digital assistants (PDAs), electronic notebooks, smartphones, tablet PCs (personal computers), and wearable devices, are becoming widely used. These electronic devices are constantly undergoing improvements in their hardware and / or software to support and enhance their functionality.

[0003] Meanwhile, users share their daily lives through social networking services (SNS) and communicate with numerous users with similar interests and tastes. Users may want to post images on SNS that express their unique personalities, different from those of other users. Electronic devices provide various methods for extracting desired areas from images or videos to create new images or videos.

[0004] In one embodiment, a method and device for recognizing an object from an image (or video), separating the recognized objects into one or more objects when there are multiple objects, and generating and utilizing the separated objects as clipping data (e.g., stickers) may be disclosed.

[0005] An electronic device according to one embodiment of the present disclosure includes a display (160), a memory (130), and a processor (120), and instructions stored in the memory may cause the electronic device to recognize a plurality of objects from an image or video displayed on the display, separate the recognized plurality of objects into one or more based on a user input, generate the separated objects as clipping data, display the generated clipping data on the display, and store the displayed clipping data together with metadata of the image or video including the plurality of objects in the memory when the processor is executed.

[0006] An operating method of an electronic device (101) according to an embodiment of the present disclosure may include an operation of recognizing a plurality of objects from an image or video displayed on a display (160) of the electronic device, an operation of separating the recognized plurality of objects into one or more based on a user input, an operation of generating the separated objects as clipping data, an operation of displaying the generated clipping data on the display, and an operation of storing the displayed clipping data together with metadata of the image or video including the plurality of objects in a memory (130) of the electronic device.

[0007] According to one embodiment, multiple objects can be recognized at once from an image or video to generate clipping data.

[0008] In one embodiment, when multiple objects are recognized, one or more objects can be created or edited as clipping data.

[0009] According to one embodiment, the usability of clipping data can be increased by categorizing and storing clipping data based on metadata of an image or video.

[0010] FIG. 1 is a block diagram of an electronic device within a network environment according to one embodiment.

[0011] FIG. 2 is a diagram illustrating an example of recognizing an object and generating clipping data in an electronic device according to one embodiment.

[0012] FIG. 3 is a flowchart illustrating an operating method of an electronic device according to one embodiment.

[0013] FIG. 4A is a diagram illustrating an example of recognizing multiple objects in an electronic device according to one embodiment.

[0014] FIG. 4b is a diagram illustrating an example of separating one object into multiple objects in an electronic device according to one embodiment.

[0015] FIG. 5 is a flowchart illustrating a method for generating clipping data of an electronic device according to one embodiment.

[0016] FIG. 6A is a diagram illustrating an example of generating clipping data in an electronic device according to one embodiment.

[0017] FIG. 6b is a diagram illustrating an example of generating clipping data by correcting an object in an electronic device according to one embodiment.

[0018] FIG. 7 is a flowchart illustrating a method of utilizing clipping data of an electronic device according to one embodiment.

[0019] FIGS. 8A and 8B are drawings illustrating an example of inserting clipping data in an electronic device according to one embodiment.

[0020] FIG. 9 is a flowchart illustrating a method for combining clipping data of an electronic device according to one embodiment.

[0021] FIG. 10 is a flowchart illustrating an object editing method of an electronic device according to one embodiment.

[0022] FIGS. 11A to 11C are drawings illustrating an example of editing an object in an electronic device according to one embodiment.

[0023] FIG. 12 is a diagram illustrating an example of generating, editing, and utilizing clipping data from a selected image in an electronic device according to one embodiment.

[0024] FIG. 13 is a diagram illustrating an example of generating, editing, and utilizing clipping data from a selected image in an electronic device according to one embodiment.

[0025] FIGS. 14A and 14B are diagrams illustrating an example of utilizing clipping data in an electronic device according to one embodiment.

[0026] FIGS. 15A and 15B are diagrams illustrating an example of utilizing clipping data in an electronic device according to one embodiment.

[0027] FIG. 1 is a block diagram of an electronic device (101) within a network environment (100) according to various embodiments.

[0028] Referring to FIG. 1, in a network environment (100), an electronic device (101) may communicate with an electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of an electronic device (104) or a server (108) via a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0029] The processor (120) may, for example, execute software (e.g., a program (140)) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) and perform various data processing or operations. According to one embodiment, as at least a part of the data processing or operations, the processor (120) may store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the commands or data stored in the volatile memory (132), and store result data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or an auxiliary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together with the main processor (121). For example, when the electronic device (101) includes the main processor (121) and the auxiliary processor (123), the auxiliary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a given function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0030] The auxiliary processor (123) may control at least a portion of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0031] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).

[0032] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0033] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0034] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0035] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0036] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0037] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0038] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0039] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0040] The haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. According to one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0041] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0042] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0043] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0044] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0045] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0046] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas, for example, by the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device via the at least one selected antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0047] According to various embodiments, the antenna module (197) may form a mmWave antenna module. In one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.

[0048] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0049] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server utilizing machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0050] Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to the embodiments of this document are not limited to the aforementioned devices.

[0051] The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another component (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0052] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0053] Various embodiments of the present document may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0054] According to one embodiment, the method according to various embodiments disclosed in this document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) through an application store (e.g., Play Store™) or directly between two user devices (e.g., smart phones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0055] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and placed in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0056] FIG. 2 is a diagram illustrating an example of recognizing an object and generating clipping data in an electronic device according to one embodiment.

[0057] Referring to FIG. 2, a processor (e.g., the processor (120) of FIG. 1) of an electronic device (e.g., the electronic device (101)) according to an embodiment may display a first user interface (210) including an image or video on a display (e.g., the display module (160) of FIG. 1). The image or video may be stored in a memory (e.g., the memory (130) of FIG. 1) of the electronic device (101), stored in a cloud server (e.g., the server (108) of FIG. 1), downloaded from the server (108), or initiated by the server (108). The first user interface (210) may be an execution screen of a gallery application, an execution screen of a search application that displays an image or video on a web page, or an execution screen of a social network service (SNS) related application. Since the image or video is stored, captured, downloaded, or displayed, the first user interface (210) may be one of the execution screens of another application in addition to the ones mentioned above. Hereinafter, the first user interface (210) will be described as an example in which the image is displayed, but the present invention is not limited by the description. When an image or video is displayed on the display module (160), the processor (120) can recognize an object from the displayed image or video.

[0058] The above object may include all things that can be created as clipping data, such as a person, an object (e.g., a car, a streetlight, a desk), an animal, or a background. For example, the background is a concept that collectively refers to at least some or all things excluding a person, an object, and an animal, such as the sky, the sea, or a mountain. The processor (120) may detect (or receive) a user input (201) on the first user interface (210). For example, the user input (201) may be an input for selecting a point (or area) on an image displayed on the first user interface (210), for example, an input for selecting an object. The processor (120) may also recognize an object from the first user interface (210) based on the user input (201). The processor (120) may recognize an object and display it differently for each recognized object. For example, the processor (120) may display the boundary of the recognized object by distinguishing it for each object.

[0059] The processor (120) may provide (or display) a second user interface (230) based on a user input (201). When multiple objects are recognized, the processor (120) may separate the multiple objects into one or more. Referring to the second user interface (230), the processor (120) may separate the multiple objects into a first object (231), a second object (233), a third object (235), a fourth object (237), and a fifth object (239). The first object (231) to the fourth object (237) may each be an object including one person. The fifth object (239) may be an object including two people.

[0060] According to one embodiment, the processor (120) may analyze the recognized plurality of objects and, based on the analysis results, separate the plurality of objects into one or more. For example, the processor (120) may analyze metadata of an image including the recognized plurality of objects, an image or video stored in the memory (130), and an image or video stored (or registered) in association with the user of the electronic device (101) in the server (108). Based on the analysis results, the processor (120) may identify that the first object (231) is the user of the electronic device (101), the second object (233) is the child of the user, and the first object (231) to the third object (235) are the user's family members. Since the technology for recognizing and analyzing objects included in an image corresponds to the prior art, a detailed description thereof may be omitted.

[0061] The processor (120) may provide (or display) a third user interface (250) that generates clipping data for a separated object. According to one embodiment, the processor (120) may automatically enter the third user interface (250), or may change to the third user interface (250) based on a user input (e.g., menu selection, gesture, etc.). The clipping data may be image or video data including each recognized object, such as, for example, a sticker, an emoticon, or the like. The clipping data may include an image (e.g., a still image) or a video (e.g., a moving image). The processor (120) may generate the clipping data as a moving image based on the preceding / following frames of a video including the recognized object. The third user interface (250) may include first to fifth clipping data (251) generated from an image or video displayed on the display module (160). The clipping data depicted in the drawing may be in the form of a rectangle containing the object, or may be formed along the boundary of the object.

[0062] For example, the first clipping data (251) may correspond to the first object (231), the second clipping data (253) may correspond to the second object (233), and the third clipping data (255) may correspond to the third object (235). The fourth clipping data (257) may correspond to the first object (231) to the third object (235), and the fifth clipping data (259) may correspond to the fifth object (239). The clipping data may correspond (or match) to one object one-to-one, or may include multiple objects. The processor (120) may group adjacent objects (e.g., the first object (231) to the third object (235)) to generate the fourth clipping data (257). Alternatively, the processor (120) may generate fourth clipping data (257) based on metadata (e.g., family) between objects through the object analysis.

[0063] An electronic device according to one embodiment of the present disclosure includes a display (160), a memory (130), and a processor (120), and instructions stored in the memory may cause the electronic device to recognize a plurality of objects from an image or video displayed on the display, separate the recognized plurality of objects into one or more based on a user input, generate the separated objects as clipping data, display the generated clipping data on the display, and store the displayed clipping data together with metadata of the image or video including the plurality of objects in the memory when the processor is executed.

[0064] The instructions may cause the electronic device, when the processor is executed, to detect a second user input for selecting a first object after recognizing the plurality of objects, edit the selected first object based on a third user input, and store the edited first object and the plurality of objects as one object.

[0065] The instructions, when executed by the processor, may cause the electronic device to edit at least one of the size, position, angle, or shape of the first object based on the third user input.

[0066] The instructions, when executed by the processor, may cause the electronic device to merge the edited first object into a designated area of ​​the image or video.

[0067] The instructions, when executed by the processor, may cause the electronic device to adjust at least one of the size, position, angle, or shape of the edited first object when combining the edited first object.

[0068] The instructions, when executed by the processor, may cause the electronic device to reposition the edited first object in the image or the video based on the edited first object and store the repositioned first object in the memory together with metadata of the image or the video.

[0069] The instructions, when executed by the processor, may cause the electronic device to recognize at least one of a person, an object, an animal, or a background from the displayed image or video as an object, and, if the recognized object is one, to divide the recognized object into a plurality of regions of interest and identify the recognized object as a plurality of objects.

[0070] The above instructions, when executed by the processor, may cause the electronic device to group clipping data stored in the memory based on the metadata.

[0071] The instructions may cause the electronic device, when the processor is executed, to detect a first designated input with first clipping data inserted, identify second clipping data within a first group including the first clipping data in response to the first designated input, and change the first clipping data to the second clipping data.

[0072] The instructions may cause the electronic device, when the processor is executed, to detect a second designated input while the first clipping data is inserted, identify third clipping data within a second group including the first clipping data in response to the second designated input, and change the first clipping data to the third clipping data.

[0073] Figure 3 is a flowchart (300) illustrating an operating method of an electronic device according to one embodiment.

[0074] Referring to FIG. 3, in operation 301, a processor (e.g., processor 120 of FIG. 1) of an electronic device (e.g., electronic device 101) according to an embodiment may recognize an object. The processor (120) recognizes an object, and the recognized objects may be plural. The processor (120) may display an image or a video on a display (e.g., display module (160) of FIG. 1). The image or the video may be stored in a memory of the electronic device (101) (e.g., memory (130) of FIG. 1), stored in a cloud server (e.g., server (108) of FIG. 1), downloaded from the server (108), or disclosed in the server (108). The processor (120) may recognize an object from the displayed image or the displayed video. The object may include at least one of a person, an object, an animal, or a background.

[0075] According to one embodiment, although the drawing illustrates that operation 301 is performed first, operation 301 may be performed after operation 303, or may be performed in parallel (or simultaneously) with operation 305. The processor (120) recognizes an object, and if there are multiple recognized objects, it may display them differently for each object. For example, the processor (120) may display the boundary lines of the recognized objects by distinguishing them for each object.

[0076] In operation 303, the processor (120) may detect a user input. The user input may be selecting (e.g., touching) a point (or area) on the displayed image (or video). Alternatively, the user input may be an input for selecting at least one object among objects included in the displayed image. Alternatively, the user input may be a request for editing (e.g., selecting an edit item) from a menu item supported by the electronic device (101) or a request for generating clipping data (e.g., selecting data generation). Alternatively, the processor (120) may automatically (e.g., without user input) generate the clipping data when recognizing the object. In this case, operation 303 may be omitted.

[0077] In operation 305, the processor (120) may divide the object into one or more. For example, the displayed image (or the video) may include one person or two or more people. The processor (120) may determine whether the object included in the displayed image is one (e.g., single) or multiple. If the object included in the displayed image is single, the processor (120) may divide the single object into multiple regions of interest. Even if the object included in the displayed image is multiple, the processor (120) may divide the multiple objects into multiple regions of interest. For example, if the displayed image includes one person (e.g., upper body), the processor (120) may divide the one person into two regions of interest, such as the face or the entire person (e.g., the entire upper body), or three regions of interest, such as the face, the face to the shoulders, or the entire person (e.g., the entire upper body). Alternatively, if the image displayed above includes an animal (e.g., a full body image), the processor (120) may segment the animal into two regions of interest: the face or the full body image, or three regions of interest: the face, the tail or the full body image.

[0078] If there are multiple objects included in the image displayed above, the processor (120) may separate the multiple objects into one or multiple objects. According to one embodiment, the processor (120) may analyze the recognized multiple objects and separate the multiple objects into one or multiple objects based on the analyzed results. For example, the processor (120) may analyze metadata of an image including the recognized multiple objects, an image or video stored in the memory (130), or an image or video stored (or registered) in association with a user of the electronic device (101) in the server (108). Since the technology for recognizing and analyzing objects included in an image corresponds to the prior art, a detailed description thereof may be omitted. For example, the processor (120) may separate the multiple objects into one object, or separate the multiple objects into one object including two or more objects.

[0079] In operation 307, the processor (120) may generate clipping data based on the separated object. The clipping data may be, for example, a sticker, an emoticon, or an emoticon. The clipping data may include an image (e.g., a still image) or a video (e.g., a moving image). The processor (120) may generate the clipping data as a moving image based on the preceding / following frames of the image including the recognized object. For example, the processor (120) may generate one object as one clipping data, or multiple objects as one clipping data. Alternatively, the processor (120) may generate multiple regions of interest as clipping data. When the processor (120) recognizes the object, the processor (120) may automatically (e.g., without user input) generate the clipping data.

[0080] In operation 309, the processor (120) can display the generated clipping data. The processor (120) can display only the generated clipping data on the display module (160). The processor (120) can display the generated clipping data together with the image or the video on the display module (160).

[0081] In operation 311, the processor (120) may store the clipping data together with metadata. The metadata may refer to metadata of an image (or video) including the plurality of recognized objects. The metadata may include at least one of the creation (or shooting, downloading, or editing) date, time, and location of the image. Alternatively, the processor (120) may utilize data (e.g., family, pets, or travel) acquired through object analysis or image analysis as the metadata. According to one embodiment, the processor (120) may categorize (classify) the clipping data based on the metadata and store the clipping data in the memory (130). For example, the processor (120) may categorize the first clipping data into a user of the electronic device (101), a photo of the user's upper body, or the user's first overseas trip and store the same in the memory (130). Alternatively, the processor (120) may categorize the second clipping data into animals, pets of the user of the electronic device (101), or full photos and store them in the memory (130).

[0082] FIG. 4A is a diagram illustrating an example of recognizing multiple objects in an electronic device according to one embodiment.

[0083] Referring to FIG. 4A, a processor (e.g., processor 120 of FIG. 1) of an electronic device (e.g., electronic device 101) according to an embodiment may display a first user interface (410) including an image or video on a display (e.g., display module (160) of FIG. 1). The image or video may be stored in a memory of the electronic device (101) (e.g., memory 130 of FIG. 1), stored in a cloud server (e.g., server 108 of FIG. 1), downloaded from the server (108), or disclosed in the server (108). The first user interface (410) may include an image (or video) including a plurality of people. The processor (120) may detect (or receive) a user input (401) while displaying the first user interface (410). The user input (401) may be selecting (e.g., touching) a point on the displayed image (or video). Alternatively, the user input (401) may be an input for selecting at least one object (e.g., a child) among the objects included in the displayed image. Alternatively, the user input (401) may include an input for selecting a menu item, such as editing, saving, or canceling the image or video.

[0084] The processor (120) can recognize multiple objects from the displayed image based on the user input (401). For example, the processor (120) can recognize multiple objects, each including one, two, three, five, or six people, from the first user interface (410). The multiple people can be multiple objects. The processor (120) can separate the multiple objects into one or multiple objects. For example, the processor (120) can separate the multiple objects into a first object (435), such as the second user interface (430). Alternatively, the processor (120) can separate the second object (453) from the multiple objects, such as the third user interface (450). Alternatively, the processor (120) can provide the second user interface (430) according to a user input selecting the first object (435). The processor (120) may also provide a third user interface (450) according to a user input selecting a second object (453).

[0085] The processor (120) may display the remaining objects other than the selected object in a visually distinct manner. For example, the processor (120) may display the remaining objects more dimly, dim or blur them, or change the transparency to be higher (or lower) than the selected object.

[0086] According to one embodiment, the processor (120) may deselect a selected object if a user input for re-selecting the selected object is detected. In addition, the processor (120) may select the first object (435) and the second object (453) together if a user input for selecting the second object (453) is detected within a threshold time while the first object (435) is selected.

[0087] FIG. 4b is a diagram illustrating an example of separating one object into multiple objects in an electronic device according to one embodiment.

[0088] Referring to FIG. 4B, the processor (120) may display a fourth user interface (460) including an image or video on the display module (160). The fourth user interface (460) may include a cat image (or video). The processor (120) may detect a user input (461) while displaying the fourth user interface (460). The user input (461) may be selecting (e.g., touching) a point on the displayed image (or video). The processor (120) may recognize an object from the image based on the user input (461) and determine whether the recognized object is a single object or multiple objects. The processor (120) may divide the recognized object into multiple regions of interest. The processor (120) may divide a single object into multiple regions of interest, or, in the case of multiple objects, may divide one object among the multiple objects into multiple regions. Alternatively, user input for selecting one object may be a touch input, and user input for selecting multiple areas may be a long press input.

[0089] For example, the processor (120) can distinguish regions of interest in various ways. A first way may be to distinguish regions of interest such that a large region includes a small region, such as face -> upper body including face -> whole body, for a single object including the entire body of a person. A second way may be to prevent regions of interest from overlapping each other, such as face -> hand -> leg. The processor (120) may distinguish regions of interest in a third way that is a combination of the first and second ways.

[0090] The method for specifying a region of interest can be such that the user can adjust the region of interest (e.g., a rectangular region) containing the object, or the region of interest containing the object can be automatically changed based on user input. For example, if a long press is performed on a region of interest containing a face, the region of interest changes from face -> upper body -> full body at regular intervals, and the region of interest containing the object at the time of hand release can be selected. Alternatively, the processor (120) can select multiple regions as a method for specifying a region of interest.

[0091] Multiple clipping data of the same object selected in the above manner are grouped and stored, and when clipping data is loaded, it can be changed from one clipping data to another according to user input.

[0092] For example, the processor (120) may distinguish a face region (471) from a single object (e.g., a cat), such as the fifth user interface (470). Alternatively, the processor (120) may distinguish a paw region (481) from a single object (e.g., a cat), such as the sixth user interface (480). That is, the processor (120) may distinguish a face region (471), a paw region (481), or an entire cat (463) from a single object (e.g., a cat). The processor (120) may provide a seventh user interface (490) based on a user input (461). The seventh user interface (490) may provide menu items such as copy, share, edit, and create data for the displayed image. When the processor (120) receives a user selection of an item (491) related to data generation on the seventh user interface (490), the processor (120) can provide an eighth user interface (495) that generates a cat image as clip data.

[0093] The processor (120) may generate first clipping data (497) containing only the cat's face, or may generate second clipping data (498) containing the entire cat's appearance. According to one embodiment, when generating clipping data, the processor (120) may adjust at least one of the object's size, position, angle, or shape. Alternatively, the processor (120) may change the object's expression or pose, or may regenerate the clipped portion of the object through out-painting to generate clipping data.

[0094] According to one embodiment, the processor (120) may analyze images or videos stored in the memory (130) or stored in the same category in the server (108) to generate clipping data. For example, the processor (120) may analyze images or videos stored in the memory (130) or the server (108) to automatically separate cat images and generate clipping data by analyzing objects similar to cat objects.

[0095] FIG. 5 is a flowchart (500) illustrating a method for generating clipping data of an electronic device according to one embodiment. It may be a more detailed version of the flowchart of FIG. 3.

[0096] Referring to FIG. 5, in operation 501, a processor (e.g., processor 120 of FIG. 1) of an electronic device (e.g., electronic device 101) according to an embodiment may recognize an object. The processor (120) may recognize an object from an image or video displayed on a display (e.g., display module (160) of FIG. 1). The image or video may be stored in a memory of the electronic device (101) (e.g., memory (130) of FIG. 1), stored in a cloud server (e.g., server (108) of FIG. 1), downloaded from the server (108), or disclosed in the server (108). The object may include at least one of a person, an object, an animal, or a background. The processor (120) can recognize an object from an image (or video) based on user input (e.g., an edit request, an object selection), or can recognize an object from an image displayed on the display module (160) regardless of user input.

[0097] In operation 503, the processor (120) can determine whether the recognized object is a single object. The recognized object may be any one of a person, an object, an animal, or a background, or at least one of a person, an object, an animal, or a background. That is, the recognized object may include a person and an object, a single person, multiple persons, or a person and an animal. If the recognized object is a single object, the processor (120) can perform operation 505, and if the recognized object is not a single object (e.g., multiple objects), the processor (120) can perform operation 511.

[0098] If the recognized object is a single object, in operation 505, the processor (120) may divide the single object into multiple regions of interest. For example, if the displayed image includes a person (e.g., upper body), the processor (120) may divide the person into two regions of interest, the face or the entire body (e.g., the entire upper body), or three regions of interest, the face, the face to the shoulder, or the entire body (e.g., the entire upper body). Alternatively, if the displayed image includes an animal (e.g., the entire body), the processor (120) may divide the animal into two regions of interest, the face or the entire body, or three regions of interest, the face, the tail, or the entire body.

[0099] In operation 507, the processor (120) may generate multiple distinct regions of interest as clipping data. The clipping data may be, for example, stickers, emojis, or emoticons. The clipping data may include an image (e.g., a still image) or a video (e.g., a moving image). For example, the processor (120) may generate first clipping data that includes only the face of a person, and second clipping data that includes the upper body of the person. Alternatively, the processor (120) may generate first clipping data that includes only the face of an animal, second clipping data that includes only the tail of the animal, and third clipping data that includes the entire image of the animal.

[0100] In operation 509, the processor (120) may categorize and store the clipping data. The processor (120) may perform operation 509 after performing operation 507 or operation 513. The processor (120) may categorize (classify) the clipping data based on metadata and store the clipping data in the memory (130). The metadata may refer to metadata of an image (or video) including the recognized plurality of objects. The metadata may include at least one of the creation (or shooting, downloading, or editing) date, time, and location of the image. Alternatively, the processor (120) may utilize data acquired through object analysis or image analysis (e.g., friends, domestic travel, or overseas travel) as the metadata. The processor (120) may store the generated clipping data by classifying it as any one of a cat, a dog, and a pig. Alternatively, the processor (120) may store the generated clipping data by classifying it as a user, a family member, or a lover. Alternatively, the processor (120) may store the generated clipping data by dividing it into a first location (e.g., New York Times Square), a second location (e.g., New York Chinatown), or a third location (e.g., New York Soho Street).

[0101] If the recognized object is not a single object (e.g., multiple), in operation 511, the processor (120) may separate the multiple objects into one or more. According to one embodiment, the processor (120) may analyze the recognized multiple objects and separate the multiple objects into one or more based on the analyzed results. For example, the processor (120) may analyze metadata of an image including the recognized multiple objects, an image or video stored in the memory (130), or an image or video stored (or registered) in association with a user of the electronic device (101) in the server (108). For example, the processor (120) may separate the multiple objects into one object, or separate the multiple objects into one object including two or more objects.

[0102] In operation 513, the processor (120) may generate clipping data based on the separated object. For example, the processor (120) may generate one object as one clipping data, or multiple objects as one clipping data. Alternatively, the processor (120) may generate multiple regions of interest as clipping data.

[0103] FIG. 6A is a diagram illustrating an example of generating clipping data in an electronic device according to one embodiment.

[0104] Referring to FIG. 6A, a processor (e.g., processor (120) of FIG. 1) of an electronic device (e.g., electronic device (101)) according to an embodiment may extract metadata of an image (or video) (600) displayed on a display (e.g., display module (160) of FIG. 1) when generating clipping data. The metadata may include at least one of a creation (or shooting, download, or editing) date, time, or location of the image. Alternatively, the processor (120) may utilize data obtained through object analysis or image analysis (e.g., similar time, same place, same person) as the metadata. The processor (120) may analyze the image (600), recognize an object from the image (600), and generate first clipping data (601), second clipping data (603), third clipping data (605), fourth clipping data (607), fifth clipping data (610), or sixth clipping data (630) based on metadata of the image (600). The first clipping data (601) to the third clipping data (605) may be a single object, and the fourth clipping data (607) to the sixth clipping data (630) may include multiple objects. The processor (120) may group adjacent objects to generate the fourth clipping data (257) to the sixth clipping data (630).

[0105] FIG. 6b is a diagram illustrating an example of generating clipping data by correcting an object in an electronic device according to one embodiment.

[0106] Referring to FIG. 6B, when generating clipping data, the processor (120) can determine whether there is a portion that is covered (or cut off) by another object, and correct (or create) the covered portion. For example, the processor (120) can create a cut-off area (655) of the object (650) to generate clipping data. The processor (120) can categorize only the clipping data generated by out-painting and provide it to the user interface (670). According to one embodiment, when generating clipping data, the processor (120) can adjust at least one of the size, position, angle, or shape of the object. Alternatively, the processor (120) can change the expression or pose of the object, and generate a cut-off portion of the object through out-painting to generate clipping data.

[0107] FIG. 7 is a flowchart (700) illustrating a method of utilizing clipping data of an electronic device according to one embodiment.

[0108] Referring to FIG. 7, in operation 701, a processor (e.g., processor (120) of FIG. 1) of an electronic device (e.g., electronic device (101)) according to an embodiment may group clipping data. Grouping may mean that clipping data having identical or similar metadata are collected. For example, the processor (120) may group clipping data acquired at the same (or similar) location (e.g., California), group clipping data corresponding to the same person, or group clipping data corresponding to the upper body (or face) of the same person. The processor (120) may group and store the clipping data.

[0109] In operation 703, the processor (120) may insert first clipping data. For example, the processor (120) may insert (or add, combine) the first clipping data into an image displayed in an image (or video) editing application. Alternatively, the processor (120) may insert the first clipping data into an image (or video) displayed on a display (e.g., a display module (160) of FIG. 1).

[0110] In operation 705, the processor (120) may determine whether a first designated input is detected. The first designated input may be a user input detected on the inserted first clipping data, and may be a touch input that swipes (or drags, scrolls) in a horizontal direction (e.g., left → right, right → left). If the first designated input is detected, the processor (120) may perform operation 707, and if the first designated input is not detected, the processor (120) may perform operation 711.

[0111] In the drawing, if the first designated input is not detected, it can be explained that the second designated input is detected. For inputs other than the first designated input or the second designated input (e.g., selecting a cancel button, an edit button, or a save button), a function corresponding to the input can be performed. Since inserting clipping data and then saving (or canceling, editing) corresponds to the prior art, a detailed description thereof can be omitted.

[0112] If the first designated input is detected, in operation 707, the processor (120) may identify second clipping data within the first group. For example, the first group may be a group to which the first clipping data belongs, and may correspond to, for example, the same person (e.g., a picture of a child). The second clipping data may be included within the first group to which the first clipping data belongs, but may be different from the first clipping data.

[0113] According to one embodiment, the group identifying the second clipping data may vary depending on the direction of the horizontal swipe. For example, when swiping from left to right, the processor (120) may identify the second clipping data within the first group. For example, when swiping from right to left, the processor (120) may identify the second clipping data within the third group. The third group is different from the first group and may correspond to the upper body of a person. The third group may be a grouping of clipping data corresponding to the upper bodies of different people.

[0114] In operation 709, the processor (120) may change the first clipping data to the second clipping data. The processor (120) may change the user interface into which the first clipping data is inserted to a user interface into which the second clipping data is inserted and display the changed interface. In other words, the processor (120) may provide a user interface into which the second clipping data is inserted instead of the first clipping data.

[0115] If the first designated input is not detected, in operation 711, the processor (120) may identify third clipping data within the second group. If the first designated input is not detected, the second designated input may be detected. The second designated input may be a user input detected on the inserted first clipping data, and may be a touch input that swipes (or drags, scrolls) in a vertical direction (e.g., up → down, down → up). For example, the second group may be a group to which the first clipping data belongs, and may correspond to, for example, the same location. The third clipping data may be included in the second group to which the first clipping data belongs, but may be different from the first clipping data.

[0116] In operation 713, the processor (120) may change the first clipping data to the third clipping data. The processor (120) may change the user interface into which the first clipping data is inserted to a user interface into which the third clipping data is inserted and display the changed interface. In other words, the processor (120) may provide a user interface into which the third clipping data is inserted instead of the first clipping data.

[0117] FIGS. 8A and 8B are drawings illustrating an example of inserting clipping data in an electronic device according to one embodiment.

[0118] Referring to FIG. 8A, a processor (e.g., the processor (120) of FIG. 1) of an electronic device (e.g., the electronic device (101)) according to an embodiment may display a first user interface (810) including an image or video on a display (e.g., the display module (160) of FIG. 1). The processor (120) may receive a user's selection of an image (or video) into which clipping data is to be inserted, and display the selected image as the first user interface (810). The processor (120) may first select clipping data to be inserted, and then receive a selection of an image into which the selected clipping data is to be inserted. The processor (120) may provide a second user interface (820) or a third user interface (830) according to a request for insertion of clipping data. The second user interface (820) may insert clipping data (823) on top of the displayed image (821) (e.g., a layer on top of the image).

[0119] When a touch (825) is detected from a user on the clipping data (823), the processor (120) may insert the clipping data (823) above the displayed image (821), as in the second user interface (820). Alternatively, the third user interface (830) may insert the clipping data (833) below the displayed image (831) (e.g., a layer below the image). When a touch (835) is detected from a user on the displayed image (831), the processor (120) may insert the clipping data (833) below the displayed image (831), as in the third user interface (830).

[0120] According to one embodiment, when inserting the clipping data, the processor (120) may analyze the displayed image (e.g., metadata) and adjust at least one of the size, position, angle, or shape of the clipping data. For example, the processor (120) may adjust the size of the clipping data to be smaller or larger to fit an object included in the displayed image, or place the clipping data on the left or right side of the object. According to one embodiment, the processor (120) may receive a user selection of an area in which to insert the clipping data on the displayed image. Alternatively, the processor (120) may analyze the displayed image without user input and insert the clipping data into a designated area of ​​the displayed image.

[0121] Referring to FIG. 8B, the processor (120) may display a fourth user interface (850) including an image or video on the display module (160). The processor (120) may provide a fifth user interface (870) in response to a request for insertion of clipping data. The fifth user interface (870) may include first clipping data (871). The processor (120) may receive a user selection of an area in which to insert the first clipping data (871) on the displayed image (851). Alternatively, the processor (120) may analyze the displayed image to insert the first clipping data (871).

[0122] The processor (120) may provide a sixth user interface (890) when a first designated input is detected on the first clipping data (871). The first designated input is an input for replacing the first clipping data with other clipping data within the same group as the first clipping data, and for example, the first designated input may include, but is not limited to, a swipe input in the first designated direction. The sixth user interface (890) may include second clipping data (891). The processor (120) may identify the second clipping data (891) within the first group to which the first clipping data (871) belongs, and provide the sixth user interface (890) changed to the second clipping data (891) instead of the first clipping data (871). According to the first designated input, the second clipping data (891) may be a different child photo from the first clipping data (871). For example, when the second designated input is detected, the processor (120) may insert another photo of the same child as the first clipping data (871) as the second clipping data. The second designated input is an input for replacing the first clipping data with another clipping data within the same group as the first clipping data, and for example, the second designated input may include, but is not limited to, a swipe input in a second designated direction. The second designated direction may be different from (e.g., opposite to) the first designated direction.

[0123] FIG. 9 is a flowchart (900) illustrating a method for combining clipping data of an electronic device according to one embodiment.

[0124] Referring to FIG. 9, in operation 901, a processor (e.g., processor (120) of FIG. 1) of an electronic device (e.g., electronic device (101)) according to an embodiment may select a target background and clipping data. The target background may refer to an image or video into which clipping data is to be inserted. The target background may include one or more objects.

[0125] In operation 903, the processor (120) may analyze the target background. The processor (120) may analyze an image (or video) of the target background, or recognize an object included in the target background.

[0126] In operation 905, the processor (120) may determine whether a representative object exists in the target background. The representative object may be a person. Alternatively, the representative object may be set as an object or an animal. If the representative object exists in the target background, the processor (120) may perform operation 907, and if the representative object does not exist in the target background, the processor (120) may perform operation 908.

[0127] If a representative object exists in the target background, in operation 907, the processor (120) may determine that the clipping data is to be placed outside the representative object area. For example, if the representative object is located in the center of the image, the processor (120) may determine that the clipping data is to be placed in an adjacent position to the representative object. Alternatively, the processor (120) may place the clipping data differently depending on the representative object or the selected clipping data. For example, if the representative object is a person and the selected clipping data is an animal, the processor (120) may determine that the selected clipping data is to be placed to the left of the representative object. Alternatively, if the representative object is an object (e.g., a car) and the selected clipping data is a person, the processor (120) may determine that the selected clipping data is to be placed below the representative object. These examples are provided merely to aid understanding of the invention, and the invention is not limited by the examples.

[0128] If there is no representative object in the target background, in operation 908, the processor (120) may determine to place the clipping data at the center of the target background. If the target background is sand, a forest, or the sea, the processor (120) may place the clipping data at the center of the target background.

[0129] At operation 909, the processor (120) may determine whether the arrangement of the clipping data needs to be adjusted. The processor (120) may adjust at least one of the size, position, angle, or shape of the clipping data based on the target background and the selected clipping data. If the arrangement of the clipping data needs to be adjusted, the processor (120) may perform operation 911, and if the arrangement of the clipping data does not need to be adjusted, the processor (120) may perform operation 913.

[0130] When adjusting the arrangement of the above clipping data, in operation 911, the processor (120) may adjust the arrangement of the clipping data to display it on the target background. For example, the processor (120) may adjust the size of the clipping data to be smaller or larger to fit the target background, adjust the arrangement position of the clipping data, or adjust the angle (e.g., angular direction) of the clipping data to look at the representative object.

[0131] If the arrangement of the clipping data does not require adjustment, in operation 913, the processor (120) may insert and display the selected clipping data into the target background. The processor (120) may also adjust the arrangement of the selected clipping data based on user input. The user may confirm the clipping data inserted into the target background and adjust the arrangement of the clipping data.

[0132] FIG. 10 is a flowchart (1000) illustrating a method for editing an object of an electronic device according to one embodiment.

[0133] Referring to FIG. 10, in operation 1001, a processor (e.g., processor 120 of FIG. 1) of an electronic device (e.g., electronic device (101)) according to an embodiment may detect a user input for selecting an object. The processor (120) may recognize an object in an image or video displayed on a display (e.g., display module (160) of FIG. 1), and then detect a user input for selecting at least one of the recognized objects. Alternatively, the processor (120) may recognize an object and display it differently for each recognized object. For example, the processor (120) may display the boundary line of the recognized object by distinguishing it for each object. The processor (120) may detect a user input for requesting object editing through a menu item, or receive a user input for selecting and editing an object.

[0134] In operation 1003, the processor (120) may provide a user interface for editing an object. The processor (120) may execute an application associated with image (or video) editing and display an execution screen of the application including the object.

[0135] In operation 1005, the processor (120) may edit the selected object. The processor (120) may edit the selected object based on user input. The processor (120) may edit at least one of the size, position, angle, or shape of the selected object based on the user input.

[0136] In operation 1007, the processor (120) may combine the edited object with a designated area. The designated area may be selected by the user. Alternatively, the processor (120) may analyze the image or video to which the edited object is to be combined to determine the area to which the edited object is to be combined. For example, the processor (120) may rearrange some objects among a plurality of objects. Based on the edited object, the processor (120) may rearrange the edited object in the image or video to which the edited object is to be combined and store the rearranged object in a memory (e.g., the memory (130) of FIG. 1) together with metadata of the image or video.

[0137] FIGS. 11A to 11C are drawings illustrating an example of editing an object in an electronic device according to one embodiment.

[0138] Referring to FIG. 11A, a processor (e.g., the processor (120) of FIG. 1) of an electronic device (e.g., the electronic device (101)) according to an embodiment may display a first user interface (1110) including an image or video on a display (e.g., the display module (160) of FIG. 1). The processor (120) may detect a user input (1105) of selecting a menu item to see more in the first user interface (1110). The processor (120) may display a second user interface (1120) based on the user input (1105). The second user interface (1120) may further provide a data creation item or an editing item as a menu item (1123) on the displayed image (or video). The processor (120) may detect a user input (1125) of selecting an editing item from the menu item (1123) of the second user interface (1120).

[0139] Referring to FIG. 11B, the processor (120) may provide a third user interface (1130) or a fourth user interface (1140) for editing an object based on a user input (1125). The user may select an object to be edited from among a plurality of objects through the third user interface (1130) or the fourth user interface (1140). The processor (120) may receive a user input (1131) of drawing an object to be edited in the third user interface (1130). The processor (120) may receive a user input (1145) of tapping an object to be edited in the fourth user interface (1140). When an object to be edited is selected, the processor (120) may provide a fifth user interface (1150). The processor (120) may edit the selected object based on the user input through the fifth user interface (1150).

[0140] Referring to FIG. 11C, the processor (120) may receive a user input (1165) of tapping an object to be edited among a plurality of objects, such as the sixth user interface (1160). The processor (120) may receive a user interaction (1185) of moving the position of the selected object, such as the seventh user interface (1180). The processor (120) may provide an eighth user interface (1190) that moves the selected object from left to right according to the user interaction (1195). The processor (120) may edit at least one of the size, position, angle, or shape of the selected object based on the user interaction (1195).

[0141] According to one embodiment, the method of editing an object may vary. The size of an object can be changed by pinch-zooming in / out of a selected object or by using a rectangular area encircled when selecting an object. The angle can be changed by rotating the selected object while multi-touching or by selecting a rotation icon. The position can be changed by touch and drag input, and all of these values ​​can also be changed by entering values ​​in the properties. According to one embodiment, when combining the edited object, the processor (120) can adjust at least one of the size, position, angle, or shape of the edited object. The processor (120) can rearrange the edited object on an image or video to which the edited object is to be combined based on the edited object and store the edited object in a memory (e.g., the memory (130) of FIG. 1) together with metadata of the image or video.

[0142] FIG. 12 is a diagram illustrating an example of generating, editing, and utilizing clipping data from a selected image in an electronic device according to one embodiment.

[0143] Referring to FIG. 12, a processor (e.g., processor (120) of FIG. 1) of an electronic device (e.g., electronic device (101)) according to one embodiment may select an image for generating clipping data (e.g., sticker) in a gallery application, generate clipping data from the selected image, edit the clipping data, and utilize the clipping data (e.g., utilize the clipping data as an emoticon that can be inserted into a message).

[0144] The first user interface (1210) may be an execution screen (e.g., a gallery interface) of a gallery application that selects and displays an image for generating clipping data in the gallery application. The processor (120) may recognize and separate an object (1223) including six people based on a user input (1205) for selecting an object (1219) within the image. The user input (1205) for selecting the object (1219) may include, but is not limited to, touch, multi-touch, or swipe. The processor (120) may execute at least one function related to the object through a menu (1213) displayed adjacent to the recognized object (region of interest). The processor (120) may provide an editing interface for editing the object based on a user input (1203) for selecting an Edit item of the provided menu (1213).

[0145] The second user interface (1250) may be an editing interface and may include editing menus (1251, 1253, 1255). The processor (120) may recognize / separate a selected object (1271) from among objects including multiple people based on a user input (1273) when a manual selection item (Select manually) (1283) is selected, and may insert the selected object (1271) using an additionally provided customized decoration sticker (1260). The processor (120) may automatically recognize the inserted customized decoration sticker and photo sticker and add them as a composite sticker. When the processor (120) receives a user input (1287) selecting an editing completion button, the processor (120) may complete editing and store the final edited clipping data (e.g., a composite photo sticker including a customized decoration sticker).

[0146] According to one embodiment, the processor (120) may reposition the edited object in an image or video to be combined with the edited object based on the edited object and store the edited object in a memory (e.g., memory (130) of FIG. 1) together with metadata of the image or video.

[0147] FIG. 13 is a diagram illustrating an example of generating, editing, and utilizing clipping data from a selected image in an electronic device according to one embodiment.

[0148] Referring to FIG. 13, a processor (e.g., processor (120) of FIG. 1) of an electronic device (e.g., electronic device (101)) according to an embodiment may provide a first user interface (1310) as a screen for utilizing (or sharing) clipping data. The first user interface (1310) may include an execution screen of a message application. By a user input (1311) of selecting a gallery application icon within the first user interface (1310), created / edited clipping data (photo stickers, composite stickers) (1313, 1323, 1321, 1317) may be provided.

[0149] The processor (120) may provide a second user interface (1350) based on a user input (1315) for selecting clipping data (1313) to be shared. The processor (120) may display the selected clipping data (1351) and, upon receiving a user input for sharing the clipping data (1351), share (transmit) the clipping data. The second user interface (1350) may include an execution screen of a message application that transmitted the clipping data (1351).

[0150] FIGS. 14A and 14B are diagrams illustrating an example of utilizing clipping data in an electronic device according to one embodiment.

[0151] Referring to FIG. 14A, a processor (e.g., the processor (120) of FIG. 1) of an electronic device (e.g., the electronic device (101)) according to an embodiment may provide effect items to be applied to clipping data to be stored when storing (or generating) clipping data. A first user interface (1410) may be an image (1401) including an object (1402). The processor (120) may recognize the object (1402) in the image (1401) and detect a user input (1403) for selecting the recognized object (1402). Upon detecting the user input (1403), the processor (120) may provide a second user interface (1430). The second user interface (1430) may include a menu item (1433) for generating (or storing) the object (1402) as clipping data (e.g., a sticker) on the image (1431). The processor (120) may create and store the object (1402) as clipping data when a user input (1435) is detected selecting Save (e.g., Save as Sticker) from a menu item (1433).

[0152] The processor (120) may generate clipping data (1451), and when storing the generated clipping data (1451), may provide at least one effect item applicable to the clipping data. The processor (120) may provide at least one effect item, such as the third user interface (1450). The at least one effect item may include at least one of Original (1454), Effect 1 (e.g., Outline) (1455), Effect 2 (e.g., Rough cut) (1456), Effect 3 (e.g., Vintage) (1457), or Effect 4 (e.g., Cartoon) (1458). The effect items illustrated in the drawing are examples for explaining the invention, and the types of effects are not limited to the examples illustrated in the drawing. When any one effect among at least one effect item is selected in the third user interface (1450) and Save (1453, Done) is selected, the processor (120) may store the clipping data to which the selected effect is applied.

[0153] Referring to FIG. 14B, the processor (120) may provide a clipping data list (1471) including one or more clipping data, such as a fourth user interface (1470), and may select at least one clipping data from the clipping data list (1471). When the processor (120) stores clipping data to which an effect is applied, the clipping data to which the effect is applied may be immediately confirmed in the clipping data list (1471). The fourth user interface (1470) may be an example in which a user input (1473) for selecting at least one clipping data from the clipping data list (1471) is detected. The processor (120) may provide an editing item (1493) (e.g., edit, delete) for editing the clipping data selected from the clipping data list (1491), such as a fifth user interface (1490).

[0154] FIGS. 15A and 15B are diagrams illustrating an example of utilizing clipping data in an electronic device according to one embodiment.

[0155] Referring to FIG. 15A, a processor (e.g., processor 120 of FIG. 1) of an electronic device (e.g., electronic device 101) according to an embodiment may provide a first user interface (1510) including at least one clipping data. The first user interface (1510) may include a clipping data list (1513) including at least one clipping data according to a user request on an execution screen of a message application. The processor (120) may detect a user input (1511) for adding clipping data in the first user interface (1510). The processor (120) may provide a second user interface (1530) in response to the user input (1511). The second user interface (1530) may provide an image list (1531) including at least one image that can be generated as clipping data (e.g., sticker). The processor (120) can detect a user input (1533) selecting one image from the image list (1531).

[0156] Referring to FIG. 15B, the processor (120) may provide a third user interface (1550) in response to a user input (1533). The third user interface (1550) may recognize an object to be clipped for an image selected from a gallery application and automatically generate clipping data (1551) from the recognized object. When a next button (1553) is selected through the third user interface (1550), the processor (120) may provide a fourth user interface (1570). The fourth user interface (1570) may include clipping data (1571) and an image list (1578) with various effects applied to the clipping data (1571). The image list (1578) may include an original image of the clipping data (1571) (e.g., Original), an image with at least one effect of Outline, Rough cut, Vintage, or Cartoon applied to the clipping data (1571).

[0157] An operating method of an electronic device (101) according to an embodiment of the present disclosure may include an operation of recognizing a plurality of objects from an image or video displayed on a display (160) of the electronic device, an operation of separating the recognized plurality of objects into one or more based on a user input, an operation of generating the separated objects as clipping data, an operation of displaying the generated clipping data on the display, and an operation of storing the displayed clipping data together with metadata of the image or video including the plurality of objects in a memory (130) of the electronic device.

[0158] The method may include an operation of detecting a second user input for selecting a first object after recognizing the plurality of objects, an operation of editing the selected first object based on a third user input, and an operation of saving the edited first object and the plurality of objects as one object.

[0159] The above editing action may include an action of editing at least one of the size, position, angle, or shape of the first object based on the third user input.

[0160] The method may further include an action of combining the edited first object to a designated area of ​​the image or video.

[0161] The above combining action may include an action of adjusting at least one of the size, position, angle, or shape of the edited first object when combining the edited first object.

[0162] The method may further include an operation of rearranging the edited first object in the image or the video based on the edited first object and storing the rearranged first object in the memory together with metadata of the image or the video.

[0163] The method may further include an operation of recognizing at least one of a person, an object, an animal, or a background from the displayed image or video as an object, and, if the recognized object is one, an operation of dividing the recognized object into a plurality of regions of interest and identifying the recognized object as a plurality of objects.

[0164] The method may further include an operation of grouping clipping data stored in the memory based on the metadata.

[0165] The method may further include an operation of detecting a first designated input while the first clipping data is inserted, an operation of identifying second clipping data within a first group including the first clipping data in response to the first designated input, and an operation of changing the first clipping data to the second clipping data.

[0166] The method may further include an operation of detecting a second specified input while the first clipping data is inserted, an operation of identifying a third clipping data within a second group including the first clipping data in response to the second specified input, and an operation of changing the first clipping data to the third clipping data.

[0167] The various embodiments of the present invention disclosed in this specification and drawings are merely specific examples presented to facilitate easy explanation of the technical content of the present invention and aid understanding thereof, and are not intended to limit the scope of the present invention. Therefore, the scope of the present invention should be interpreted to include all modifications or variations derived based on the technical concept of the present invention, in addition to the embodiments disclosed herein.

Claims

1. In an electronic device (101), display (160); memory (130); and Contains a processor (120), The instructions stored in the above memory are executed by the electronic device when the processor is executed. Recognize multiple objects from images or videos displayed on the above display, Separate the above recognized multiple objects into one or more based on user input, Create the above separated object as clipping data, Displaying the clipping data generated above on the display, An electronic device that stores the clipping data displayed above in the memory together with metadata of the image or the video including the plurality of objects.

2. In the first paragraph, the instructions, when the processor is executed, the electronic device, After recognizing the above multiple objects, detecting a second user input selecting the first object, Editing the selected first object based on third user input, An electronic device that stores the above-mentioned edited first object and the above-mentioned plurality of objects as a single object.

3. In the second paragraph, the instructions, when the processor is executed, the electronic device, An electronic device that edits at least one of the size, position, angle, or shape of the first object based on the third user input.

4. In the second paragraph, the instructions, when the processor is executed, the electronic device, An electronic device that merges the edited first object into a designated area of ​​the image or video.

5. In the fourth paragraph, the instructions, when the processor is executed, the electronic device, An electronic device that adjusts at least one of the size, position, angle, or shape of the edited first object when combining the edited first object.

6. In the second paragraph, the instructions, when the processor is executed, the electronic device, An electronic device for rearranging the edited first object in the image or the video based on the edited first object and storing the edited first object in the memory together with metadata of the image or the video.

7. In the first paragraph, the instructions, when the processor is executed, the electronic device, Recognize at least one of a person, object, animal or background as an object from the image or video displayed above, An electronic device that, when the above-mentioned recognized object is one, divides the above-mentioned recognized object into multiple regions of interest and identifies it as multiple objects.

8. In the first paragraph, the instructions, when the processor is executed, the electronic device, An electronic device for grouping clipping data stored in the memory based on the above metadata.

9. In the 8th paragraph, the instructions, when the processor is executed, the electronic device, With the first clipping data inserted, the first specified input is detected, In response to the first designated input, identify second clipping data within the first group including the first clipping data; An electronic device that changes the first clipping data into the second clipping data.

10. In the 8th paragraph, the instructions, when the processor is executed, the electronic device, With the first clipping data inserted, the second specified input is detected, In response to the second specified input, identify third clipping data within the second group including the first clipping data; An electronic device that changes the first clipping data into the third clipping data.

11. In the operating method of an electronic device (101), An operation of recognizing a plurality of objects from an image or video displayed on a display (160) of the electronic device; An action of separating the recognized plurality of objects into one or more objects based on user input; An action to create the above separated object as clipping data; An operation of displaying the generated clipping data on the display; and A method comprising the action of storing the clipping data displayed above together with metadata of the image or the video including the plurality of objects in a memory of the electronic device.

12. In paragraph 11, After recognizing the above multiple objects, an action of detecting a second user input selecting a first object; An action of editing the selected first object based on third user input; and A method comprising an action of saving the above-mentioned edited first object and the plurality of objects as a single object.

13. In paragraph 12, the editing operation is: A method comprising an action of editing at least one of the size, position, angle, or shape of the first object based on the third user input.

14. In paragraph 12, A method further comprising an action of combining the edited first object to a designated area of ​​the image or the video.

15. In paragraph 14, the combining operation is: A method comprising an action of adjusting at least one of the size, position, angle, or shape of the edited first object when combining the edited first object.

Images