Apparatus and method for generating sticker by using behavioral competency

Abstract

The present disclosure relates to an apparatus and a method for generating sticker by using behavioral competency, and may comprise: identifying an object in an image; analyzing the object so as to determine at least one movable element included in the object; storing, as metadata associated with the object, information associated with the movable element; generating a sticker including the object; storing, in association with the sticker, the metadata corresponding to the object; when a context is input, generating at least one recommended sticker corresponding to the context by using the context, the sticker and the metadata corresponding to the object included in the sticker; and displaying the recommended sticker on a display.

Description

Device and method for generating stickers using behavioral capabilities

[0001] The present embodiments relate to a device and method for generating stickers using behavioral capabilities.

[0002] As electronic devices' functionality evolves, they are being utilized in a variety of ways. For example, text messaging allows users to exchange simple text messages. However, recent trends have led to the use of stickers (e.g., stickers, images, special characters, emoticons, and icons) in addition to simple text. Users can enjoy the fun of using stickers when sending and receiving text messages.

[0003] Stickers attached to these text messages may be created by content providers (third-party providers). These stickers may then be registered with messaging service servers and provided to users.

[0004] Electronic devices may store basic stickers provided by messaging service servers. Additionally, in addition to the basic stickers, various stickers downloaded by users may be stored.

[0005] Saved stickers can be displayed when entering a separate emoticon input mode while in text message input mode. If a user inputs a selection from among the displayed stickers, the selected sticker can be input. This allows a text message containing the selected sticker to be sent.

[0006] These stickers can be created by users themselves, often by taking photos. However, creating stickers tailored to various scenarios requires considerable effort.

[0007] The above information may be provided as background art to aid in understanding the present disclosure. No claim or determination is made as to whether any of the above is applicable as prior art related to the present disclosure.

[0008] A method for generating a sticker according to one embodiment may include an operation of identifying an object in an image. The method may include an operation of analyzing the object to determine at least one movable element included in the object. The method may include an operation of storing information associated with the movable element as metadata associated with the object. The method may include an operation of analyzing metadata for the object. The method may include an operation of generating a sticker including the object; and an operation of storing the metadata corresponding to the object in association with the sticker.

[0009] An electronic device according to one embodiment may include a display, one or more processors, and a memory storing instructions. When the instructions are individually or collectively executed by the one or more processors, the one or more processors may perform an operation of identifying an object in an image. When the instructions are individually or collectively executed by the one or more processors, the one or more processors may perform an operation of analyzing the object to determine at least one movable element included in the object; and an operation of storing information associated with the movable element as metadata associated with the object. When the instructions are individually or collectively executed by the one or more processors, the one or more processors may perform an operation of generating a sticker including the object. When the above instructions are individually or collectively executed by the one or more processors, the one or more processors may perform an operation of storing the metadata corresponding to the object in association with the sticker.

[0010] An electronic device according to one embodiment may include a display, one or more processors, and a memory storing instructions. When the instructions are individually or collectively executed by the one or more processors, the one or more processors may perform an operation of identifying an object in an image. When the instructions are individually or collectively executed by the one or more processors, the one or more processors may perform an operation of analyzing the object to determine at least one changeable element included in the object. When the instructions are individually or collectively executed by the one or more processors, the one or more processors may perform an operation of storing information associated with the changeable element as metadata associated with the object. When the above instructions are individually or collectively executed by the one or more processors, the one or more processors may perform an operation of generating a sticker including the object. When the above instructions are individually or collectively executed by the one or more processors, the one or more processors may perform an operation of displaying the generated sticker on the display.

[0011] A computer-readable recording medium according to one embodiment stores instructions, which, when executed by one or more processors, can perform an operation of identifying an object in an image. When the instructions are individually or collectively executed by the one or more processors, the one or more processors can perform an operation of analyzing the object to determine at least one movable element included in the object. When the instructions are individually or collectively executed by the one or more processors, the one or more processors can perform an operation of storing information associated with the movable element as metadata associated with the object. When the instructions are individually or collectively executed by the one or more processors, the one or more processors can perform an operation of generating a sticker including the object. When the above instructions are individually or collectively executed by the one or more processors, the one or more processors may perform an operation of storing the metadata corresponding to the object in association with the sticker.

[0012] In connection with the description of the drawings, the same or similar reference numerals may be used for the same or similar components.

[0013] FIG. 1 is a block diagram of an electronic device within a network environment, according to one embodiment.

[0014] FIG. 2 is a block diagram of a system for generating content using generative artificial intelligence according to one embodiment.

[0015] FIG. 3 is a diagram illustrating a configuration of an electronic device that generates a sticker using action capabilities according to one embodiment.

[0016] FIG. 4 is a flowchart illustrating an operation of storing metadata corresponding to a sticker including an object in an electronic device according to one embodiment.

[0017] FIG. 5 is a flowchart illustrating an operation of generating a recommended sticker by considering metadata according to context in an electronic device according to one embodiment.

[0018] FIG. 6 is a diagram illustrating an example of generating multiple recommendation stickers by considering the entire object, metadata, and context in an electronic device according to one embodiment.

[0019] FIG. 7 is a diagram illustrating an example of generating multiple recommendation stickers by considering partial objects, metadata, and context in an electronic device according to one embodiment.

[0020] FIG. 8 is a diagram illustrating an example of generating a plurality of recommended stickers by considering a plurality of objects, metadata, and context in an electronic device according to one embodiment.

[0021] FIG. 9 is a diagram illustrating an example of a result of analyzing metadata of an object in an image in an electronic device according to one embodiment.

[0022] FIG. 10 is a diagram illustrating an example of suggesting a recommendation sticker in which multiple objects interact with each other in an electronic device according to one embodiment.

[0023] FIG. 11 is a diagram illustrating an example of suggesting a recommended sticker when an object is an object in an electronic device according to one embodiment.

[0024] FIG. 12 is a diagram illustrating an example of suggesting a recommended sticker when an object is an object in an electronic device according to one embodiment.

[0025] FIG. 13 is a diagram illustrating an example of suggesting a recommended sticker based on a conversation in an electronic device according to one embodiment.

[0026] FIG. 14 is a diagram illustrating an example of a recommended emoticon and an example of generating a recommended sticker when the character “congratulations” is entered according to one embodiment.

[0027] FIG. 15 is a diagram illustrating an example of a recommended emoticon and an example of a recommended sticker generated when the character “sad” is entered according to one embodiment.

[0028] FIG. 16 is a diagram illustrating an example of suggesting a recommended sticker when an object is a thing in an electronic device according to one embodiment.

[0029] Hereinafter, embodiments are described in detail with reference to the attached drawings. However, the embodiments may be modified in various ways, and the scope of the patent application is not limited or restricted by these embodiments. It should be understood that all modifications, equivalents, or alternatives to the embodiments are included within the scope of the patent application.

[0030] The terms used in this disclosure are described using commonly used terms, taking into account the functions described herein. However, these terms may refer to various other terms depending on the intentions of those skilled in the art, precedents, or the emergence of new technologies. Therefore, the terms used in this disclosure should not be interpreted solely based on their names, but rather based on the meanings of the terms and the overall content of this disclosure.

[0031] The appearances of phrases such as “according to one embodiment” in various places throughout this disclosure are not necessarily all referring to the same embodiment.

[0032] An embodiment of the present disclosure may be represented by functional block configurations and various processing operations. Some or all of these functional blocks may be implemented by various hardware and / or software configurations that perform specific functions. For example, the functional blocks of the present disclosure may be implemented by one or more microprocessors or by circuit configurations for a given function. Furthermore, for example, the functional blocks of the present disclosure may be implemented in various programming or scripting languages. The functional blocks may be implemented by algorithms that execute on one or more processors. Furthermore, the present disclosure may employ background technology for electronic configuration, signal processing, and / or data processing. Terms such as “element” and “configuration” may be used broadly and are not limited to mechanical and physical configurations.

[0033] FIG. 1 is a block diagram of an electronic device within a network environment, according to one embodiment.

[0034] Referring to FIG. 1, in a network environment (100), an electronic device (101) may communicate with an electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of an electronic device (104) or a server (108) via a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0035] The processor (120) may, for example, execute software (e.g., a program (140)) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) and perform various data processing or operations. According to one embodiment, as at least a part of the data processing or operations, the processor (120) may store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the commands or data stored in the volatile memory (132), and store result data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or an auxiliary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together with the main processor (121). For example, when the electronic device (101) includes the main processor (121) and the auxiliary processor (123), the auxiliary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a given function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0036] The processor (120) can control the operations of the electronic device (101) of FIG. 1 by executing commands stored in the memory (130).

[0037] The auxiliary processor (123) may control at least a portion of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0038] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).

[0039] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0040] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0041] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0042] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0043] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0044] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0045] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0046] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0047] The haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. According to one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0048] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0049] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0050] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0051] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0052] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0053] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas, for example, by the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device via the at least one selected antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0054] According to various embodiments, the antenna module (197) may form a mmWave antenna module. In one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.

[0055] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0056] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In one embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server utilizing machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0057]

[0058] According to one embodiment, the fields in which AI technology is applied may be diverse. For example, it may consist of technologies for linguistic understanding, visual understanding, inference / prediction, knowledge representation, and / or motion control. Linguistic understanding refers to technologies for recognizing, applying, and processing human language / text, and may include natural language processing, machine translation, dialogue systems, question-answering, and speech recognition / synthesis. Visual understanding refers to technologies for recognizing and processing objects similar to human vision, and may include object recognition, object tracking, image search, person recognition, scene understanding, spatial understanding, and image enhancement. Inference / prediction refers to technologies for logically inferring and predicting information by judging it, and may include knowledge / probability-based inference, optimized prediction, preference-based planning, and recommendations. Knowledge representation refers to technologies for automatically processing human experience information into knowledge data, and may include knowledge construction (data creation / classification) and knowledge management (data utilization). Motion control refers to technologies for controlling autonomous vehicle driving and robot movements, and may include motion control (navigation, collision, driving), and manipulation control (behavior control).

[0059] FIG. 2 is a block diagram of a system for generating content using generative artificial intelligence according to one embodiment.

[0060] The artificial intelligence system (200) may include a user query / response interface (210), an AI framework (220), a generative AI model (230), a knowledge component (240), and / or an application / service component (250).

[0061] Referring to FIG. 2, a user query / response interface (210) may receive an input. The input may include user input and / or data acquired or generated by an electronic device (e.g., the electronic device (101) of FIG. 1 or the electronic device (300) of FIG. 3). The data may include images, videos, and / or sensor data generated by at least one processor (e.g., at least one processor (120) or processor (310)) of the electronic device (e.g., illuminance data around the electronic device acquired from a sensor or sensor hub (e.g., a coprocessor (123), posture data (or orientation data) of the electronic device, a temperature inside the electronic device (e.g., a temperature of the display module (160) or a temperature of the at least one processor (120)), size information of a display area of ​​the display module (160), and / or an image acquired through an image sensor (e.g., included in a camera module (180)) of the electronic device). For example, the user input may be in the form of natural language, touch data obtained through a touch circuit included in the display module (160) (e.g., used to identify input from a finger and / or stylus), an image, audio, and / or video. In addition, the user query / response interface (210) may also transmit related information related to the input when transmitting the user input. The related information may include various side information at the point of user input. For example, the side information included in the side information may include application information currently being used by the user or location information of the user.Additionally, user input may be a mixture of natural language, images, sounds, and related information as described above. Furthermore, user input may also be non-natural language input, such as selecting a menu.

[0062] The user query / response interface (210) can output results of a generative artificial intelligence system to the user. The output may include results (or result information) generated or acquired by the system (200) based at least in part on the input. The output may be in the form of a natural language-based response or specific content, and may also be provided in the form of an action requested by the user. For example, the output may have a format according to the user settings of the electronic device.

[0063] The AI ​​framework (220) can receive user input. The AI ​​framework (220) can coordinate and control each component necessary to perform the user's intent based on the user input (e.g., the user's query or inquiry).

[0064] The AI ​​framework (220) may include a prompt design component (221), an APIs / plug-in management component (222), and an output modification component (223).

[0065] The prompt design component (221) can receive user input through the user query / response interface (210).

[0066] The prompt design component (221) can be used to generate prompts suitable for inputting user input into a large language model (LLM), a large vision model (LVM), or a large multimodal model (LMM). The prompt design component (221) can be an AI component that uses a machine learning algorithm or a neural network to develop improved prompts over time. The prompt design component (221) can access knowledge repositories (240) containing user preference data, a prompt library, and prompt examples based on user input to generate prompts, and pass the generated prompts to a generative model (e.g., an LLM, an LVM, or an LMM).

[0067] The APIs / Plugins management component (222) may communicate with external information when there is a request for additional information when passing user input as input to a generative model. For example, the APIs / Plugins management component (222) may establish a channel for communicating with the outside of the AI ​​framework (220) via APIs, and may enable access to various data sources (e.g., knowledge repositories (240)) through the established channel. In addition, the APIs / Plugins management component (222) may be used to request another component (e.g., application / service component (250)) to perform feedback (or response) according to the prompt. For example, when the APIs / Plugins management component (222) needs to perform an action that performs the user input as a final result rather than an intermediate result in an application or service, the APIs / Plugins management component (222) may request the application / service component (250) to perform the action via the API. Information obtained from an external source may be used to generate a prompt in the prompt design component (221) along with user input or may be passed as input to a generative AI model (230).

[0068] The output modification component (223) can at least partially tune (or adjust) (or change) the results (e.g., content) obtained (or output) from the generative AI model (230). For example, the output modification component (223) can determine the relevance (e.g., score) between the output (e.g., content) of the generative AI model (230) and the user input. For example, the output modification component (223) can verify whether the content generated through the LLM, LVM, and / or LMM is not irrelevant, does not contain biased information (e.g., selective information), or does not contain harmful information (e.g., violent content or profanity). Additionally, the output modification component (223) can determine the degree of matching (e.g., score) between the output of the generative AI model (230) and the user input (e.g., intent of the user input). The output modification component (223) can determine to what extent the output matches the result desired by the user, and if an additional process is required (e.g., if it is determined that the output of the generative AI model (230) does not correspond to the user input), the output can be modified to correspond to the user input. The output modification component (223) can additionally configure and provide the user with a hint (e.g., a hint for generating a prompt) to avoid unwanted output.

[0069] A generative AI model (230) may refer to an artificial intelligence neural network that generates feedback (e.g., new data) in response to a prompt. For example, the feedback may further include additional data and / or information related to or relative to the prompt. For example, the feedback may include new content related to the prompt. The generative AI model (230) may include an image generation model and / or a language generation model. Representative image generation models include a generative adversarial network (GAN) and a variational autoencoder (VAE), and examples include a diffusion-based generative model that uses a VAE and a transformer structure. The language generation model may be a model trained to output the most statistically appropriate output value based on an input value. Examples include models such as CHAT-GPT 3 and CHAT-GPT 4. There are also LMMs that can recognize various types of data input, such as text, images, and voice, and generate new data corresponding to them.

[0070] In one embodiment, the AI ​​framework (220) and / or the generative AI model (230) may be included within an AI module (e.g., including a processing circuit) within the electronic device. For example, the AI ​​module may be operatively coupled with at least one processor (e.g., at least one processor (120) or processor (310)) of the electronic device. For example, the AI ​​module may be operatively coupled with a sensor hub of the electronic device for one or more sensors within the electronic device.

[0071]

[0072] FIG. 3 is a diagram illustrating a configuration of an electronic device that generates a sticker using action capabilities according to one embodiment.

[0073] Referring to FIG. 3, the electronic device (300) may be configured to include a processor (310), a display unit (320), a memory (330), and an interface unit (340).

[0074] The display unit (320) can display status information (or indicators), numbers and characters, moving pictures and / or still pictures generated during the operation of the electronic device (300). In addition, according to the present disclosure, an image (e.g., a sticker, a recommended sticker) corresponding to a request from a user can be displayed.

[0075] Here, the sticker may be an image corresponding to a designated object separated from an image. At this time, the designated object may be changed into various forms such as a person / animal / object, or a virtual image corresponding to the designated object. The sticker may also be composed of a combination of multiple objects. The sticker may also be composed of a form in which only a portion of the designated object is extracted. The sticker may also be composed of a form consisting of the designated object(s) and a background image.

[0076] The display unit (320) may have a configuration corresponding to the display module (160) of FIG. 1.

[0077] The memory (330) can store an operating system, application programs, and storage data (e.g., phone numbers, SMS messages, compressed image files, and videos) for controlling the overall operation of the electronic device (300). In addition, the memory (330) can store stickers and metadata generated according to the present disclosure.

[0078] The memory (330) may have a configuration corresponding to the memory (130) of FIG. 1.

[0079] The interface unit (340) is a device that provides an interface with a user of the electronic device (300) and can receive user input. In addition, the interface unit (340) can receive context input and / or selection of recommended stickers as user input according to the present disclosure.

[0080] The interface section (340) may have a configuration corresponding to the interface (177) of FIG. 1.

[0081] The processor (310) may include an image analysis unit (311), a sticker management unit (312), a context analysis unit (313), and / or a sticker generation unit (314). The processor (310) may have a configuration corresponding to the processor (120) of FIG. 1.

[0082] The image analysis unit (311) can identify an object in an image and analyze metadata about the object. At this time, the image may be any image stored in the electronic device (300), or an image selected by the user from any image stored in the electronic device (300). The image may be an image corresponding to an object cut out from a source image by the user, or may be an original image including both a background and an object. In addition, the image may include not only a still image but also a moving image. The image analysis unit (311) can identify an object by receiving an object image cut out, but when receiving a source image, it can identify the object by separating the object from the source image.

[0083] The image analysis unit (311) can analyze an object to determine at least one movable element included in the object, and store information related to the movable element as metadata associated with the object.

[0084] The image analysis unit (311) can analyze an object to determine at least one changeable element included in the object, and store information associated with the changeable element as metadata associated with the object.

[0085] At this time, the metadata may include at least one of information about the object, information confirming whether the object includes the entire object or a portion of the object, information indicating a movable part of the object, and information indicating a changeable part of the object. Here, the changeable part may include not only a dynamic part but also a static part (e.g., the color of the object, the pattern of the object, the shading of the object).

[0086] Additionally, metadata can be user-defined, either by adding tags to each changeable element or by specifying the changeable elements on the image. This allows the user to decide whether the changeable elements should be formatted as a photo or cartoon.

[0087] Additionally, metadata can include sound data information related to the object. For example, if the object is an adult female, metadata may include the sounds of her laughing or crying. If the object is a car, metadata may include the sound of its horn or engine. If the object is a tree, metadata may include the sounds of wind or rustling leaves.

[0088] The image analysis unit (311) can extract metadata for the image through a large multimodal model (LMM) / large language model (LLM) (350).

[0089] The image analysis unit (311) can also extract metadata through multimodal large language models (MM-LLMs) in addition to LMM / LLM (350).

[0090] The sticker management unit (312) can create a sticker containing an object.

[0091] When creating a sticker containing an object, the sticker management unit (312) can create the sticker by including the object's background. The object's background may be the background of the object extracted from an image, a preset background, or a background created by the user's selection.

[0092] The sticker management unit (312) can store metadata corresponding to an object in conjunction with a sticker.

[0093] The sticker management unit (312) can create objects by changing the original format of the image (e.g., photo format, cartoon pocket format, landscape format) or a format different from the original format, depending on the changeable element. For example, facial expressions can be created in cartoon format, while torso or limbs can be created in photo format or vice versa. In this case, the metadata can further include format information for each changeable element.

[0094] In order to store metadata corresponding to an object in conjunction with a sticker, the sticker management unit (312) may store metadata corresponding to an object by including the metadata corresponding to the object in the sticker, store metadata by matching it with the sticker using a database, or store metadata corresponding to the sticker by matching it through an external server (e.g., server (108) of FIG. 1).

[0095] Meanwhile, the image analysis unit (311) can extract the object's movement from the video if the image being extracted is a video. Then, the sticker management unit (312) can save it in the form of a GIF (graphics interchange format) or a separate moving sticker (multi-frame sticker).

[0096] The context analysis unit (313) can detect whether context is input. For example, the context may include at least one of a sentence input by the user and / or a preset number of recent sentences used in a conversation. Additionally, the context may further include situational information (e.g., meetings, departures from work, holidays, public holidays, physical information such as the user's heart rate, based on calendar entries) or environmental information (e.g., the user's current location, weather, season, temperature).

[0097] The sticker generation unit (314) can generate at least one recommended sticker corresponding to the context using metadata corresponding to the context, the sticker, and / or the object included in the sticker. For example, the metadata can include at least one of information about the object, information for determining whether the object includes the entire object or a portion of the object, information indicating a movable part of the object, and information indicating a changeable part of the object. Here, the changeable part can include not only a dynamic part but also a static part (e.g., the color of the object, the pattern of the object, the shading of the object).

[0098] When generating a recommended sticker, the sticker generation unit (314) may consider whether the user has used the corresponding sticker and the number of times the sticker has been used. When generating a recommended sticker, the sticker generation unit (314) may make recommendations based on the user's personal information (age, gender) and usage history (conversation content). When generating a recommended sticker, the sticker generation unit (314) may make recommendations based on the location of the electronic device (300) (user location) and the type of the electronic device (300) (e.g., mobile phone, tablet, HMD (head-mounted display)).

[0099] The sticker generation unit (314) can input metadata corresponding to a context, a sticker, and an object included in the sticker into a generative artificial intelligence model to generate at least one recommended sticker corresponding to the context. The generative artificial intelligence model may be a large vision model (LVM) (360) or a model combining two or more of LMM, MM-LLM, and LVM. In this case, the generative artificial intelligence model can analyze metadata corresponding to the context, the sticker, and an object included in the sticker to identify the user's intention, and modify the object included in the sticker to perform an action appropriate for the context to generate a recommended sticker.

[0100] When generating a recommended sticker, the sticker generation unit (314) may generate a recommended sticker using the entire image selected by the user or an object included in the image instead of the sticker.

[0101] The sticker generation unit (314) can generate a dynamic sticker of a moving object as a recommended sticker.

[0102] The sticker generation unit (314) can generate a sticker in which multiple objects interact when the sticker includes multiple objects.

[0103] The sticker generation unit (314) can display recommended stickers on the display unit (320).

[0104] When at least two or more recommended stickers are selected by the user at the same time, the sticker generation unit (314) can generate a new recommended sticker through a generative artificial intelligence model by combining metadata corresponding to each of the two or more selected recommended stickers, and display the generated new recommended sticker on the display.

[0105] For example, if a sticker with a smiling emotion and a sticker with a sad emotion are selected, the sticker generation unit (314) can generate a sticker expressing a happy or sad emotion. Alternatively, if a sticker of a person with a smiling expression and a sticker of a pet with a smiling expression are selected, the sticker generation unit (314) can generate a sticker that displays two or more objects together in a single sticker.

[0106] When displaying a recommended sticker on the display unit (320), the sticker generation unit (314) can output related sound data together if the metadata includes related sound data.

[0107] The sticker generation unit (314) can generate a recommended sticker by considering the size of the display unit (320). For example, if the size of the display unit (320) is small (e.g., a small screen of a terminal with multiple displays), a recommended sticker can be generated using a sticker having metadata of the upper body. On the other hand, if the size of the display unit (320) is large (e.g., a large screen of a terminal with multiple displays), a recommended sticker can be generated using a sticker having metadata of the entire body.

[0108] The processor (310) can control the overall operation of the electronic device (300). For example, the processor (310) can perform the functions of the image analysis unit (311), the sticker management unit (312), the context analysis unit (313), and / or the sticker generation unit (314). The image analysis unit (311), the sticker management unit (312), the context analysis unit (313), and the sticker generation unit (314) are illustrated separately to explain each function separately. Therefore, the processor (310) can include at least one processor configured to perform each function of the image analysis unit (311), the sticker management unit (312), the context analysis unit (313), and / or the sticker generation unit (314). In addition, the processor (310) can include at least one processor configured to perform some of the functions of the image analysis unit (311), the sticker management unit (312), the context analysis unit (313), and the sticker generation unit (314).

[0109] The processor (310) can control the operations of the electronic device (300) of FIG. 3 by executing commands stored in the memory (330).

[0110] Each "processor (310)" in this disclosure may include a processing circuit or may include multiple processors. For example, as used in this disclosure, including in the claims, the term "processor (310)" may include various processing circuits including at least one processor, wherein one or more of the at least one processor may be configured to perform various functions described in this disclosure, individually and / or collectively, in a distributed manner. When "processor," "at least one processor," and "one or more processors" are described in this disclosure as being configured to perform multiple functions, these terms include, but are not limited to, situations where one processor performs some of the recited functions and another processor performs other of the recited functions, and situations where a single processor may perform all of the recited functions. Furthermore, the at least one processor may include a combination of processors that perform various recited / disclosed functions, for example, in a distributed manner. The at least one processor may execute program instructions to achieve or perform various functions.

[0111]

[0112] FIG. 4 is a flowchart illustrating an operation of storing metadata corresponding to a sticker including an object in an electronic device according to one embodiment.

[0113] Referring to FIG. 4, in operation 410, the electronic device (300) can identify an object in an image. At this time, the image may be any image stored in the electronic device (300), or an image selected by the user from any image stored in the electronic device (300). The image may be an image corresponding to an object cropped from a source image by the user, or may be an original image including both a background and an object. In operation 410, the electronic device (300) may receive an image of an object cropped, but when receiving a source image, it can identify the object by separating the object from the source image.

[0114] In operation 420, the electronic device (300) may analyze metadata for an object. More specifically, in operation 420, the electronic device (300) may analyze the object to determine at least one movable element included in the object, and store information associated with the movable element as metadata associated with the object. Alternatively, in operation 420, the electronic device (300) may analyze the object to determine at least one changeable element included in the object, and store information associated with the changeable element as metadata associated with the object. At this time, the metadata may include at least one of information about the object, information for determining whether the metadata includes the entire object or a portion of the object, information indicating a movable portion of the object, and information indicating a changeable portion of the object. Here, the changeable portion may include not only a dynamic portion but also a static portion (e.g., a color of the object, a pattern of the object, a shading of the object).

[0115] In operation 420, the electronic device (300) can extract metadata for an image through a large language model (LLM).

[0116] The electronic device (300) provides an image to the LMM, and can ask the LMM questions such as "Is the image including the entire object? Is the image including a part of the object? And where is the movable part of the object in the image?"

[0117] The electronic device (300) can store the upper body and two arms as metadata when it receives a response from the LMM such as “this is an upper body image and both arms can move.”

[0118] Alternatively, the electronic device (300) can generate a caption for an image through multimodal large language models (MM-LLMs) and extract movable object information through large language models (LLMs).

[0119] For example, MM-LLMs can generate detailed descriptions such as "a child with a full body running figure", and LLMs can generate answers such as "two arms, two legs" through prompts such as "which part of the body can move with the given sentence?" and store the answers as metadata.

[0120] In operation 430, the electronic device (300) can generate a sticker containing an object.

[0121] In operation 430, when the electronic device (300) creates a sticker including an object, the sticker may include the background of the object. The background of the object may be a background of the object extracted from an image, a preset background, or a background generated by a user's selection.

[0122] In operation 430, the electronic device (300) can generate an image by changing the original format (e.g., photo format, cartoon pocket format, landscape format) of the object or a format different from the original format, depending on the changeable elements. For example, facial expressions can be generated in cartoon format, while torso or limbs can be generated in photo format or vice versa. In this case, the metadata can further include format information for each changeable element.

[0123] Additionally, metadata can be set by the user by adding tags to each changeable element or by specifying changeable elements on the image. For example, the electronic device (300) can receive user input and determine whether the changeable element should be formatted as a photo or cartoon.

[0124] In operation 440, the electronic device (300) can store metadata corresponding to an object in conjunction with a sticker.

[0125] In operation 440, the electronic device (300) can store metadata corresponding to an object in a sticker, store metadata by matching it with a sticker using a database, or store metadata corresponding to a sticker by matching it with an external server.

[0126]

[0127] FIG. 5 is a flowchart illustrating an operation of generating a recommended sticker by considering metadata according to context in an electronic device according to one embodiment.

[0128] Referring to FIG. 5, in operation 510, the electronic device (300) may detect whether a context is input. At this time, the context may include at least one of a sentence input by the user and / or a preset number of recent sentences used in the conversation. Additionally, the context may further include situational information (e.g., meetings, leaving work, holidays, public holidays, physical information such as the user's heart rate based on calendar entries) or environmental information (e.g., the user's current location, weather, season, temperature).

[0129] When a context is input in operation 510, in operation 520, the electronic device (300) may generate at least one recommended sticker corresponding to the context using metadata corresponding to the context, the sticker, and / or the object included in the sticker. At this time, the metadata may include at least one of information about the object, information for confirming whether the entire object or a portion of the object is included, information indicating a movable part of the object, and information indicating a changeable part of the object.

[0130] In operation 520, the electronic device (300) may input metadata corresponding to the context, the sticker, and the object included in the sticker into a large vision model (LVM) to generate at least one recommended sticker corresponding to the context. For example, the metadata may be used when generating the sticker, and may be utilized for properties of the sticker generated according to the context (e.g., object movement, object shape, whether there is a background, etc.). For example, the electronic device (300) may include the shape of the object included in the sticker (e.g., the range of motion).

[0131] In operation 520, the electronic device (300) can generate a dynamic sticker with a moving object as a recommended sticker. In this case, the dynamic sticker with a moving object can be generated by modifying the objects contained in the included image frames using metadata to fit a single context and determining the order of the entire image frames. In other words, the image frames contained in the dynamic sticker can be modified to fit a single theme (e.g., metadata) and arranged so as to be naturally connected.

[0132] In operation 520, the electronic device (300) can generate a sticker in which multiple objects interact when the sticker includes multiple objects.

[0133] In operation 520, the electronic device (300) may input only the context without the sticker into a large vision model (LVM) to generate at least one recommended sticker corresponding to the context.

[0134] In operation 530, the electronic device (300) can display recommended stickers on the display. For example, when text is entered in an installed messenger application, the electronic device (300) can display and present at least one recommended sticker on the display. For example, the electronic device (300) can display recommended stickers on the display in real time when a context is entered, without the user having to perform a separate operation to select a sticker.

[0135] In operation 530, when generating a recommended sticker, the electronic device (300) may also generate the background of the object based on the context. In this case, the electronic device (300) may generate the object and background as a single sticker.

[0136]

[0137] In the embodiments of FIGS. 4 and 5, the operations may be performed sequentially, but are not necessarily performed sequentially. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

[0138] According to one embodiment, operations 410 to 440 and operations 510 to 530 may be understood to be performed in a processor (e.g., processor (120) of FIG. 1, processor (310) of FIG. 3) of an electronic device (e.g., electronic device (101) of FIG. 1, electronic device (300) of FIG. 3).

[0139]

[0140] FIG. 6 is a diagram illustrating an example of generating multiple recommendation stickers by considering the entire object, metadata, and context in an electronic device according to one embodiment.

[0141] Referring to FIG. 6, the metadata of the sticker (610) includes information that the entire body can be moved, and the sticker generation unit (314) can generate the recommended stickers (631, 632, 633) by transforming the entire body when generating the recommended stickers (631, 632, 633).

[0142] The sticker generation unit (314) can generate recommended stickers (631, 632, 633) based on a context (620) based on a sticker (610) including metadata about the entire body.

[0143] The sticker generation unit (314) can input metadata about the context (620), the sticker (610), and the body corresponding to the object included in the sticker into a large-scale vision model (LVM) (360) to generate at least one recommended sticker (631, 632, 633) corresponding to the context (620).

[0144] Alternatively, the sticker generation unit (314) may generate a prompt corresponding to the context (620) based on metadata about the entire body, and input the sticker (610) and the prompt into a large-scale vision model (LVM) (360) to generate at least one recommended sticker (631, 632, 633) corresponding to the context (620).

[0145] In one embodiment, when requesting the generation of recommended stickers in a messaging application, the context may be utilized as a conversation context. For example, the context may be a message entered by the user or a preset number of recent sentences used in the conversation. The sticker generation unit (314) may extract keywords related to the message to which the user will respond next from the preset number of recent sentences used in the conversation and utilize them as context. One way to extract keywords related to the message to which the user will respond next from the recent sentences is through AI auto-reply.

[0146] If the content of the context is to express joy, and the movable parts included in the metadata are arms and legs, LVM (360) can generate recommended stickers (631, 632, 633) expressing "joy" by moving the arms and legs in various poses.

[0147]

[0148] FIG. 7 is a diagram illustrating an example of generating multiple recommendation stickers by considering partial objects, metadata, and context in an electronic device according to one embodiment.

[0149] Referring to FIG. 7, the sticker generation unit (314) can generate recommended stickers (731, 732, 733) based on a context (720) based on a sticker (710) including metadata regarding the upper body.

[0150] The sticker generation unit (314) can input metadata about the context (720), the sticker (710), and the upper body corresponding to the object included in the sticker into a large-scale vision model (LVM) (360) to generate at least one recommended sticker (731, 732, 733) corresponding to the context (720).

[0151] Alternatively, the sticker generation unit (314) may generate a prompt corresponding to the context (720) based on metadata about the upper body, and input the sticker (710) and the prompt into a large-scale vision model (LVM) (360) to generate at least one recommended sticker (731, 732, 733) corresponding to the context (720).

[0152] If the content of the context is to express joy, LVM (360) can generate recommended stickers (731, 732, 733) expressing “joy” by moving the arm in various poses, if the movable part included in the metadata is an arm.

[0153]

[0154] FIG. 8 is a diagram illustrating an example of generating a plurality of recommended stickers by considering a plurality of objects, metadata, and context in an electronic device according to one embodiment.

[0155] Referring to FIG. 8, the sticker generation unit (314) can generate recommended stickers (831, 832, 833) based on a context (820) based on a sticker (810) having a plurality of objects and metadata for each object.

[0156] For example, if one object included in a sticker (810) includes an upper body and another object includes a full body, the LVM (360) can generate recommended stickers (831, 832, 833) by moving only the upper body of the object including only the upper body and moving the full body of the object.

[0157] LVM (360) sets main and sub objects for multiple objects included in a sticker (810), and when creating recommended stickers (831, 832, 833), the sizes, positions, and / or colors of the objects (main / sub) can be created differently.

[0158]

[0159] FIG. 9 is a diagram illustrating an example of a result of analyzing metadata of an object in an image in an electronic device according to one embodiment.

[0160] Referring to FIG. 9, when there are multiple objects, the electronic device can extract information about each object as in the example (910) of FIG. 9.

[0161] For example, id is a unique value for each object, type is information indicating whether the object is dynamic or static, coordinate is information indicating the location where the object is displayed within the original image, size is information indicating the size of the object, description is a description of the object, and action meta is information about the parts of the object that can move.

[0162] In the case of example (910) of Fig. 9, it can be confirmed that there are two objects in the image, one object is a full-body running child (920) with metadata for two arms and two legs, and the other object is a brightly smiling upper-body child (930) with metadata for two arms.

[0163]

[0164] FIG. 10 is a diagram illustrating an example of suggesting a recommendation sticker in which multiple objects interact with each other in an electronic device according to one embodiment.

[0165] Referring to FIG. 10, as in the example of 1010, when the character “best” is entered in the message application, the electronic device (300) displays stickers (1011, 1012) to determine whether to create a recommended sticker based on a sticker before suggesting a recommended sticker, so that a sticker that serves as a basis for creating a recommended sticker can be selected.

[0166] As in the example of 1020, the electronic device (300) can generate recommended stickers (1021, 1022) so that the multiple objects can interact when the sticker (1011) selected by the user includes multiple objects.

[0167] The electronic device (300) can verify that each of the plurality of objects is an image corresponding to a full body. The electronic device (300) can analyze (or verify) that the metadata is capable of moving both arms and both legs based on the image corresponding to the full body. When the electronic device (300) generates a recommended sticker using a sticker including the analyzed (or verified) metadata, if the user's sentence "best" is input, the electronic device (300) can output a screen (1010) that outputs recommended stickers (1011, 1012) with "best" as the context.

[0168] The electronic device (300) can generate a recommendation sticker (1011) in which a plurality of objects move both arms and / or both legs to generate stickers, and each of the two objects expresses the content as “the best.”

[0169] Additionally, the electronic device (300) may generate a recommendation sticker (1012) by having multiple objects move both arms and / or both legs to generate a sticker, and the content, "the best," may be an action indicating a form of two objects interacting with each other (e.g., high-five).

[0170] In the preceding example, the object included in the sticker was described as a person, but this is not limited to the case and may be composed of various objects. For example, the object included in the sticker may include an object.

[0171] The electronic device (300) of the present disclosure can extract a designated part as metadata and change the designated part in the case of an object that does not have joints and thus does not have movable metadata, unlike an animal or a human.

[0172] FIG. 11 is a diagram illustrating an example of suggesting a recommended sticker when an object is an object in an electronic device according to one embodiment.

[0173] Referring to FIG. 11, in the case of a sticker having a tree as an object, the electronic device (300) can detect an input of the context "cold" as in the example of 1110. Then, the electronic device (300) can generate a recommended sticker (1111) such as a leafless winter tree with "leaves" as metadata.

[0174] According to one embodiment, the electronic device (300) may detect an input context of "maple leaves" as in the example of 1120. Then, the electronic device (300) may generate a recommendation sticker (1112) such as a tree covered in maple leaves.

[0175]

[0176] Meanwhile, when generating a recommended sticker, the electronic device (300) can reflect different contexts depending on whether the sticker includes a part of an object or the entire object.

[0177] For example, when the electronic device (300) creates a sticker in the form of an entire car, it can create “car, movable object” as metadata.

[0178] In this case, the electronic device (300) can generate a recommendation sticker as shown in FIG. 16.

[0179] FIG. 16 is a diagram illustrating an example of suggesting a recommended sticker when an object is a thing in an electronic device according to one embodiment.

[0180] Referring to FIG. 16, as in the example of 1610, when the context of “running” is input, the electronic device (300) can generate a recommendation sticker (1611) reflecting the movement of a car accelerating and suggest it to the user.

[0181]

[0182] If the sticker contains part of an object, a recommended sticker can also be generated as shown in Figure 12 below.

[0183] FIG. 12 is a diagram illustrating an example of suggesting a recommended sticker when an object is an object in an electronic device according to one embodiment.

[0184] Referring to FIG. 12, the electronic device (300) can generate metadata such as “car, changeable area - headlight” when a sticker is created with an image of the front of a car.

[0185] In this case, as in the example of 1210, when a context such as “I don’t understand” is input, the electronic device (300) can generate and provide a recommended sticker (1211) by modifying the headlight, which is a changeable part of the front image of the car, to suit the context.

[0186]

[0187] FIG. 13 is a diagram illustrating an example of suggesting a recommended sticker based on a conversation in an electronic device according to one embodiment.

[0188] Referring to FIG. 13, as in the example of 1310, the electronic device (300) can suggest recommended stickers (1312, 1313) before the user inputs a response, using the previous conversation (1311) as a context.

[0189] For example, the electronic device (300) can provide a context-appropriate recommendation sticker even without a user inputting any characters by using at least one of a preset number of recent sentences used in a conversation as a context.

[0190]

[0191] FIG. 14 is a diagram illustrating an example of a recommended emoticon and an example of generating a recommended sticker when the character “congratulations” is entered according to one embodiment.

[0192] Referring to FIG. 14, the example of 1410 is an example of searching for an emoticon for the text “Congratulations” through a third-party app or server and displaying related emoticons (1411, 1412, 1413).

[0193] Example 1420 is an example in which, when the character "Congratulations" is entered, the sticker object is transformed by considering the context of "Congratulations" using the stickers and metadata pre-selected by the user to create a user-specific recommended sticker (1421, 1422).

[0194]

[0195] FIG. 15 is a diagram illustrating an example of a recommended emoticon and an example of a recommended sticker generated when the character “sad” is entered according to one embodiment.

[0196] Referring to FIG. 15, the example of 1510 is an example of searching for an emoticon for the character “upset” through a third-party app or server and displaying related emoticons (1511, 1512, 1513).

[0197] In the example of 1520, when the character "upset" is entered, the user's pre-selected stickers and metadata are used to transform the expression of the sticker object in consideration of the context of "upset" and create a user-specific recommended sticker (1521, 1522).

[0198]

[0199] As shown in the examples of Figs. 14 and 15, the sticker creation method of the present disclosure creates stickers by modifying them according to the context from images selected by the user rather than search results, thereby providing stickers unique to the user.

[0200]

[0201] The present disclosure provides personalized stickers by isolating objects from an image, generating stickers containing metadata, and then using artificial intelligence to generate recommended stickers based on the stickers and context. In one embodiment, personalized stickers can more accurately express an individual's personality and emotions, thereby providing a personalized experience.

[0202] While metadata was used in this disclosure to generate recommended stickers, the metadata of this disclosure can be applied to a variety of applications. For example, when creating an avatar expressing emotions, metadata can be used to specify moving and stationary parts.

[0203]

[0204] The present disclosure can provide a device and method for generating stickers utilizing behavioral capabilities. The technical challenges addressed by the present disclosure are not limited to the technical challenges mentioned above, and other technical challenges not mentioned will be readily apparent to those skilled in the art.

[0205] According to one embodiment, a method for generating a sticker may include: identifying an object in an image; analyzing the object to determine at least one movable element included in the object; storing information associated with the movable element as metadata associated with the object; generating a sticker including the object; and storing the metadata corresponding to the object in association with the sticker.

[0206] According to one embodiment, the operation of analyzing the object to determine at least one movable element contained in the object may analyze the metadata for the object using large multimodal models (LMMs) or multimodal large language models (MM-LLMs).

[0207] According to one embodiment, the operation of generating a sticker including the object includes an operation of generating the sticker by including the background of the object, wherein the background of the object may be a background of the object extracted from the image, a preset background, or a background generated by a user's selection.

[0208] According to one embodiment, when a context is input, the method may further include: generating at least one recommended sticker corresponding to the context using the context, the sticker, and the metadata corresponding to the object included in the sticker; and displaying the recommended sticker on a display (160, 320).

[0209] In one embodiment, the context may include at least one of a sentence input by the user and a preset number of recent sentences used in the conversation.

[0210] According to one embodiment, the operation of generating the at least one recommended sticker may input the metadata corresponding to the context, the sticker, and the object included in the sticker into a large vision model (LVM) to generate the at least one recommended sticker corresponding to the context.

[0211] According to one embodiment, the operation of generating at least one recommended sticker corresponding to the context may include an operation of generating a dynamic sticker in which the object moves as the recommended sticker.

[0212] In one embodiment, the operation of generating at least one recommended sticker using the metadata may include an operation of generating a sticker in which the plurality of objects interact, if the sticker includes a plurality of objects.

[0213] According to one embodiment, the metadata may include at least one of information about the object, information confirming whether the object includes the entire object or a portion of the object, information indicating a movable portion of the object, and information indicating a changeable portion of the object.

[0214] According to one embodiment, the operation of storing the metadata corresponding to the object in conjunction with the sticker may include storing the metadata corresponding to the object in the sticker, storing the metadata by matching it with the sticker using a database, or storing the metadata by matching it with the sticker through an external server.

[0215] According to one embodiment, a computer-readable recording medium may store commands, which, when executed by one or more processors (120, 310), may perform the following actions: identifying an object in an image; analyzing the object to determine at least one movable element included in the object; storing information associated with the movable element as metadata associated with the object; generating a sticker including the object; and storing the metadata corresponding to the object in association with the sticker.

[0216] According to one embodiment, an electronic device (101, 300) includes a display (160, 320); one or more processors (120, 310); and a memory (130, 330) for storing instructions, wherein when the instructions are individually or collectively executed by the one or more processors (120, 310), the one or more processors (120, 310) can perform the following operations: identifying an object in an image; analyzing the object to determine at least one movable element included in the object; storing information associated with the movable element as metadata associated with the object; generating a sticker including the object; and storing the metadata corresponding to the object in association with the sticker.

[0217] According to one embodiment, an electronic device (101, 300) includes a display (160, 320); one or more processors (120, 310); and a memory (130, 330) for storing instructions.

[0218] When the above instructions are individually or collectively executed by the one or more processors (120, 310), the one or more processors (120, 310) may perform the following operations: identifying an object in an image; analyzing the object to determine at least one changeable element included in the object; storing information associated with the changeable element as metadata associated with the object; generating a sticker including the object; and displaying the generated sticker on the display (160, 320).

[0219] According to one embodiment, the image may be any image stored in the electronic device (101, 300), or an image selected by the user from any images stored in the electronic device (101, 300).

[0220] According to one embodiment, when performing an operation to generate a sticker including the object, the one or more processors (120, 310) perform an operation to generate the object by changing the object into the original format of the image or a format different from the original format for each changeable element, and the metadata may include format information for each of the changeable elements.

[0221] According to one embodiment, the metadata may be set by the user in such a way that tags are added to each of the changeable elements or the changeable elements are designated on the image.

[0222] According to one embodiment, when performing an operation to generate a sticker including the object, the one or more processors (120, 310) may, when a context is input, perform an operation to generate at least one recommended sticker corresponding to the context using the context and the metadata.

[0223] According to one embodiment, when the one or more processors (120, 310) perform an operation of displaying the generated sticker on the display (160, 320), when text is input on a messenger application installed on the electronic device (101, 300), the one or more processors (120, 310) may perform an operation of displaying and presenting the at least one recommended sticker on the display (160, 320).

[0224] According to one embodiment, the one or more processors (120, 310) may perform an operation of generating a new recommendation sticker through a generative artificial intelligence model by combining metadata corresponding to the two or more selected recommendation stickers when at least two recommendation stickers are selected; and an operation of displaying the generated new recommendation sticker on the display (160, 320).

[0225] According to one embodiment, the metadata includes sound data information related to the object, and the one or more processors (120, 310) may perform an operation of outputting sound data related to the object together when performing an operation of displaying the generated sticker on the display (160, 320).

[0226] The effects that can be obtained from the present disclosure are not limited to the effects mentioned above, and other effects that are not mentioned will be clearly understood by a person having ordinary skill in the art to which the present disclosure pertains.

[0227] Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to the embodiments of this document are not limited to the aforementioned devices.

[0228] The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another component (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0229] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0230] Various embodiments of the present document may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0231] According to one embodiment, the method according to various embodiments disclosed in this document may be provided as a computer program product. The computer program product may be traded between sellers and buyers as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or may be provided through an application store (e.g., Play Store). TM ) or directly between two user devices (e.g., smart phones), online distribution (e.g., downloading or uploading). In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored or temporarily created in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0232] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separately arranged in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0233] The embodiments disclosed in this specification and drawings are merely specific examples presented to easily explain the technical contents according to the embodiments of this document and to help understand the embodiments, and are not intended to limit the scope of the embodiments of this document. Therefore, the scope of the various embodiments of this document should be interpreted to include all changes or modified forms derived based on the technical ideas of the various embodiments, in addition to the embodiments disclosed herein.

Claims

1. In the electronic device (101, 300), display(160, 320); one or more processors; and Memory for storing instructions (130, 330) Including, When the above instructions are individually or collectively executed by the one or more processors, the one or more processors, The action of identifying objects in an image; An action of analyzing the object to determine at least one mutable element contained in the object; An action of storing information associated with the above changeable element as metadata associated with the object; An action to create a sticker containing the above object; and An action is performed to display the generated sticker on the display (160, 320). Electronic devices (101, 300).

2. In paragraph 1, The image above is, Any image stored in the above electronic device (101, 300), or An image selected by the user from any image stored in the electronic device (101, 300). Electronic devices (101, 300).

3. In paragraph 1, One or more of the above processors, When performing an action to create a sticker containing the above object, An action to create the above object by changing it into the original format of the image or a format different from the original format for each changeable element. , and The above metadata is, Contains format information for each of the above changeable elements. Electronic devices (101, 300).

4. In paragraph 3, The above metadata is, Tags are added for each of the above changeable elements or are set by the user in a way that the changeable elements are specified on the image. Electronic devices (101, 300).

5. In paragraph 1, One or more of the above processors, When performing an action to create a sticker containing the above object, When a context is input, an operation of generating at least one recommended sticker corresponding to the context using the context and the metadata. An electronic device (101, 300) that performs the operation.

6. In paragraph 5, One or more of the above processors, When performing an operation of displaying the generated sticker on the display (160, 320), An operation in which, when text is entered on a messenger application installed on the electronic device (101, 300), at least one recommended sticker is displayed and presented on the display (160, 320). An electronic device (101, 300) that performs the operation.

7. In paragraph 6, One or more of the above processors, When at least two or more recommended stickers are selected, an operation of generating a new recommended sticker through a generative artificial intelligence model by combining metadata corresponding to each of the two or more selected recommended stickers; and An action of displaying the newly generated recommended sticker on the display (160, 320). Electronic device (101, 300) that receives the signal.

8. In paragraph 1, The above metadata is, Contains sound data information related to the above object, One or more of the above processors, When performing an action of displaying the generated sticker on the display (160, 320), an action of outputting sound data related to the object together An electronic device (101, 300) that performs the operation.

9. How to create a sticker: The action of identifying objects in an image; An action of analyzing the object to determine at least one movable element contained in the object; An action of storing information associated with the above movable element as metadata associated with the object; An action to create a sticker containing the above object; and An action of storing the metadata corresponding to the object in conjunction with the sticker. How to include.

10. In paragraph 9, The action of creating a sticker containing the above object is: An action that creates the background of the object by including it in the sticker. Including, The background of the above object is, The background of the object extracted from the image above, or A preset background, or Background generated by user's choice method.

11. In paragraph 9, When a context is input, an operation of generating at least one recommended sticker corresponding to the context using the metadata corresponding to the context, the sticker, and the object included in the sticker; and Action to display the above recommended stickers on the display How to include more.

12. In paragraph 11, The action of generating at least one recommended sticker is as follows: Inputting the metadata corresponding to the context, the sticker, and the object included in the sticker into a large vision model (LVM) to generate at least one recommended sticker corresponding to the context. method.

13. In paragraph 11, The operation of generating at least one recommended sticker corresponding to the above context is: An action to create a dynamic sticker with the above object moving as the above recommended sticker. How to include.

14. In paragraph 9, The action of generating at least one recommended sticker using the above metadata is: An action to create a sticker in which the multiple objects interact when the above sticker contains multiple objects. How to include.

15. In paragraph 9, The operation of storing the metadata corresponding to the object in conjunction with the sticker is as follows: or save the metadata corresponding to the object in the sticker, Store the metadata by matching it with the above sticker using a database, or Matching and storing the metadata corresponding to the sticker through an external server method.

Images