Electronic device and control method thereof

Abstract

An electronic device is disclosed. One or more processors, when instructions are individually or collectively executed: input images stored in a memory to a trained first artificial intelligence model to obtain a plurality of pieces of emotion information for each image; when a region is selected in an object image displayed on a display, display, via the display, a first UI including a first image corresponding to the selected region and the plurality of pieces of emotion information; when one of the plurality of pieces of emotion information is selected, identify, on the basis of the obtained plurality of pieces of emotion information for each image, an image corresponding to the first image and the selected emotion information from among the images stored in the memory; display, via the display, a second UI for selecting a sticker image, generated on the basis of the identified image, and an emotion level; and when the emotion level is selected via the second UI, input the selected emotion level and the sticker image to a trained second artificial intelligence model to obtain a sticker image corresponding to the selected emotion level.

Description

Electronic device and method of controlling the same

[0001] The present disclosure relates to an electronic device and a method for controlling the same, and more particularly, to an electronic device for obtaining an emotional sticker image and a method for controlling the same.

[0002] Recently, with the development of electronic devices, technology has been developed to select an area from a captured image stored in the device and generate a sticker image corresponding to the selected area.

[0003] Previously, it was possible to select emotional information related to a generated sticker image and send the generated sticker image when text related to the emotion was entered.

[0004] However, there was difficulty in automatically generating sticker images that included various emotions about the user by utilizing the generated sticker images.

[0005] An electronic device according to one or more embodiments of the present disclosure includes a display, a memory for storing instructions, and one or more processors including processing circuitry.

[0006] According to one or more embodiments, the one or more processors, when the instructions are individually or collectively executed, input images stored in the memory into a learned first artificial intelligence model to obtain multiple emotional information for each image, and when an area is selected from an object image displayed on the display, display a first UI including a first image corresponding to the selected area and the multiple emotional information through the display, and when one of the multiple emotional information is selected, identify an image corresponding to the first image and the selected emotional information among the images stored in the memory based on the acquired multiple emotional information for each image, and display a second UI for selecting a sticker image and an emotional level generated based on the identified image through the display, and when the emotional level is selected through the second UI, input the selected emotional level and the sticker image into a learned second artificial intelligence model to obtain a sticker image corresponding to the selected emotional level.

[0007] According to one or more embodiments, the one or more processors, when the instructions are individually or collectively executed, acquire a plurality of sticker images corresponding to different emotional levels for the identified image and the selected emotional information through the learned first artificial intelligence model, and store the acquired plurality of sticker images in the memory.

[0008] According to one or more embodiments, the learned first artificial intelligence model is trained to, when an image and emotional information are input, identify a candidate image corresponding to the input image among images stored in the memory, identify a target image corresponding to the input emotional information among the identified candidate images, and convert the identified target image into a sticker image and output it.

[0009] According to one or more embodiments, when the instructions are individually or collectively executed, if the emotion level is selected through the second UI, the one or more processors display a third UI for adjusting the emotion level corresponding to the sticker image through the display, and if the emotion level is adjusted through the third UI, the one or more processors display an image corresponding to the adjusted emotion level among a plurality of sticker images stored in the memory through the display.

[0010] According to one or more embodiments, the different emotion levels corresponding to the selected emotion information include at least two levels from a first level to a fourth level, and a first sticker image corresponding to the first level includes an object of a first expression changed to correspond to the emotion information, a second sticker image corresponding to the second level includes an object of a second expression changed to correspond to the emotion information, a third sticker image corresponding to the third level includes an object of the second expression changed to correspond to the emotion information and an effect image related to the emotion information, and a fourth sticker image corresponding to the fourth level includes a motion object of the second expression changed to correspond to the emotion information and an effect image related to the emotion information.

[0011] According to one or more embodiments, the one or more processors, when the instructions are individually or collectively executed, identify a second image corresponding to the selected area when an area is selected from an object image displayed on the display, input the identified second image to a learned second artificial intelligence model to obtain a sticker image corresponding to the second image and store the obtained sticker image in the memory, and the learned second artificial intelligence model is trained to identify an image corresponding to the input image among images stored in the memory when an image is input, correct the input image based on the identified image, and obtain a sticker image based on the corrected image.

[0012] According to one or more embodiments, the learned second artificial intelligence model is trained to correct the input image through at least one of complementing a main object included in the input image, deleting a sub-object excluding the main object, and correcting the resolution based on the identified image, and to obtain a sticker image based on the corrected image.

[0013] According to one or more embodiments, the one or more processors, when the instructions are individually or collectively executed, obtain a wallpaper image including a plurality of sticker images using the learned second artificial intelligence model, and provide the obtained wallpaper image as a lock screen image.

[0014] According to one or more embodiments, the one or more processors, when the instructions are individually or collectively executed, provide a pattern lock method by using the plurality of sticker images included in the wallpaper as UI elements when the wallpaper image is provided as the lock screen image.

[0015] According to one or more embodiments, the one or more processors provide the pattern lock method through at least one of a dragging order for a path including the plurality of stickers, a touching order for the plurality of sticker images, and a number of touching times for each of the plurality of stickers when the instructions are individually or collectively executed.

[0016] According to one or more embodiments, a method for controlling an electronic device includes the steps of: inputting images stored in the electronic device into a learned first artificial intelligence model to obtain a plurality of emotional information for each image; when an area is selected from an object image, displaying a first UI including a first image corresponding to the selected area and the plurality of emotional information; when one of the plurality of emotional information is selected, identifying an image corresponding to the first image and the selected emotional information among images stored in the electronic device based on the obtained plurality of emotional information for each image; displaying a second UI for selecting a sticker image and an emotional level generated based on the identified image; and when the emotional level is selected through the second UI, inputting the selected emotional level and the sticker image into a learned second artificial intelligence model to obtain a sticker image corresponding to the selected emotional level.

[0017] A non-transitory computer-readable storage medium storing computer instructions that, when executed by a processor of an electronic device, cause the display device to perform an operation, the operation comprises: a step of inputting images stored in the electronic device into a learned first artificial intelligence model to obtain a plurality of pieces of emotional information for each image; a step of displaying a first UI including a first image corresponding to the selected area and the plurality of pieces of emotional information when an area of ​​an object image is selected; a step of identifying an image corresponding to the first image and the selected emotional information among images stored in the electronic device based on the acquired plurality of pieces of emotional information when one of the plurality of pieces of emotional information is selected; a step of displaying a second UI for selecting a sticker image and an emotional level generated based on the identified image; and a step of inputting the selected emotional level and the sticker image into a learned second artificial intelligence model when the emotional level is selected through the second UI to obtain a sticker image corresponding to the selected emotional level.

[0018] FIG. 1 is a diagram illustrating the operation of an electronic device according to one or more embodiments.

[0019] FIG. 2 is a block diagram illustrating a configuration of an electronic device according to one or more embodiments.

[0020] FIG. 3 is a block diagram illustrating a detailed configuration of an electronic device according to one or more embodiments of the present disclosure.

[0021] FIG. 4 is a diagram illustrating a UI including emotional information of an electronic device according to one or more embodiments.

[0022] FIG. 5 is a diagram illustrating a process for acquiring a sticker image of another electronic device in one or more embodiments.

[0023] FIG. 6 is a diagram illustrating an emotion extraction learning process of a first artificial intelligence model according to one or more embodiments.

[0024] FIG. 7 is a diagram illustrating a process for obtaining a sticker image corresponding to an emotional level of an electronic device according to one or more embodiments.

[0025] FIG. 8 is a drawing for explaining a sticker image corresponding to an emotional level of an electronic device according to one or more embodiments.

[0026] FIG. 9 is a diagram illustrating an emotion level control UI of an electronic device according to one or more embodiments.

[0027] FIG. 10 is a flowchart illustrating a process for obtaining a sticker image corresponding to emotional information of an electronic device according to one or more embodiments.

[0028] FIG. 11 is a diagram illustrating a process for acquiring a photographed image including the same object of an electronic device according to one or more embodiments.

[0029] FIG. 12 is a diagram illustrating a process for obtaining a corrected sticker image of an electronic device according to one or more embodiments.

[0030] FIG. 13 is a flowchart illustrating a sticker image supplementation process of an electronic device according to one or more embodiments.

[0031] FIG. 14 is a diagram illustrating a process for obtaining a wallpaper image of an electronic device according to one or more embodiments.

[0032] FIGS. 15a, 15b, 15c and 15d are drawings illustrating a pattern lock method according to one or more embodiments.

[0033] FIG. 16 is a flowchart illustrating a process for providing a pattern lock method using a wallpaper of an electronic device according to one or more embodiments.

[0034] FIG. 17 is a flowchart illustrating the overall operation process of an electronic device according to one or more embodiments.

[0035] The terms used in the various embodiments of this disclosure have been selected from widely used, current terms, taking into account the functions of this disclosure. However, these terms may vary depending on the intentions of those skilled in the art, precedents, the emergence of new technologies, etc. Furthermore, in certain cases, terms may be arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the description of the relevant disclosure. Therefore, the terms used in this disclosure should be defined based on the meaning of the terms and the overall content of this disclosure, rather than simply their names.

[0036] In this disclosure, expressions such as “has,” “can have,” “includes,” or “may include” indicate the presence of a corresponding feature (e.g., a component such as a number, function, operation, or part), and do not exclude the presence of additional features.

[0037] The expression "at least one of A and / or B" should be understood to mean either "A" or "B" or "A and B".

[0038] The expressions “first,” “second,” “first,” or “second,” etc., used in this disclosure can describe various components, regardless of order and / or importance, and are only used to distinguish one component from another, but do not limit the components.

[0039] When it is said that a component (e.g., a first component) is “(operatively or communicatively) coupled with / to” or “connected to” another component (e.g., a second component), it should be understood that the component may be directly coupled to the other component, or may be connected through another component (e.g., a third component).

[0040] Singular expressions include plural expressions unless the context clearly dictates otherwise. In this disclosure, terms such as "comprise" or "consist of" are intended to indicate the presence of a feature, number, step, operation, component, part, or combination thereof described in the specification, but should be understood not to preclude the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0041] In the present disclosure, a "module" or "part" performs at least one function or operation and may be implemented as hardware or software, or as a combination of hardware and software. Furthermore, multiple "modules" or multiple "parts" may be integrated into at least one module and implemented as at least one processor (not shown), excluding any "modules" or "parts" that need to be implemented as specific hardware.

[0042] In this disclosure, the term user may refer to a person using an electronic device or a device used by the person.

[0043] An embodiment of the present disclosure will be described in more detail with reference to the attached drawings below.

[0044] FIG. 1 is a diagram illustrating the operation of an electronic device according to one or more embodiments.

[0045] The electronic device (100) can obtain a sticker image corresponding to a captured image from a captured image stored within the device. Here, the electronic device (100) can be implemented as various types of electronic devices, such as a mobile device, a tablet, a PC, a laptop PC, a wearable device, a kiosk, a TV, etc. For convenience of explanation, the present disclosure will be described assuming that the electronic device (100) is a mobile device.

[0046] According to one embodiment, the electronic device (100) may obtain a sticker image based on a captured image stored in the electronic device (100). For example, when a user selects a certain area in a captured image obtained through a camera (not shown), the electronic device (100) may obtain a sticker image corresponding to the selected area. For example, when a user selects a certain area in a captured image received from a server, the electronic device (100) may obtain a sticker image corresponding to the selected area.

[0047] Sticker images can be small images or graphics used in digital devices. For example, sticker images can be used to express emotions, add fun to conversations, and provide visual decoration in messenger apps, social media, digital documents, etc. For example, sticker images can take various forms, such as captured photos, cartoons, illustrations, and 3D graphics. Furthermore, sticker images can be in file formats such as PNG or GIF. Meanwhile, sticker images can be referred to in various ways, such as digital sticker images, graphic sticker images, and emoji sticker images, but in this disclosure, they will be collectively referred to as sticker images.

[0048] A sticker image according to an example may include an image generated by extracting only objects selected by the user from among multiple objects included in the image. For example, if the user selects only the puppy image in a captured image of the user holding the puppy, the electronic device (100) may identify a border area containing the puppy and extract only the puppy image from the identified area. Thereafter, the electronic device (100) may generate a sticker image corresponding to the extracted puppy image.

[0049] For example, a sticker image may be an image that removes the background from the entire image and extracts only the objects contained within the image. For example, if a user selects only the image of the puppy from an image taken of a user holding a puppy in their home, the sticker image may be an image that removes the background of the home and extracts only the image of the puppy.

[0050] Referring to FIG. 1, the electronic device (100) may receive a user input for selecting an object included in a first captured image (10-1) among a plurality of captured images (10-1 to 10-n) (n is an integer greater than or equal to 1) stored in the electronic device (100). For example, if the user touches a specific object in the first captured image (10-1) for a preset period of time, the electronic device (100) may provide a guide UI for creating a sticker. For example, the guide UI may include a guide graphic for indicating a selected area and a guide menu for receiving a sticker creation command. For example, the guide UI may include a guide graphic (13) including a border line including an object (12) selected by the user, as illustrated in FIG. 1, and a guide menu (14) including the text "Create Sticker." However, the guide graphic may include various methods for indicating a selected area, such as highlighting the selected object, for example, displaying it in a dark shade or hatched form. The guide menu can include various texts that guide sticker creation, such as "Save Sticker" and "Create Sticker." In other examples, the guide menu doesn't necessarily have to contain text; it can also be displayed as an image, icon, or other element that suggests sticker creation.

[0051] Thereafter, when a user command for sticker creation is received through the guide menu, the electronic device (100) can obtain sticker images (14-1 to 14-n) (n is an integer greater than or equal to 1) corresponding to objects within the selected area based on the user input. As another example, the sticker creation command may be received as a voice command such as "Create a sticker" rather than through the guide UI displayed on the screen.

[0052] According to one embodiment, the electronic device (100) can obtain emotional information corresponding to a human expression from a second captured image (10-2) including a human face among a plurality of captured images (10-1 to 10-n).

[0053] For example, the electronic device (100) may display a second captured image (10-2) and a user interface (UI) including multiple pieces of emotional information via a display. The electronic device (100) may receive a user input from the user to select one of the pieces of emotional information displayed on the UI, and may obtain the emotional information selected by the user.

[0054] For example, if the second captured image (10-2) includes a human face with a smiling expression, the electronic device (100) can obtain "Happy" emotion information by receiving a user input for selecting the "Happy" emotion from among multiple pieces of emotion information displayed on the UI. For example, if the second captured image (10-2) includes a human face with an angry expression, the electronic device (100) can obtain "Angry" emotion information by receiving a user input for selecting the "Angry" emotion.

[0055] According to one embodiment, the electronic device (100) can acquire a sticker image by utilizing an image and emotional information selected by a user in a learned artificial intelligence model. Below, various embodiments of acquiring a sticker image, as well as various embodiments of creating wallpaper using the sticker image, will be described with reference to the drawings.

[0056] FIG. 2 is a block diagram illustrating a configuration of an electronic device according to one or more embodiments.

[0057] According to FIG. 2, the electronic device (100) includes a display (110), a memory (120), and one or more processors (130). However, the present invention is not limited thereto, and the electronic device (100) may be implemented in a form in which some components are excluded, or may be implemented in a form in which other components are further included.

[0058] The display (110) is a configuration for displaying a captured image and multiple UIs on various screens. The display (110) may be implemented as a display module including a self-luminous element or a display module including a non-luminous element and a backlight. In addition, the display (110) may be implemented as an LFD display according to the above-described content. For example, the display may be implemented as various types of displays such as an LCD (Liquid Crystal Display), an OLED (Organic Light Emitting Diodes) display, an LED (Light Emitting Diodes), a micro LED, a Mini LED, a PDP (Plasma Display Panel), a QD (Quantum dot) display, a QLED (Quantum dot light-emitting diodes), etc. The display (110) may also include a driving circuit, a backlight unit, etc., which may be implemented in a form such as an a-si TFT, an LTPS (low temperature poly silicon) TFT, an OTFT (organic TFT), etc.

[0059] The memory (120) can store at least one command, data, program, etc. required for the operation of the electronic device (100). For example, the memory (120) can store a photographed image captured by a camera. For example, the memory (120) can store a plurality of sticker images acquired from an artificial intelligence model.

[0060] The memory (120) may be implemented in the form of memory embedded in the electronic device (100) or in the form of memory that can be attached or detached from the electronic device (100), depending on the purpose of data storage. For example, data for driving the electronic device (100) may be stored in a memory embedded in the electronic device (100), and data for expanding the functions of the electronic device (100) may be stored in a memory that can be attached or detached from the electronic device (100).

[0061] In the case of memory embedded in the electronic device (100), it may be implemented as at least one of volatile memory (e.g., dynamic RAM (DRAM), static RAM (SRAM), or synchronous dynamic RAM (SDRAM)), non-volatile memory (e.g., one time programmable ROM (OTPROM), programmable ROM (PROM), erasable and programmable ROM (EPROM), electrically erasable and programmable ROM (EEPROM), mask ROM, flash ROM, flash memory (e.g., NAND flash or NOR flash), hard drive, or solid state drive (SSD)).

[0062] The memory (120) may be implemented as a single memory that stores data generated in various operations according to the present disclosure, but is not limited thereto, and the memory (120) may be implemented to include multiple memories that each store different types of data or each store data generated in different stages.

[0063] One or more processors (130) control the overall operation of the electronic device (100). Specifically, one or more processors (130) may be connected to each component of the electronic device (100) to control the overall operation of the electronic device (100). For example, one or more processors (130) may be electrically connected to the display (110) and the memory (120) to control the overall operation of the electronic device (100). One or more processors (130) may include a processing circuit and may be configured with one or more processors.

[0064] One or more processors (130) can perform operations of the display device (100) according to various embodiments by executing one or more commands stored in the memory (120).

[0065] The one or more processors (130) may include one or more of a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), an Accelerated Processing Unit (APU), a Many Integrated Core (MIC), a Digital Signal Processor (DSP), a Neural Processing Unit (NPU), a hardware accelerator, or a machine learning accelerator. The one or more processors (130) may control one or any combination of other components of the refrigeration device, and may perform operations related to communication or data processing. The one or more processors (130) may execute one or more programs or instructions stored in a memory. For example, the one or more processors may perform a method according to one or more embodiments of the present disclosure by executing one or more instructions stored in a memory.

[0066] When a method according to one or more embodiments of the present disclosure includes multiple operations, the multiple operations may be performed by one processor or by multiple processors. For example, when a first operation, a second operation, and a third operation are performed by a method according to one or more embodiments, the first operation, the second operation, and the third operation may all be performed by the first processor, or the first operation and the second operation may be performed by the first processor (e.g., a general-purpose processor) and the third operation may be performed by the second processor (e.g., an artificial intelligence-specific processor).

[0067] One or more processors (130) may be implemented as a single core processor including one core, or may be implemented as one or more multicore processors including multiple cores (e.g., homogeneous multicores or heterogeneous multicores). When one or more processors (150) are implemented as a multicore processor, each of the multiple cores included in the multicore processor may include an internal processor memory, such as a cache memory or an on-chip memory, and a common cache shared by the multiple cores may be included in the multicore processor. In addition, each of the multiple cores (or some of the multiple cores) included in the multicore processor may independently read and execute a program instruction for implementing a method according to one or more embodiments of the present disclosure, or all (or some) of the multiple cores may be linked to read and execute a program instruction for implementing a method according to one or more embodiments of the present disclosure.

[0068] When a method according to one or more embodiments of the present disclosure includes a plurality of operations, the plurality of operations may be performed by one core among the plurality of cores included in a multi-core processor, or may be performed by the plurality of cores. For example, when a first operation, a second operation, and a third operation are performed by a method according to one or more embodiments, the first operation, the second operation, and the third operation may all be performed by a first core included in the multi-core processor, or the first operation and the second operation may be performed by a first core included in the multi-core processor, and the third operation may be performed by a second core included in the multi-core processor.

[0069] In the embodiments of the present disclosure, a processor may mean a system on a chip (SoC) in which one or more processors and other electronic components are integrated, a single-core processor, a multi-core processor, or a core included in a single-core processor or a multi-core processor, wherein the core may be implemented as a CPU, a GPU, an APU, a MIC, a DSP, an NPU, a hardware accelerator, or a machine learning accelerator, but the embodiments of the present disclosure are not limited thereto. Hereinafter, for the convenience of explanation, one or more processors (130) will be referred to as a processor (130).

[0070] According to one embodiment, the processor (130) may input images stored in the memory (120) into a trained first artificial intelligence model to obtain multiple emotional information for each image. For example, the trained first artificial intelligence model may be a model trained to identify multiple images corresponding to the input image and the emotional information selected by the user.

[0071] According to one embodiment, when an area is selected from an object image displayed on the display (110), the processor (130) may display an image corresponding to the selected area (hereinafter, a first image) and a UI (hereinafter, a first UI) including multiple pieces of emotional information through the display (110). The multiple pieces of emotional information may be information previously stored in the electronic device (100). The multiple pieces of emotional information may be updated by information received from an external device (e.g., a server).

[0072] For example, multiple emotional information may include emotions that people can generally feel, such as Happy, Sad, Angry, Fear, and Calm.

[0073] According to one embodiment, when one of the plurality of emotional information is selected, the processor (130) can identify the first image and the image corresponding to the selected emotional information among the images stored in the memory (120) based on the plurality of emotional information for each acquired image.

[0074] According to one embodiment, the processor (130) may display a second UI for selecting a sticker image and an emotion level generated based on the identified image through the display (110).

[0075] According to one embodiment, when an emotion level is selected through the second UI, the processor (130) may input the selected emotion level and sticker image into the learned second artificial intelligence model and display a sticker image corresponding to the selected emotion level through the display.

[0076] Here, an artificial intelligence model refers to a model that inputs specific input values ​​to a specific function based on learned data and outputs an output value. The artificial intelligence model may be a model trained by reflecting similarity between multiple pieces of text information. While artificial intelligence models can be referred to by various names, such as neural network models, deep learning models, neural network models, and generative artificial intelligence models, they will be collectively referred to as "artificial intelligence models" in this disclosure.

[0077] Here, the learning of the artificial intelligence model means that a basic artificial intelligence model (e.g., an artificial intelligence model including any random parameters) is learned using a plurality of training data by a learning algorithm, thereby creating a predefined operation rule or artificial intelligence model set to perform a desired characteristic (or purpose). Such learning may be performed through a separate server and / or system, but is not limited thereto, and may also be performed in the electronic device (100). Examples of the learning algorithm include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

[0078] Here, the artificial intelligence model can be implemented as, for example, a Convolutional Neural Network (CNN), a Recurrent Neural Network (RNN), a Restricted Boltzmann Machine (RBM), a Deep Belief Network (DBN), a Bidirectional Recurrent Deep Neural Network (BRDNN), or a Deep Q-Network, but is not limited thereto.

[0079] FIG. 3 is a block diagram illustrating a detailed configuration of an electronic device according to one or more embodiments of the present disclosure.

[0080] According to FIG. 3, the electronic device (100) includes a display (110), a memory (120), one or more processors (130), a camera (140), a communication interface (150), a speaker (160), and a microphone (170). Among the configurations illustrated in FIG. 3, a detailed description of configurations that overlap with those illustrated in FIG. 2 will be omitted.

[0081] The camera (140) is configured to capture a subject including multiple objects. For example, the processor (130) can capture multiple objects through the camera (140) and store the captured images in the memory (120).

[0082] The communication interface (150) includes a circuit and can communicate with an external device (mobile device or server). For example, the processor (130) can receive various data or information from an external device connected via the communication interface (150) and can also transmit various data or information to the external device.

[0083] The communication interface (150) may include at least one of a WiFi module, a Bluetooth module, a wireless communication module, an NFC module, and a UWB module (Ultra Wide Band). At this time, the wireless communication module may perform communication according to various communication standards such as IEEE, Zigbee, 3G (3rd Generation), 3GPP (3rd Generation Partnership Project), LTE (Long Term Evolution), 5G (5th Generation), etc.

[0084] The speaker (160) can convert and amplify a digital audio signal processed by the processor (130) into an analog audio signal and output the converted signal. For example, the speaker (160) can include at least one speaker unit, a D / A converter, an audio amplifier, etc., which can output at least one channel. For example, the speaker (160) can output various notifications, messages, information, etc. related to feedback from the processor (130).

[0085] The microphone (170) is configured to receive user voice or other sounds and convert them into audio data. For example, the processor (130) can analyze the user voice input through the microphone (170) and display a sticker image corresponding to the analyzed voice through the display (110).

[0086] FIG. 4 is a diagram illustrating a UI including emotional information of an electronic device according to one or more embodiments.

[0087] According to one embodiment, the electronic device (100) may display a first UI including a first image and a plurality of emotional information corresponding to an area selected by a user through a display (110).

[0088] According to an example, the electronic device (100) may receive a user input for selecting an area from one of a plurality of captured images (10) stored in the memory (120). When the electronic device (100) receives a user input for selecting an area from the user, the electronic device (100) may obtain a first image (450) for generating a sticker corresponding to the selected area. For example, the first image (450) may be a cropped image obtained by cropping the area selected by the user. For example, the boundary of a desired area may be selected by the user, and the electronic device (100) may obtain a cropped image including an image included in the selected boundary area. In another example, when a region is selected by the user, the electronic device (100) may obtain a cropped image by extracting an object boundary including the selected region. For example, the electronic device (100) may extract an object boundary including the selected region through object recognition, object detection, or the like. For example, as illustrated in FIG. 1, if a user selects a puppy image from a captured image of a user holding a puppy, the electronic device (100) can obtain a first image corresponding to the puppy image.

[0089] Referring to FIG. 4, the electronic device (100) can receive a user input for selecting an area in a selfie image (410). If the user touches an area including his / her face displayed on the display (110) for a preset period of time, the electronic device (100) can display a circle-shaped UI (430) and a “Create Sticker” text (440) on the selected area (420) as illustrated in FIG. 4. Meanwhile, the method for displaying the selected UI is not limited to the circle-shaped UI (430), and the selected area (420) can be displayed on the display (110) in various forms, such as displaying it in the form of a highlight line or a diagonal line including the selected area.

[0090] The electronic device (100) can obtain a first image (450) based on an area (420) selected by the user. That is, the electronic device (100) can obtain a first image (450) that includes only the user's face area from the user's self-captured image (410).

[0091] According to one embodiment, the electronic device (100) may display a first UI (470) including a first image (450) and a plurality of emotional information (460) on the display (110). In this case, the plurality of emotional information (460) is illustrated as including Happy, Sad, Angry, and Calm, but is not limited thereto, and may further include emotional information such as Fear, Surprised, etc.

[0092] For example, the electronic device (100) may display a first UI (470) including a guide phrase (480) such as "Please select an emotion to be created as a sticker," a first image (450), and multiple emotion information (460). As shown in FIG. 4, the positions where the guide phrase (480), the first image (450), and the multiple emotion information (460) are displayed are not limited thereto, and may be displayed in any order from top to bottom or from left to right.

[0093] FIG. 5 is a diagram illustrating a process for acquiring a sticker image of another electronic device in one or more embodiments.

[0094] According to one embodiment, the electronic device (100) can obtain a plurality of sticker images corresponding to different emotional levels for each identified image and selected emotional information through a learned first artificial intelligence model.

[0095] According to one embodiment, when one of a plurality of pieces of emotional information (460) is selected, the electronic device (100) may input the selected emotional information and the first image (450) into a learned first artificial intelligence model to obtain a sticker image corresponding to the first image (450) and the selected emotional information. The electronic device (100) may store at least one obtained sticker image in the memory (120).

[0096] Referring to FIG. 5, the electronic device (100) may receive a user input for selecting one of a plurality of emotional information items displayed on the first UI (470). The electronic device (100) may receive a user input corresponding to emotional information related to the first image (450) from the user, or may receive a user input corresponding to emotional information unrelated to the first image.

[0097] For example, if the first image (450) includes a facial expression of an angry user (51), the user (51) may select “Angry” emotional information related to the first image (450) or “Sad” emotional information unrelated to the first image (450). That is, the process of selecting emotional information displayed on the first UI (470) may be a process for obtaining a sticker image corresponding to the emotional information selected by the user (51) for the same person (user (51)) as the person displayed on the first image (450).

[0098] For example, for the same person as the user (51) included in the first image (450), a sticker image of the user (51) with an angry expression corresponding to the emotional information selected by the user (51) may be obtained, or a sticker image of the user (51) with a sad expression may be obtained.

[0099] According to one embodiment, when emotional information is selected by a user, the electronic device (100) may display the selected emotional information on the first UI (470). As illustrated in FIG. 5, when the user selects (510) the "Angry" emotional information for the first image (450), the electronic device (100) may display the selected emotional information on the first UI (470) in various ways, such as displaying the selected emotional information in a different color or highlight than other emotional information.

[0100] According to one embodiment, the electronic device (100) may input a first image (450) and selected emotional information (510) into a trained first artificial intelligence model (200). Here, when an image and emotional information are input, the first artificial intelligence model (200) may be trained to identify an image corresponding to the input image and emotional information among the captured images stored in the memory, and obtain a sticker image (520) based on the identified image.

[0101] The learned first artificial intelligence model (200) can be implemented as a generative AI model. A generative AI model is an artificial intelligence model that generates new content based on a given input, and can generate various types of data such as text, images, and voice. A generative AI model can be pre-trained on a large amount of data (e.g., text data) and then fine-tuned for various tasks based on the pre-trained data. A generative AI model can learn patterns of input data and generate new data based on a deep learning architecture such as a Recurrent Neural Network (RNN), a Long Short-Term Memory (LSTM), a Gated Recurrent Unit (GRU), or a Transformer. For example, a generative AI model can be implemented as various AI models such as a Stable diffusion model, a Generative Adversarial Network (GAN) model, Style Transfer Models, AI Content Generators, Transformer-Based Models, and Genetic Algorithms for Layout Optimization.

[0102] FIG. 6 is a diagram illustrating an emotion extraction learning process of a first artificial intelligence model according to one or more embodiments.

[0103] According to one embodiment, when image and emotional information are input, the first artificial intelligence model (200) can identify a candidate image corresponding to the input image among the captured images stored in the memory (120).

[0104] A candidate image may include an image that includes the same object as an object included in an input image among multiple captured images stored in the memory (120). In addition, it may also include an image that includes the same person as a person included in the input image.

[0105] Referring to FIG. 6, when a first image (610) corresponding to an area selected by a user among the captured images is input to the first artificial intelligence model (200), the first artificial intelligence model (200) can identify a face area from the first image (610) and extract feature points from the identified face area.

[0106] For example, the first artificial intelligence model (200) may generate an embedding vector using the CNN, DBN, etc. described above, using feature points extracted from the face region of the first image (610). Similarly, the first artificial intelligence model (200) may extract feature points of the face region from each of a plurality of captured images stored in the memory (120), and generate an embedding vector for the face region included in each image. The feature points of the face region represent important points in the face image and are located at major parts of the face (eyes, nose, mouth, facial contour, etc.), and may be composed of tens to hundreds of points. The embedding vector may be a result of converting high-dimensional data (e.g., words, sentences, images) into a low-dimensional vector space.

[0107] For example, the first artificial intelligence model (200) can calculate the similarity by comparing the embedding vector corresponding to the first image with the embedding vector corresponding to each of the plurality of captured images stored in the memory (120). For example, the first artificial intelligence model (200) can calculate the similarity based on a similarity calculation method such as cosine similarity, Euclidean distance, Manhattan distance, etc. The first artificial intelligence model (200) can identify the plurality of candidate images (630-1 to 630-4) by identifying the images exceeding a preset similarity as the same person as the face included in the first image (610).

[0108] According to one embodiment, the first artificial intelligence model (200) can identify emotional information related to each image (630-1 to 630-4) from a plurality of candidate images (630-1 to 630-4).

[0109] The first artificial intelligence model (200) can collect multiple facial image data containing multiple emotional information and perform an image preprocessing process on the collected data. The image preprocessing process may include aligning facial regions in the multiple facial image data, adjusting their size, and converting them to grayscale.

[0110] The first artificial intelligence model (200) can identify convolution layers and pooling layers corresponding to each of multiple facial images through a convolutional neural network (CNN). The first artificial intelligence model (200) can extract feature points of each facial image based on the convolution layers and pooling layers.

[0111] The first artificial intelligence model (200) can perform emotion classification based on the extracted features and acquire data corresponding to the features and classified emotions. Thereafter, the first artificial intelligence model (200) can extract emotions from the facial image using the acquired data.

[0112] Referring to FIG. 6, the learned first artificial intelligence model (200) can identify emotional information (631-1 to 631-4) associated with each of a plurality of candidate images (630-1 to 630-4). For example, the first artificial intelligence model (200) can identify "Sad" emotional information (631-1) from the first candidate image (630-1), "Happy" emotional information (631-2) from the second candidate image (630-2), "Calm" emotional information (631-3) from the third candidate image (630-3), and "Angry" emotional information (631-4) from the fourth candidate image (630-4).

[0113] According to one embodiment, the first artificial intelligence model (200) can identify a target image corresponding to the input emotional information (620) among the identified candidate images (630-1 to 630-4). The target image may include an image corresponding to the emotional information selected by the user among a plurality of candidate images (630-1 to 630-4).

[0114] Referring to FIG. 6, the first artificial intelligence model (200) can identify the fourth candidate image (630-4) corresponding to the input emotional information “Angry” as the target image.

[0115] According to one embodiment, the first artificial intelligence model (200) can convert the identified target image into a sticker image and output it. The first artificial intelligence model (200) can convert the fourth candidate image (630-4) identified as the target image into a sticker image through a cartoonization or characterization process.

[0116] The first artificial intelligence model (200) can identify major lines and extract outlines from a target image using an edge detection algorithm. The first artificial intelligence model (200) can perform a process of simplifying colors displayed in the target image using an image quantization mechanism, etc. Here, image quantization refers to a mechanism that simplifies colors by reducing the number of colors expressing a digital image. Thereafter, the first artificial intelligence model (200) can convert a realistic image, such as the target image shown, into a sticker image using a cartooning filter.

[0117] Returning to FIG. 5 again, the electronic device (100) can obtain a sticker image (520) corresponding to the first image (450) and the selected emotional information (510) through the first artificial intelligence model (200). For example, when the electronic device (100) receives a user input corresponding to the emotional information "Sad" from the user, the electronic device (100) can obtain a sticker image corresponding to the first candidate image (630-1).

[0118] FIG. 7 is a diagram illustrating a process for obtaining a sticker image corresponding to an emotional level of an electronic device according to one or more embodiments.

[0119] According to one embodiment, the electronic device (100) may acquire and store multiple sticker images corresponding to different emotional levels for selected emotional information in the memory (120). Here, the emotional level refers to the emotional level for the selected emotion classified into multiple levels. The emotional levels may be classified into Level 1 to Level 4, but are not limited thereto, and may be classified into various levels, such as Level 2 or Level 3.

[0120] In Fig. 7, the description will be made assuming that the user has selected the "Angry" emotion information. Referring to Fig. 7, the electronic device (100) can input the first image (610) and the selected "Angry" emotion information (620) into the first artificial intelligence model (200). The first artificial intelligence model (200) can identify a plurality of target images (631-4) corresponding to the first image (610) and the "Angry" emotion information.

[0121] According to one embodiment, the first artificial intelligence model (200) may be trained to identify a plurality of target images corresponding to emotional information input from a plurality of facial images, and classify the target images according to the emotional degree. For example, the first artificial intelligence model (200) may be trained to classify the emotional degree of the input "Angry" emotional information by considering various factors such as the shape of the eyebrows, the shape of the lips, the color of the skin, and the presence or absence of wrinkles on the forehead from the plurality of identified target images. Accordingly, the first artificial intelligence model (200) may classify the plurality of identified target images in order of the emotional degree from low to high.

[0122] For example, if the input emotion information is “Angry,” the first artificial intelligence model (200) can identify multiple facial images including a user with an angry expression, and identify the facial image with the lowest degree of anger as an image corresponding to the first level, and the facial image with the highest degree of anger as an image corresponding to the fourth level.

[0123] According to one embodiment, the first artificial intelligence model (200) may convert a plurality of facial images corresponding to each level into sticker images (710 to 730) and store the converted plurality of sticker images (710 to 730) in the memory (120). As illustrated in FIG. 7, the electronic device (100) may acquire sticker images (710 to 730) with increasingly angry expressions as they progress from the first level to the third level.

[0124] FIG. 8 is a drawing for explaining a sticker image corresponding to an emotional level of an electronic device according to one or more embodiments.

[0125] According to one embodiment, the different emotional levels corresponding to the selected emotional information may include at least two levels from the first level to the fourth level.

[0126] For example, a sticker image (hereinafter, referred to as the first sticker image) (810) corresponding to the first level may include an object with a first expression that has been changed to correspond to emotional information. Here, the object with the first expression may include an expression object for emotional information identified by an artificial intelligence model learned from a user's face image stored in the memory (120). For example, the first sticker image may include a sticker image of a smiling user, a sticker image of an angry user, and a sticker image of a sad user.

[0127] For example, a sticker image corresponding to the second level (hereinafter referred to as the second sticker image) (820) may include an object with a second expression that has been changed to correspond to emotional information. Here, the object with the second expression may include an object with an expression with a higher emotional level than the object with the first expression. For example, the second sticker image may include a sticker image of a user laughing out loud with an open mouth, a sticker image of a user being angry with wide eyes and a wide mouth, or a sticker image of a user being sad with tears in their eyes.

[0128] For example, a sticker image corresponding to the third level (hereinafter referred to as the third sticker image) (830) may include an object of a second expression changed to correspond to the emotion information and an effect image related to the emotion information. Here, the effect image related to the emotion information may include an image related to the emotion so that the emotion information can be emphasized. For example, in a sticker image of a smiling user, an effect image such as a musical note, a heart, a star, etc. may be added to further emphasize the user's "Happy" emotion. For example, in a sticker image of an angry user, an effect image such as a flame, a fist, etc. may be added to further emphasize the user's "Angry" emotion. Accordingly, the third sticker image may be a sticker image that includes the above-described effect image in the object of the second expression.

[0129] For example, a sticker image corresponding to the fourth level (hereinafter referred to as the fourth sticker image) (840) may include a motion object of a second facial expression that has been changed to correspond to emotional information and an effect image related to the emotional information. Here, the motion object may include a dynamic object that includes a short animation effect. For example, a motion object of an angry user may include a motion of clenching a fist instead of an open hand, and may include a motion of displaying a flame image in the user's eyes.

[0130] The expression objects, effect images, and motion objects of the first to fourth levels described above are not limited thereto, and may be implemented in various forms, such as outputting sound effects corresponding to emotional information through a speaker (160).

[0131] FIG. 9 is a diagram illustrating an emotion level control UI of an electronic device according to one or more embodiments.

[0132] According to one embodiment, when a user command for selecting a sticker image is input, the electronic device (100) may display a UI (hereinafter referred to as a second UI) including a sticker image stored in the memory (120) and emotional information corresponding to the sticker image through the display (110).

[0133] Referring to FIG. 9, the electronic device (100) can receive a user input for sending a sticker image from a messenger UI (910) for entering text. The electronic device (100) can display a second UI (940) including a plurality of sticker images stored in the memory (120) and emotional information corresponding to the sticker images at the bottom of the text input field.

[0134] Meanwhile, in FIG. 9, the second UI (940) is illustrated to include sticker images for four pieces of emotional information. However, if the electronic device (100) obtains sticker images for only two pieces of emotional information (e.g., Happy, Angry) from the user through the first artificial intelligence model, the second UI (940) may be displayed including only the two pieces of emotional information and the sticker images corresponding to each of the two pieces of emotional information. In other words, only the emotional information and sticker images corresponding to the emotional information selected by the user may be included in the second UI (940). In FIG. 9, the description will be made assuming that the user has selected four pieces of emotional information.

[0135] According to one embodiment, the electronic device (100) may receive a user input for selecting one of a plurality of emotional information items displayed on the second UI (940). For example, the electronic device (100) may receive a user input for selecting the "Happy" emotion item from among the plurality of emotional information items.

[0136] According to one embodiment, when an emotion and sticker image displayed through the second UI (940) are selected, the electronic device (100) may display a third UI (950) through the display (110) for adjusting an emotion level corresponding to the sticker image. Here, the third UI (950) may include a sticker image (980) corresponding to one of the first to fourth levels corresponding to the selected emotion information and a scale bar (970) for adjusting the emotion level.

[0137] According to one embodiment, when the emotional level is adjusted through the third UI (950), the electronic device (100) can display an image corresponding to the adjusted emotional level among a plurality of sticker images stored in the memory (120) through the display (110).

[0138] When the electronic device (100) receives a user input for adjusting the value of the scale bar (970), the electronic device (100) can display a sticker image for one of the first to fourth levels corresponding to the adjusted value through the third UI (950). For example, when the user sets the scale bar (970) to 25 (960), the electronic device (100) can display a sticker image (980) corresponding to the second level on the third UI (950). For example, when the user sets the scale bar (970) to 75 (970), the electronic device (100) can display a sticker image (990) corresponding to the third level on the third UI (950).

[0139] The electronic device (100) can display a UI (920) including a third UI (950) and a second UI (940) whose emotional level values ​​are set to 25 through the display (110). The electronic device (100) can display a UI (930) including a third UI (950) and a second UI (940) whose emotional level values ​​are set to 75 through the display (110).

[0140] According to one embodiment, the electronic device (100) may identify text input by a user and display a third UI (950) including a sticker image corresponding to the identified text through the display (110). For example, if the user inputs the text “I’m so happy today,” the electronic device (100) may identify the emotion “Happy” based on the text “I’m happy,” and display a third UI (950) including a sticker image corresponding to the emotion “Happy” through the display (110).

[0141] According to one embodiment, the electronic device (100) may obtain multiple images corresponding to each of a plurality of emotional levels from a first image through an artificial intelligence model (e.g., a first artificial intelligence model or a second artificial intelligence model). For example, when the first image and selected emotional information are input into the artificial intelligence model, the electronic device (100) may adjust eyes, a mouth, eyebrows, etc. included in the first image to obtain multiple images corresponding to each of a plurality of emotional levels.

[0142] For example, if the emotional information corresponding to the first image is 'Happy', when the first image is input to an artificial intelligence model, the artificial intelligence model can output the first image of the first level, the first image of the second level, the first image of the third level, and the first image of the fourth level corresponding to the emotional information 'Happy'.

[0143] For example, if the emotion information and emotion level corresponding to the first image is 'Angry' corresponding to the third level, when the first image is input to the artificial intelligence model, the artificial intelligence model can output the first image of the first level corresponding to the 'Angry' emotion information and the first image of the second level.

[0144] For example, an artificial intelligence model may obtain multiple pieces of emotional information from multiple images stored within an electronic device (100). The artificial intelligence model may obtain one piece of emotional information from among the multiple pieces of emotional information based on the facial expressions included in each image.

[0145] For example, the artificial intelligence model may be a model trained to obtain multiple emotional information from multiple images based on multiple images, emotional information, tagging information, etc.

[0146] For example, the artificial intelligence model may be a model trained to label tagging information and emotional information for each of a plurality of images stored in memory (120). When a first image is input, the artificial intelligence model may analyze the first image to identify tagging information, and extract candidate images from among the plurality of images based on the identified tagging information.

[0147] For example, an AI model can identify priorities based on emotional and tagging information input from extracted candidate images. Based on the identified priorities, the AI ​​model can output multiple images corresponding to the first image and the emotional information.

[0148] FIG. 10 is a flowchart illustrating a process for obtaining a sticker image corresponding to emotional information of an electronic device according to one or more embodiments.

[0149] Referring to FIG. 10, in operation 1011, the electronic device (100) can store a photographed image taken by a user within the electronic device (100).

[0150] In operation 1012, the electronic device (100) may receive user input for selecting an image area to be used by the user in the captured image.

[0151] In operation 1013, the electronic device (100) can analyze facial features in a first image (450) corresponding to a selected area through a first artificial intelligence model (200).

[0152] In operation 1014, the electronic device (100) can compare the first image (450) with the captured image stored in the memory (120) through the first artificial intelligence model (200), and identify the captured image that includes the same person as the person included in the first image (450).

[0153] In operation 1015, the electronic device (100) can identify emotional information by analyzing the captured image identified through the first artificial intelligence model (200).

[0154] In operation 1016, the electronic device (100) can generate multiple sets of emotional information from the captured image identified through the first artificial intelligence model (200).

[0155] In operation 1017, the electronic device (100) can convert a plurality of captured images corresponding to a plurality of emotional information into sticker images through the first artificial intelligence model (200).

[0156] In operation 1018, the electronic device (100) may receive a user input for selecting a sticker image to be used from among a plurality of sticker images.

[0157] In operation 1019, the electronic device (100) can identify data including a plurality of sticker images corresponding to an emotion selected by the user through the first artificial intelligence model (200).

[0158] In operation 1020, the electronic device (100) can identify multiple images corresponding to multiple emotional levels for an emotion selected by the user through the first artificial intelligence model (200).

[0159] In operation 1021, the electronic device (100) can convert a plurality of images identified through the first artificial intelligence model (200) into sticker images.

[0160] In operation 1022, the electronic device (100) may transmit a sticker image corresponding to the emotion level selected by the user.

[0161] FIG. 11 is a diagram illustrating a process for acquiring a photographed image including the same object of an electronic device according to one or more embodiments.

[0162] According to one embodiment, when an area is selected from an object image displayed on a display (110), the electronic device (100) can identify an image (hereinafter, a second image) corresponding to the selected area.

[0163] Referring to FIG. 11, when an area is selected from a captured image (1110) including multiple objects, the electronic device (100) can extract only an image (1111) corresponding to the selected area. The electronic device (100) can identify a second image (1120) based on the extracted image. For example, in a captured image including an image of a puppy holding a pillow, if the user selects only the puppy image, the electronic device (100) can identify the second image (1120) based on the selected puppy image.

[0164] According to one embodiment, the electronic device (100) may input a second image (1120) into a learned artificial intelligence model (hereinafter referred to as the second artificial intelligence model) to acquire a plurality of images including objects identical to objects included in the second image (1120) from captured images stored in the electronic device (100). Here, the second artificial intelligence model may be a model learned to acquire a plurality of images including objects identical to objects included in the input image from captured images stored in the memory (120).

[0165] According to one embodiment, the electronic device (100) may obtain a sticker image corresponding to the second image (1120) based on a plurality of images that include the same object as the object included in the second image (1120). The electronic device (100) may store the obtained sticker image in the memory (120).

[0166] Here, the second artificial intelligence model (300) learned can extract feature points of an object corresponding to an area selected by a user and generate an embedding vector for the object in the same manner as the first artificial intelligence model (200) described above. The second artificial intelligence model (300) can identify a plurality of captured images including the same object as the object included in the second image (1110) from a plurality of captured images stored in the memory. A detailed description thereof has been described above, and thus, any redundant description will be omitted. In addition, the second artificial intelligence model (300) learned may be at least a part of the first artificial intelligence model (200) learned.

[0167] Referring to FIG. 11, the electronic device (100) can identify a plurality of captured images (1130-1 to 1130-3) that include the same puppy image as the puppy image included in the second image (1120) through the second artificial intelligence model (300). For example, the electronic device (100) can identify a captured image (1130-1) of a lying puppy, a captured image (1130-2) of a walking puppy, and a captured image (1130-3) of a user holding a puppy based on the second image (1111).

[0168] In one example, the second image (1120) identified by the electronic device (100) may have a portion obscured by a pillow or other background image. For example, the second image (1120) including the puppy image illustrated in FIG. 11 may include an image in which a portion of the puppy's overall shape, such as the tail portion at the upper right, is omitted.

[0169] For example, the second image (1120) identified by the electronic device (100) may include object images other than the puppy image. For example, the second image (1120) illustrated in FIG. 11 may include a puppy image and a pillow image.

[0170] FIG. 12 is a diagram illustrating a process for obtaining a corrected sticker image of an electronic device according to one or more embodiments.

[0171] According to one embodiment, the electronic device (100) can identify a plurality of images including the same object as an input image through the learned second artificial intelligence model (300), and correct the input image based on the identified images. For example, the electronic device (100) can correct the second image (1120) based on a plurality of captured images (1130-1 to 1130-3). Here, the learned second artificial intelligence model (300) can be trained to identify an image corresponding to the input image among captured images stored in the memory (120) when an image is input, correct the input image based on the identified image, and obtain a sticker image based on the corrected image.

[0172] The learned second artificial intelligence model (300) can be trained to correct the input image through at least one of complementing the main object included in the input image, deleting a sub-object excluding the main object, and correcting the resolution based on the identified image, and to obtain a sticker image based on the corrected image.

[0173] Referring to FIG. 12, the second artificial intelligence model (300) can identify a plurality of captured images (1130-1 to 1130-3) including the same puppy image as the puppy image included in the second image (1120) based on the input second image (1120).

[0174] The second artificial intelligence model (300) can compensate for (1220) the tail portion at the upper right of the second image (1120) based on multiple captured images (1130-1 to 1130-3) and perform correction by deleting sub-objects (pillows) excluding the main object (puppy).

[0175] According to one embodiment, the electronic device (100) can obtain a sticker image based on a corrected image (1210) through a second artificial intelligence model (300).

[0176] According to one embodiment, the electronic device (100) may receive a user input to supplement a region of the second image (1120) into a shape or form desired by the user. For example, if the electronic device (100) receives a user input to set a dog's tail to a pig's tail or a rabbit's tail, the electronic device (100) may supplement the second image (1120) with a tail shape according to the user input based on a plurality of captured images (1130-1 to 1130-3) identified through the second artificial intelligence model (300).

[0177] FIG. 13 is a flowchart illustrating a sticker image supplementation process of an electronic device according to one or more embodiments.

[0178] Referring to FIG. 13, in operation 1310, the electronic device (100) can store a photographed image taken by the user within the electronic device (100).

[0179] In operation 1320, the electronic device (100) may receive a user input for selecting an image area to be used by the user in the captured image.

[0180] In operation 1330, the electronic device (100) can generate an image area selected by the user as a sticker image based on user input.

[0181] In operation 1340, the electronic device (100) can analyze the sticker image generated through the second artificial intelligence model (300).

[0182] In operation 1350, the electronic device (100) can identify an image containing the same object as an object contained within an image area selected by the user through the second artificial intelligence model (300).

[0183] In operation 1360, the electronic device (100) can correct a sticker image corresponding to an area selected by the user based on the identified image.

[0184] FIG. 14 is a diagram illustrating a process for obtaining a wallpaper image of an electronic device according to one or more embodiments.

[0185] According to one embodiment, the electronic device (100) can obtain multiple sticker images from multiple captured images containing the same object as the input image through the second artificial intelligence model, and obtain a wallpaper image using the obtained sticker images.

[0186] Here, the wallpaper image may include a background screen image displayed on the display (110). The wallpaper image may be an image for personalizing the desktop displayed on the electronic device (100). The wallpaper image is not limited thereto and may be referred to in various ways, such as a background image or a home screen image, but in the present disclosure, it will be collectively referred to as a wallpaper image.

[0187] Referring to FIG. 14, the electronic device (100) can acquire an image (hereinafter referred to as a third image) corresponding to an area (1421) selected by the user from one image (1410) among a plurality of captured images stored in the memory (120). The electronic device (100) can display the image (1420) including the area (1421) selected by the user through the display (110). For example, if the user selects (1421) a cat located on the left side from a captured image (1410) including two cats, the electronic device (100) can acquire a third image corresponding to the image of the cat on the left side.

[0188] According to one embodiment, the electronic device (100) may input the acquired third image into the second artificial intelligence model (300). The second artificial intelligence model (300) may identify a plurality of images including the same cat image as the cat image included in the input third image from the captured images stored in the memory (120). Thereafter, the second artificial intelligence model (300) may obtain a plurality of sticker images corresponding to the identified plurality of images.

[0189] According to one embodiment, the electronic device (100) may acquire a wallpaper image based on a plurality of acquired sticker images through the second artificial intelligence model (300). For example, as illustrated in FIG. 14, the electronic device (100) may acquire a wallpaper image (1430) including a plurality of cat sticker images in which the sizes and directions of the cat sticker images are randomly set based on a third image and a cat image identical to the cat image included in the third image.

[0190] In addition, if a user creates multiple different sticker images based on multiple captured images containing different objects, the electronic device (100) may obtain randomly arranged wallpapers based on the multiple different sticker images.

[0191] According to one embodiment, the electronic device (100) may provide the acquired wallpaper image (1430) as a lock screen image.

[0192] In one embodiment, the first AI model may be trained to handle all actions handled by the second AI model. For example, the first AI model may acquire multiple images containing objects identical to those contained in the input sticker image.

[0193] In one embodiment, the second AI model may be trained to process all actions processed by the first AI model. For example, the second AI model may obtain multiple sticker images related to the input image and emotional information.

[0194] FIGS. 15a, 15b, 15c and 15d are drawings illustrating a pattern lock method according to one or more embodiments.

[0195] According to one embodiment, when a wallpaper image (1430) is provided as a lock screen image, the electronic device (100) may provide a pattern lock method by using multiple sticker images included in the wallpaper as UI elements. Here, the UI element may mean an input element for interacting with a user by using multiple objects included in the UI. For example, when multiple objects are included in the UI, the electronic device (100) may identify each object as an input element and use the identified input elements as UI elements to interact with the user.

[0196] According to one embodiment, the electronic device (100) may identify each of a plurality of sticker images included in the acquired wallpaper image (1430) as a UI element. The electronic device (100) may receive a user input from the user to set a pattern lock method using each UI element.

[0197] According to one embodiment, the electronic device (100) may provide a pattern lock method through at least one of a dragging order for a path including a plurality of stickers, a touch order for a plurality of sticker images, and a number of touches for each of the plurality of stickers.

[0198] Referring to FIG. 15A, the electronic device (100) may provide a pattern lock method based on the order in which each UI element is dragged. For example, the electronic device (100) may provide a pattern lock method in which a first cat sticker image (1511) is dragged along a first path (1515), a second cat sticker image (1512) is dragged along a second path (1516), a third cat sticker image (1513) is dragged along a third path (1517), and finally a fourth cat sticker image (1514) is touched.

[0199] Referring to FIG. 15b, the electronic device (100) may provide a pattern lock method based on the order in which each UI element is touched. For example, the electronic device (100) may provide a pattern lock method in which the first cat sticker image (1518) is touched first, the second cat sticker image (1519) is touched second, the third cat sticker image (1520) is touched third, and the fourth cat sticker image (1521) is touched last.

[0200] Referring to FIG. 15c, the electronic device (100) may provide a pattern lock method based on the order in which different UI elements are touched. If the electronic device (100) acquires a wallpaper image using a plurality of different sticker images, the electronic device (100) may provide a pattern lock method based on the order in which the different sticker images are touched. For example, the electronic device (100) may provide a pattern lock method in which a fish sticker image (1522) is touched first, a puppy sticker image (1523) is touched second, and a cat sticker image (1524) is touched last.

[0201] Referring to FIG. 15d, the electronic device (100) may provide a pattern lock method based on the number of times different UI elements are touched. For example, the electronic device (100) may provide a pattern lock method in which a fish sticker image (1525) is touched once, a puppy sticker image (1526) is touched three times, and a cat sticker image (1527) is touched twice. Although FIG. 15d only considers the number of touches, the electronic device (100) may provide a pattern lock method that takes into account the number and order of touches.

[0202] FIG. 16 is a flowchart illustrating a process for providing a pattern lock method using a wallpaper of an electronic device according to one or more embodiments.

[0203] Referring to FIG. 16, at operation 1610, the electronic device (100) may receive a user input for selecting an image area to be used by the user in the captured image.

[0204] In operation 1611, the electronic device (100) may generate a sticker image from an image area selected by the user based on user input.

[0205] In operation 1612, the electronic device (100) can select at least one sticker image to be created as a wallpaper through the AI ​​wallpaper creation UI.

[0206] In operation 1613, the electronic device (100) may analyze at least one sticker image selected through the second artificial intelligence model (200).

[0207] In operation 1614, the electronic device (100) can compare at least one sticker image with a captured image stored in the memory (120) through the second artificial intelligence model (200).

[0208] In operation 1615, the electronic device (100) can identify whether there is a captured image that includes an object identical to an object included in at least one sticker image through the second artificial intelligence model (200).

[0209] In operation 1616, the electronic device (100) may not reflect additional sticker images on the wallpaper if there is no captured image containing the same object through the second artificial intelligence model (200) (S1615: N).

[0210] In operation 1617, the electronic device (100) can automatically generate an identified captured image as a sticker image if there is a captured image containing the same object through the second artificial intelligence model (200) (S1615: Y).

[0211] In operation 1618, the electronic device (100) can generate a wallpaper including a plurality of sticker images through the second artificial intelligence model (200).

[0212] In operation 1619, the electronic device (100) may display a UI for setting a screen lock method through the display (110).

[0213] At operation 1620, the electronic device (100) may receive user input from the user to select an AI wallpaper method.

[0214] In operation 1621, the electronic device (100) may receive a user input for selecting multiple sticker images from the user.

[0215] In operation 1622, the electronic device (100) can identify UI elements by analyzing objects within the wallpaper image.

[0216] In operation 1623, the electronic device (100) may provide a user lock screen through a pattern lock method based on a UI element.

[0217] FIG. 17 is a flowchart illustrating the overall operation process of an electronic device according to one or more embodiments.

[0218] Referring to FIG. 17, in operation 1710, the electronic device (100) may display a first UI including a first image and a plurality of emotional information corresponding to an area selected by the user.

[0219] In operation 1720, when one of the plurality of emotional information is selected, the electronic device (100) can input the selected emotional information and the first image into the learned first artificial intelligence model.

[0220] In operation 1730, the electronic device (100) may obtain and store a sticker image corresponding to the first image and the selected emotional information.

[0221] The method of displaying a first UI including a first image and a plurality of pieces of emotional information and obtaining a sticker image corresponding to the first image and the selected emotional information has been specifically described in the various embodiments described above, so a redundant description will be omitted.

[0222] The control method described in FIG. 17 can be performed by an electronic device (100) having the configuration of FIG. 2 described above, but is not necessarily limited thereto, and can also be performed by electronic devices having various configurations.

[0223] The various embodiments described above may be implemented as a single embodiment, or at least one embodiment may be combined with each other in whole or in part and implemented together in one device.

[0224] According to the various embodiments described above, a sticker image of the user can be created based on a photographed image taken by the user, and a personalized sticker image can be transmitted by adjusting the emotional level of the created sticker image.

[0225] Meanwhile, the various embodiments described above may be applied to a product as an embodiment alone, but at least some of the contents may be implemented in combination with other embodiments of the present disclosure.

[0226] The various embodiments described above can be implemented as software including instructions stored in a machine-readable storage medium that can be read by a machine (e.g., a computer). The device is a device that can call instructions stored in the storage medium and operate according to the called instructions, and may include an electronic device (e.g., electronic device (100)) according to the disclosed embodiments. When an instruction is executed by a processor, the processor can perform a function corresponding to the instruction directly or by using other components under the control of the processor. The instruction may include code generated or executed by a compiler or interpreter. The machine-readable storage medium can be provided in the form of a non-transitory computer-readable storage medium. Here, 'non-transitory' means that the storage medium does not contain a signal and is tangible, but does not distinguish between data being stored semi-permanently or temporarily in the storage medium.

[0227] Additionally, according to one embodiment of the present disclosure, the method according to the various embodiments described above may be provided as included in a computer program product.

[0228] Specifically, a non-transitory readable storage medium or a computer program product storing computer instructions for causing an operation to be performed, including a step of displaying a first UI including a first image and a plurality of emotional information corresponding to the selected area when an area is selected from an object image, and a step of inputting the selected emotional information and the first image into a learned first artificial intelligence model when one of the plurality of emotional information is selected, thereby obtaining and storing a sticker image corresponding to the first image and the selected emotional information.

[0229] The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or online through an application store (e.g., Play Store™). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily created in a storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0230] In addition, computer instructions or programs for performing the control methods of electronic devices according to the various embodiments described above may be stored in a non-transitory computer-readable medium. The computer instructions stored in such a non-transitory computer-readable medium, when executed by a processor of a specific device, cause the specific device to perform processing operations in the device according to the various embodiments described above. A non-transitory computer-readable medium refers to a medium that stores data semi-permanently and can be read by a device, rather than a medium that stores data for a short period of time, such as a register, cache, or memory. Specific examples of non-transitory computer-readable media may include a CD, DVD, hard disk, Blu-ray disk, USB, memory card, or ROM.

[0231] Although the preferred embodiments of the present disclosure have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above, and various modifications may be made by a person skilled in the art to which the present disclosure pertains without departing from the gist of the present disclosure as claimed in the claims, and such modifications should not be understood individually from the technical idea or prospect of the present disclosure.

Claims

1. In electronic devices, display; memory that stores instructions; and one or more processors including processing circuitry; One or more of the above processors, When the above instructions are executed individually or collectively, By inputting the images stored in the above memory into the learned first artificial intelligence model, multiple emotional information for each image is obtained, and when an area is selected from the object image displayed on the display, a first UI including a first image corresponding to the selected area and the multiple emotional information is displayed through the display. When one of the plurality of emotional information is selected, an image corresponding to the first image and the selected emotional information is identified among the images stored in the memory based on the plurality of emotional information for each image obtained, A second UI for selecting a sticker image and an emotion level generated based on the identified image is displayed through the display, An electronic device that, when the emotion level is selected through the second UI, inputs the selected emotion level and the sticker image into a learned second artificial intelligence model to obtain a sticker image corresponding to the selected emotion level.

2. In paragraph 1, The one or more processors, when the instructions are executed individually or collectively, Acquire a plurality of sticker images corresponding to different emotional levels for the identified image and the selected emotional information through the learned first artificial intelligence model, An electronic device that stores the plurality of sticker images obtained above in the memory.

3. In paragraph 1, The first artificial intelligence model learned above is: When image and emotion information are input, a candidate image corresponding to the input image is identified among the images stored in the memory, Identifying a target image corresponding to the input emotional information among the above-mentioned identified candidate images, An electronic device trained to convert the identified target image into a sticker image and output it.

4. In paragraph 1 The one or more processors, when the instructions are executed individually or collectively, When the emotional level is selected through the second UI, a third UI for adjusting the emotional level corresponding to the sticker image is displayed through the display. An electronic device that, when the emotional level is adjusted through the third UI, displays an image corresponding to the adjusted emotional level among a plurality of sticker images stored in the memory through the display.

5. In paragraph 1, The different emotional levels corresponding to the above selected emotional information include at least two levels from the first level to the fourth level, The first sticker image corresponding to the first level includes an object with a first expression changed to correspond to the emotional information, The second sticker image corresponding to the second level includes an object with a second expression changed to correspond to the emotional information, The third sticker image corresponding to the third level includes an object of the second expression changed to correspond to the emotional information and an effect image related to the emotional information. An electronic device, wherein the fourth sticker image corresponding to the fourth level includes a motion object of the second facial expression changed to correspond to the emotional information and an effect image related to the emotional information.

6. In paragraph 1, The one or more processors, when the instructions are executed individually or collectively, When an area is selected from the object image displayed on the above display, a second image corresponding to the selected area is identified, The identified second image is input into the learned second artificial intelligence model to obtain a sticker image corresponding to the second image and store it in the memory. The second artificial intelligence model learned above is, An electronic device that is trained to identify an image corresponding to the input image among images stored in the memory when an image is input, correct the input image based on the identified image, and obtain a sticker image based on the corrected image.

7. In paragraph 1, The second artificial intelligence model learned above is, An electronic device trained to correct the input image based on the identified image by at least one of complementing the main object included in the input image, deleting a sub-object excluding the main object, or correcting the resolution, and to obtain a sticker image based on the corrected image.

8. In paragraph 1, The one or more processors, when the instructions are executed individually or collectively, Using the second artificial intelligence model learned above, GK obtains a wallpaper image including multiple sticker images, An electronic device that provides the obtained wallpaper image as a lock screen image.

9. In paragraph 8, The one or more processors, when the instructions are executed individually or collectively, An electronic device that provides a pattern lock method by using the plurality of sticker images included in the wallpaper as UI elements when the above wallpaper image is provided as the lock screen image.

10. In paragraph 9, The one or more processors, when the instructions are executed individually or collectively, An electronic device providing the pattern lock method through at least one of a dragging order for a path including the plurality of stickers, a touch order for the plurality of sticker images, or a number of touches for each of the plurality of stickers.

11. A method for controlling an electronic device, A step of inputting images stored in the electronic device into a learned first artificial intelligence model to obtain multiple emotional information for each image; When an area is selected in an object image, a step of displaying a first UI including a first image corresponding to the selected area and the plurality of emotional information; When one of the plurality of emotional information is selected, a step of identifying an image corresponding to the first image and the selected emotional information among images stored in the electronic device based on the plurality of emotional information for each image obtained; A step of displaying a second UI for selecting a sticker image and an emotion level generated based on the identified image; and A control method comprising: when the emotion level is selected through the second UI, a step of inputting the selected emotion level and the sticker image into a learned second artificial intelligence model to obtain a sticker image corresponding to the selected emotion level.

12. In paragraph 11, A step of obtaining a plurality of sticker images corresponding to different emotional levels for the identified image and the selected emotional information through the learned first artificial intelligence model; and A control method comprising: a step of storing the acquired plurality of sticker images in the electronic device; 13. In paragraph 11, The first artificial intelligence model learned above is: A control method, wherein when image and emotional information are input, a candidate image corresponding to the input image is identified among images stored in the electronic device, a target image corresponding to the input emotional information is identified among the identified candidate images, and the identified target image is converted into a sticker image and output.

14. In Article 11 When the emotion level is selected through the second UI, a step of displaying a third UI for adjusting the emotion level corresponding to the sticker image; and A control method comprising: when the emotional level is adjusted through the third UI, displaying an image corresponding to the adjusted emotional level among a plurality of sticker images stored in the electronic device.

15. A non-transitory computer-readable storage medium storing computer instructions that, when executed by a processor of an electronic device, cause the electronic device to perform an operation, the operation comprising: A step of inputting images stored in the electronic device into a learned first artificial intelligence model to obtain multiple emotional information for each image; When an area is selected in an object image, a step of displaying a first UI including a first image corresponding to the selected area and the plurality of emotional information; When one of the plurality of emotional information is selected, a step of identifying an image corresponding to the first image and the selected emotional information among images stored in the electronic device based on the plurality of emotional information for each image obtained; A step of displaying a second UI for selecting a sticker image and an emotion level generated based on the identified image; and A non-transitory computer-readable storage medium, comprising: a step of inputting the selected emotion level and the sticker image into a learned second artificial intelligence model when the emotion level is selected through the second UI, thereby obtaining a sticker image corresponding to the selected emotion level.

Images