Electronic device and control method therefor

Abstract

Disclosed is an electronic device. The electronic device comprises a display, a memory for storing one or more instructions, and one or more processors. The one or more processors execute the one or more instructions to: acquire a text prompt on the basis of information about articles stored in a repository; acquire a personalized UI image by inputting the text prompt, UI layout information, and color information related to the electronic device to a trained artificial intelligence model; and provide the acquired personalized UI image via the display.

Description

Electronic device and method of controlling the same

[0001] The present disclosure relates to an electronic device and a method for controlling the same, and more particularly, to an electronic device providing a UI screen and a method for controlling the same.

[0002] Advances in electronic technology have led to the development and proliferation of various types of electronic devices. In particular, displays have been increasingly integrated into home appliances like refrigerators.

[0003] Typically, home appliances equipped with displays offer a variety of UI screens, such as wallpaper. However, users face the inconvenience of having to go through numerous manipulations to customize the UI screens to their liking.

[0004] An electronic device according to one or more embodiments includes a display, a memory storing one or more commands, and one or more processors, wherein the one or more processors obtain a text prompt based on information about an item stored in a storage by executing the one or more commands, input the text prompt, UI layout information, and color information related to the electronic device into a learned artificial intelligence model to obtain a personalized UI (User Interface) image, and provide the obtained personalized UI image through the display.

[0005] According to one or more embodiments, the electronic device is implemented as a refrigerator, and the one or more processors obtain the text prompt based on information about food stored in the storage by executing the one or more commands, and input the text prompt, the UI layout information, and the panel color information of the refrigerator into a learned artificial intelligence model to obtain a personalized background image, and the information about food stored in the storage may include at least one of cooking information using food stored in the storage, information about the time of receipt of food stored in the storage, or expiration date information of food stored in the storage.

[0006] According to one or more embodiments, the one or more processors may, by executing the one or more instructions, identify expiration date information of each food stored in the storage, identify a dish that can be cooked with food whose expiration date is within a threshold period based on the identified expiration date information, and obtain the text prompt based on the identified cookable dish.

[0007] According to one or more embodiments, the one or more processors may, by executing the one or more instructions, calculate a first score corresponding to the first dish based on the total number of food items required for the first dish, the number of food items available in the storage among the total required food items, and the number of food items stored in the storage items whose expiration dates are within a threshold period, for the identified first dish that can be cooked, calculate a second score corresponding to the second dish based on the total number of food items required for the second dish, the number of food items available in the storage among the total required food items, and the number of food items stored in the storage items whose expiration dates are within a threshold period, and obtain the text prompt based on a dish corresponding to a relatively larger score among the first score and the second score.

[0008] According to one or more embodiments, the one or more processors may, by executing the one or more commands, identify expiration date information of each food stored in the storage, and obtain the text prompt for the food whose expiration date is within a threshold period based on the identified expiration date information and reminder information for each remaining expiration date of the food. The reminder information for each remaining expiration date of the food may include status information, type information, and style information of a reminder image for each remaining expiration date of the food.

[0009] According to one or more embodiments, the one or more processors may, by executing the one or more instructions, identify a priority for the stored food based on information about the time of receipt of the food stored in the storage, and obtain the text prompt based on the identified priority.

[0010] According to one or more embodiments, the one or more processors can obtain the personalized UI image by inputting the text prompt, the UI layout information, color information related to the electronic device, and user context information into a learned artificial intelligence model by executing the one or more commands.

[0011] According to one or more embodiments, the one or more processors may, by executing the one or more instructions, obtain a text prompt related to an image of an item stored in the storage based on information about the item stored in the storage and a preset rule.

[0012] According to one or more embodiments, the learned artificial intelligence model may be implemented as a stable diffusion model. The stable diffusion model may, when the text prompt, the UI layout information, the color information, and the noise image are input, predict noise based on the input data, and obtain the personalized UI image based on the difference between the noise image and the predicted noise.

[0013] According to one or more embodiments, the one or more processors may, by executing the one or more commands, provide a UI screen for selecting at least one of a category, a style, and a color tone, obtain the color information based on the color tone selected through the UI screen, and input the category information and style information selected through the UI screen together with the text prompt, the UI layout information, and the color information into the learned artificial intelligence model to obtain the personalized UI image.

[0014] A method for controlling an electronic device according to one or more embodiments may include: obtaining a text prompt based on information about an item stored in a storage; obtaining a personalized UI (User Interface) image by inputting the text prompt, UI layout information, and color information related to the electronic device into a learned artificial intelligence model; and providing the obtained personalized UI image.

[0015] A non-transitory computer-readable medium storing computer instructions that, when executed by a processor of an electronic device according to one or more embodiments, cause the electronic device to perform an operation, the operation may include: obtaining a text prompt based on information about an item stored in a storage; obtaining a personalized UI (User Interface) image by inputting the text prompt, UI layout information, and color information related to the electronic device into a learned artificial intelligence model; and providing the obtained personalized UI image.

[0016] FIG. 1 is a drawing for explaining an implementation example of an electronic device according to one or more embodiments.

[0017] FIG. 2A is a block diagram showing the configuration of an electronic device according to one embodiment.

[0018] FIG. 2b is a block diagram specifically illustrating a configuration of an electronic device according to one or more embodiments.

[0019] FIG. 3 is a flowchart illustrating a method of controlling an electronic device according to one or more embodiments.

[0020] FIG. 4 is a flowchart illustrating a method for controlling an electronic device according to one or more embodiments.

[0021] FIG. 5 is a flowchart illustrating a method for controlling an electronic device according to one or more embodiments.

[0022] FIG. 6 is a diagram illustrating a method for obtaining a background image according to one or more embodiments.

[0023] FIG. 7 is a diagram illustrating a method for obtaining a background image according to one or more embodiments.

[0024] FIG. 8 is a diagram illustrating a method for obtaining a background image according to one or more embodiments.

[0025] FIG. 9 is a diagram illustrating a method for obtaining a background image according to one or more embodiments.

[0026] FIG. 10 is a diagram illustrating a learning method of a generative AI model according to one or more embodiments.

[0027] FIG. 11 is a diagram illustrating a method for generating a background image of a generative AI model according to one or more embodiments.

[0028] FIG. 12a is a drawing for explaining a method of providing a UI screen according to one or more embodiments.

[0029] FIG. 12b is a drawing for explaining a method of providing a UI screen according to one or more embodiments.

[0030] FIG. 12c is a drawing for explaining a method of providing a UI screen according to one or more embodiments.

[0031] FIG. 13 is a drawing for explaining a method of providing a UI screen according to one or more embodiments.

[0032] FIG. 14a is a drawing for explaining a method of providing a UI screen according to one or more embodiments.

[0033] FIG. 14b is a drawing for explaining a method of providing a UI screen according to one or more embodiments.

[0034] The terms used in this specification will be briefly explained, and the present disclosure will be described in detail.

[0035] The terms used in the embodiments of this disclosure have been selected from widely used, current terms, taking into account the functions of this disclosure. However, these terms may vary depending on the intentions or cases of those skilled in the art, the emergence of new technologies, etc. Furthermore, in certain cases, terms may be arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the description of the relevant disclosure. Therefore, the terms used in this disclosure should not be defined simply as names of terms, but rather based on the meanings of the terms and the overall content of this disclosure.

[0036] In this specification, expressions such as “has,” “can have,” “includes,” or “may include” indicate the presence of a feature (e.g., a number, function, operation, or component such as a part), and do not exclude the presence of additional features.

[0037] In this disclosure, expressions such as “A or B,” “at least one of A and / or B,” or “one or more of A or / and B” can include all possible combinations of the listed items. For example, “A or B,” “at least one of A and B,” or “at least one of A or B” can all refer to cases where (1) only A is included, (2) only B is included, or (3) both A and B are included.

[0038] As used herein, the expressions “first,” “second,” “first,” or “second,” etc., may describe various components, regardless of order and / or importance, and are only used to distinguish one component from another, but do not limit the components.

[0039] When it is said that a component (e.g., a first component) is “operatively or communicatively coupled with / to” or “connected to” another component (e.g., a second component), it should be understood that the component may be directly coupled to the other component, or may be connected through another component (e.g., a third component).

[0040] The expression "configured to" as used in the present disclosure may be used interchangeably with, for example, "suitable for," "having the capacity to," "designed to," "adapted to," "made to," or "capable of." The term "configured to" may not necessarily mean only "specifically designed to" in terms of hardware.

[0041] In some contexts, the phrase "a device configured to" may mean that the device, in conjunction with other devices or components, is "capable of" performing A, B, and C. For example, the phrase "a processor configured (or set) to perform A, B, and C" may refer to a dedicated processor (e.g., an embedded processor) for performing those operations, or a general-purpose processor (e.g., a CPU or application processor) that can perform those operations by executing one or more software programs stored in a memory device.

[0042] Singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, terms such as "comprise" or "consist of" are intended to indicate the presence of a feature, number, step, operation, component, part, or combination thereof described in the specification, but should be understood not to preclude the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0043] In the embodiments, a "module" or "part" performs at least one function or operation and may be implemented as hardware or software, or as a combination of hardware and software. Furthermore, a plurality of "modules" or "parts" may be integrated into at least one module and implemented as at least one processor (not shown), excluding any "module" or "part" that needs to be implemented as specific hardware.

[0044] Meanwhile, the various elements and areas in the drawings are schematically drawn. Therefore, the technical concept of the present invention is not limited by the relative sizes or spacing depicted in the attached drawings.

[0045] An embodiment of the present disclosure will be described in more detail with reference to the attached drawings below.

[0046] FIG. 1 is a drawing for explaining an implementation example of an electronic device according to one or more embodiments.

[0047] According to FIG. 1, the electronic device (100) may be implemented as a refrigerator, but is not limited thereto. For example, the electronic device (100) may be implemented as various devices capable of storing items (e.g., food) within a preset space, such as a refrigerator, kimchi refrigerator, wine refrigerator, oven, microwave oven, washing machine, dryer, etc.

[0048] For example, the electronic device (100) may be an Internet of Things (IoT) device that supports a WiFi module.

[0049] The electronic device (100) can download and install various applications from a server providing the applications. Applications are software that users directly use on the operating system (OS) and may be provided in the form of an icon interface on the UI screen (10) of the electronic device (100). For example, the application may communicate with a related server to perform related operations. For example, the server may be implemented as a cloud server, but is not limited thereto.

[0050] For example, an application provided on a UI screen (10) of an electronic device (100) may include an application (11) for generating images. The image generating application (11) may generate various images, including wallpaper, provided through a display (110) of the electronic device (100).

[0051] For example, a user may log in to a server associated with an image generation application (11) through a user account. In this case, the electronic device (100) may communicate with the server based on the logged-in user account.

[0052] As an example, the electronic device (100) may provide a UI screen, for example, a wallpaper image, based on a user account logged in through an image generation application (11).

[0053] FIG. 2A is a block diagram showing the configuration of an electronic device according to one embodiment.

[0054] According to FIG. 2a, the electronic device (100) includes a display (110), a memory (120), and one or more processors (130).

[0055] The electronic device (100) may be implemented as a device capable of storing items in a storage unit. As an example, the electronic device (100) may be implemented as a device capable of storing food in a storage unit, such as a refrigerator, kimchi refrigerator, wine refrigerator, oven, or microwave oven, but is not limited thereto.

[0056] The display (110) may be implemented as a display including a display that includes a self-luminous element. For example, the display (110) may be implemented as a display in various forms such as an OLED (Organic Light Emitting Diodes) display, an LED (Light Emitting Diodes), a micro LED, a Mini LED, a PDP (Plasma Display Panel), a QD (Quantum dot) display, a QLED (Quantum dot light-emitting diodes), etc. The display (110) may also include a driving circuit, a backlight unit, etc., which may be implemented in a form such as an a-si TFT, an LTPS (low temperature poly silicon) TFT, an OTFT (organic TFT), etc. In one example, a touch sensor that detects a touch operation in the form of a touch film, a touch sheet, a touch pad, etc. may be disposed on the front of the display (110) so as to be implemented so as to detect various types of touch inputs. For example, the display (110) can detect various types of touch inputs, such as a touch input by a user's hand, a touch input by an input device such as a stylus pen, and a touch input by a specific electrostatic material. Here, the input device can be implemented as a pen-type input device that can be referred to by various terms such as an electronic pen, a stylus pen, an S-pen, etc. According to an example, the display (110) can be implemented as a flat display, a curved display, a flexible display that can be folded or / and rolled, etc.

[0057] The memory (120) can store data required for various embodiments. Depending on the purpose of data storage, the memory (120) may be implemented in the form of memory embedded in the electronic device (100') or may be implemented in the form of memory that can be attached or detached from the electronic device (100). For example, data for operating the electronic device (100) may be stored in a memory embedded in the electronic device (100'), and data for expanding the functions of the electronic device (100) may be stored in a memory that can be attached or detached from the electronic device (100). Meanwhile, in the case of memory embedded in the electronic device (100), it may be implemented as at least one of volatile memory (e.g., dynamic RAM (DRAM), static RAM (SRAM), or synchronous dynamic RAM (SDRAM)), non-volatile memory (e.g., one time programmable ROM (OTPROM), programmable ROM (PROM), erasable and programmable ROM (EPROM), electrically erasable and programmable ROM (EEPROM), mask ROM, flash ROM, flash memory (e.g., NAND flash or NOR flash), hard drive, or solid state drive (SSD). In addition, in the case of memory that can be attached or detached to the electronic device (100'), it may be implemented as at least one of memory cards (e.g., compact flash (CF), secure digital (SD), micro secure digital (Micro-SD), mini secure digital (Mini-SD), extreme digital (xD), multi-media card (MMC), etc.), external memory that can be connected to a USB port (e.g., USB memory), etc. It can be implemented in the form of.

[0058] As an example, the memory (120) may store a computer program including at least one instruction or instructions for controlling the electronic device (100).

[0059] In another example, the memory (120) can store data required for various embodiments, such as preset rules used to generate text prompts, images acquired through a camera (140), information about items (e.g., food) stored in a storage unit, etc.

[0060] In the above-described embodiment, it has been described that various data are stored in the external memory (120) of the processor (130), but at least some of the above-described data may be stored in the internal memory of the processor (130) according to an implementation example of at least one of the electronic device (100) or the processor (130).

[0061] One or more processors (130) control the overall operation of the electronic device (100). Specifically, one or more processors (130) may be connected to each component of the electronic device (100) to control the overall operation of the electronic device (100). For example, one or more processors (130) may be operatively connected to the memory (120) to control the overall operation of the electronic device (100). One or more processors (130) may be configured as one or more processors.

[0062] One or more processors (130) may perform operations of the electronic device (100) according to various embodiments by executing at least one instruction stored in the memory (120).

[0063] In one example, functions related to artificial intelligence according to the present disclosure may be operated through a processor and memory of an electronic device.

[0064] One or more processors (130) may be composed of one or more processors. In this case, one or more processors may include at least one of a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and an NPU (Neural Processing Unit), but are not limited to the examples of the processors described above.

[0065] A CPU is a general-purpose processor capable of performing not only general calculations but also artificial intelligence calculations. Its multi-layered cache structure allows for the efficient execution of complex programs. CPUs are advantageous for serial processing, enabling organic linking of previous and subsequent calculation results through sequential calculations. A general-purpose processor is not limited to the examples described above, except where specifically designated as a CPU.

[0066] A GPU is a processor designed for large-scale computations, such as floating-point operations used in graphics processing. It integrates a large number of cores to perform large-scale computations in parallel. In particular, GPUs may be advantageous over CPUs in parallel processing methods, such as convolution operations. Furthermore, GPUs can be used as coprocessors to supplement the functions of CPUs. Processors for large-scale computations are not limited to the examples described above, except in cases where they are specifically referred to as GPUs.

[0067] An NPU is a processor specialized in artificial intelligence operations using artificial neural networks, and each layer of the artificial neural network can be implemented in hardware (e.g., silicon). Since NPUs are designed specifically according to the company's specifications, they have less freedom than CPUs or GPUs, but can efficiently process the artificial intelligence operations requested by the company. Meanwhile, as a processor specialized in artificial intelligence operations, an NPU can be implemented in various forms, such as a Tensor Processing Unit (TPU), an Intelligence Processing Unit (IPU), or a Vision Processing Unit (VPU). Except as specifically designated as an NPU, the artificial intelligence processor is not limited to the examples described above.

[0068] Additionally, one or more processors (130) may be implemented as a System on Chip (SoC). In this case, the SoC may further include, in addition to one or more processors (130), a memory (110), and a network interface such as a bus for data communication between the processor (130) and the memory (110).

[0069] When a plurality of processors are included in a SoC (System on Chip) included in an electronic device (100), the electronic device (100) may perform operations related to artificial intelligence (e.g., operations related to learning or inference of an artificial intelligence model (or neural network model)) by using some of the plurality of processors. For example, the electronic device may perform operations related to artificial intelligence by using at least one of a GPU, an NPU, a VPU, a TPU, and a hardware accelerator specialized in artificial intelligence operations such as convolution operations and matrix multiplication operations among the plurality of processors. However, this is only one or more embodiments, and it is of course possible to process operations related to artificial intelligence by using a CPU or a general-purpose processor.

[0070] Additionally, the electronic device (100) can perform operations related to functions related to artificial intelligence by utilizing multiple cores (e.g., dual cores, quad cores, etc.) included in a single processor. In particular, the electronic device can perform artificial intelligence operations, such as convolution operations and matrix multiplication operations, in parallel by utilizing multiple cores included in the processor.

[0071] One or more processors (130) are controlled to process input data according to predefined operation rules or artificial intelligence models (or artificial intelligence models) stored in memory (110). The predefined operation rules or artificial intelligence models are characterized by being created through learning.

[0072] Here, "created through learning" means that a predefined set of behavioral rules or an artificial intelligence model with desired characteristics is created by applying a learning algorithm to a large number of learning data. This learning may be performed on the device itself, where the artificial intelligence according to the present disclosure is performed, or through a separate server / system.

[0073] An artificial intelligence model may be composed of multiple neural network layers. At least one layer has at least one weight value and performs its own operation through the operation result of the previous layer and at least one defined operation. Examples of neural networks include a convolutional neural network (CNN), a recurrent neural network (RNN), a deep neural network (DNN), a restricted boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), deep Q-networks, and a transformer. The neural networks in the present disclosure are not limited to the above-described examples unless otherwise specified.

[0074] A learning algorithm is a method for training a target device (e.g., a robot) using a plurality of learning data sets so that the target device can make decisions or predictions on its own. Examples of learning algorithms include supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning. The learning algorithms in the present disclosure are not limited to the aforementioned examples unless otherwise specified. For convenience of explanation, one or more processors (130) will be referred to as "processors (130)" below.

[0075] FIG. 2b is a block diagram specifically illustrating a configuration of an electronic device according to one or more embodiments.

[0076] According to FIG. 2b, the electronic device (100') may include a display (110), a memory (120), one or more processors (130), a camera (140), a user interface (150), a communication interface (160), and a speaker (170). Among the configurations illustrated in FIG. 2b, a detailed description of configurations that overlap with those illustrated in FIG. 2a will be omitted.

[0077] The camera (140) can be turned on and perform photography based on a preset event. The camera (140) can convert the captured image into an electrical signal and generate image data based on the converted signal. For example, the camera (140) can include at least one of a general (or basic) camera and an ultra-wide-angle camera. For example, the camera (140) can be installed at a location capable of photographing the interior of a storage room. For example, multiple cameras can be installed at locations capable of photographing different aspects of the storage room.

[0078] The user interface (150) may be implemented as a device such as a button, a touch pad, a mouse, and a keyboard, or as a touch screen that can also perform the display function and operation input function described above.

[0079] In one example, the user interface (150) may receive user input to select at least one of a category, style, and color tone for generating a background image.

[0080] It goes without saying that the communication interface (160) can be implemented as various interfaces depending on the implementation example of the electronic device (100'). For example, the communication interface (140) can communicate with an external device, an external storage medium (e.g., a USB memory), an external server (e.g., a web hard drive), etc. through a communication method such as Bluetooth, AP-based Wi-Fi (Wireless LAN network), Zigbee, wired / wireless LAN (Local Area Network), WAN (Wide Area Network), Ethernet, IEEE 1394, HDMI (High-Definition Multimedia Interface), USB (Universal Serial Bus), MHL (Mobile High-Definition Link), AES / EBU (Audio Engineering Society / European Broadcasting Union), optical, coaxial, etc. In one example, the communication interface (160) can communicate with another electronic device, an external server, and / or a remote control device.

[0081] The speaker (170) may be configured to output various audio data as well as various notification sounds or voice messages. The processor (130) may control the speaker (170) to output feedback or various notifications in audio format according to various embodiments of the present disclosure.

[0082] In addition, the electronic device (100') may include sensors, microphones, etc., depending on the implementation example.

[0083] Sensors may include various types of sensors, such as touch sensors, proximity sensors, acceleration sensors (or gravity sensors), geomagnetic sensors, gyro sensors, pressure sensors, position sensors, distance sensors, light sensors, etc.

[0084] A microphone is a device configured to receive user voice or other sounds and convert them into audio data. However, according to another embodiment, the electronic device (100') may receive user voice input from an external device through the communication interface (160). For example, the microphone may receive user voice containing various information necessary for generating a personalized UI image.

[0085] FIG. 3 is a flowchart illustrating a method of controlling an electronic device according to one or more embodiments.

[0086] According to FIG. 3, in operation 310, the electronic device (100) can obtain information about items stored in a storage.

[0087] For example, the electronic device (100) may analyze an image captured by the camera (140) to obtain information about items stored in the storage. For example, the electronic device (100) may extract features from the captured image, identify objects based on the extracted features, and classify the identified objects to obtain information about items stored in the storage. For example, the electronic device (100) may input the captured image into an AI model trained to classify objects to obtain information about items stored in the storage.

[0088] For example, the electronic device (100) can obtain information about stored items based on information entered by the user. For example, the user can enter information about food items being stored in the warehouse, expiration date information, etc. through the UI screen.

[0089] In addition, the electronic device (100) can obtain information about items stored in a storage unit by using various conventional refrigerator entry / exit recognition technologies.

[0090] In operation 320, the electronic device (100) may obtain a text prompt based on information about the acquired item.

[0091] A text prompt (hereinafter, "prompt") may be a text input that instructs an AI model to perform a desired task. For example, the prompt may be in the form of text indicating a UI image based on acquired product information. For example, the electronic device (100) may obtain a prompt for generating a UI image based on a preset rule. Rule-based prompt generation may be a method of generating a prompt using predefined rules and patterns.

[0092] In operation 330, the electronic device (100) may input a text prompt, UI layout information, and color information into a learned artificial intelligence model to obtain a personalized UI image (or a customized UI image). In one example, the electronic device (100) may input a text prompt, UI layout information, color information, and context information into the learned artificial intelligence model to obtain a personalized UI image. For example, the context information may include at least one of user context information and context information of the electronic device (100).

[0093] UI layout can refer to the way design elements of a user interface are arranged and structured. For example, UI layout information can include information about the structure of a UI screen, the position, size, and spacing of UI elements, etc. For example, UI layout information can include UI asset information. UI asset information can refer to resources used to design and build a user interface. For example, UI assets can include UI component elements such as fonts, buttons, icons, and tooltips.

[0094] For example, the electronic device (100) can provide a plurality of UI layout information through a UI screen and identify UI layout information selected by a user.

[0095] In one example, the electronic device (100) may identify UI layout information based on user selection, user preference information, and / or user usage history. For example, user preference information may include visual preferences indicating what the user prefers in terms of visual aspects. For example, visual preferences may include a person's preferences for visual content, such as design style, color, layout, image style, font style, font size, etc.

[0096] The color information may include at least one of color information selected by the user or panel color information of the electronic device (100). For example, the color information may include color palette information. A color palette refers to a set of colors. For example, a color palette may include color information such as primary colors, secondary colors, and contrast colors.

[0097] Contextual information can include various types of information, such as device information, time information, environmental information, location information, social information, and physiological status information. For example, device information may include information about the device used by the user, such as the device type, screen size, and operating system. For example, time information may include time-related information, such as the current time, day of the week, and season. Environmental information may include information about the surrounding environment, such as weather information, temperature information, and lighting information. Social information may include social network-related information, such as the user's social media activity, friends list, and contact information. Physiological status information may include health-related data or physiological information, such as heart rate, sleep patterns, and activity level, obtained through wearable devices.

[0098] A trained AI model can be implemented as a generative AI model. Generative AI models generate new content based on given input, and can generate various types of data, such as text, images, and voice. Generative AI models can be pre-trained on large amounts of data (e.g., text data) and then fine-tuned for various tasks. Generative AI models can learn patterns in input data and generate new data based on deep learning architectures such as Recurrent Neural Networks (RNNs), Long Short-Term Memory (LSTMs), Gated Recurrent Units (GRUs), or Transformers. For example, a generative AI model can be implemented as a Stable Diffusion model. However, this is not limited to this, and generative AI models can be implemented using various AI models, such as Generative Adversarial Networks (GANs), Style Transfer Models, AI Content Generators, Transformer-Based Models, and Genetic Algorithms for Layout Optimization.

[0099] For example, when an AI model is implemented on a server, the electronic device (100) can transmit prompts, UI layout information, and color information to the server and receive personalized UI images from the server.

[0100] In another example, when the AI ​​model is implemented as an on-device model included in the electronic device (100), the electronic device (100) can obtain a personalized UI image by inputting prompts, UI layout information, and color information into the AI ​​model.

[0101] In one example, the prompt, UI layout information, and color information may be obtained from the electronic device (100). In another example, at least one of the prompt, UI layout information, and color information may be obtained from an external server.

[0102] In operation 330, the electronic device (100) may provide the acquired personalized UI image.

[0103] For example, when multiple personalized UI images are acquired, the electronic device (100) may provide a UI screen for selecting one of the multiple personalized UI images.

[0104] For example, the electronic device (100) may provide a preview screen of multiple personalized UI images. For example, the electronic device (100) may sequentially provide the preview images of multiple personalized UI images in order of priority. For example, the preview images of multiple personalized UI images may be provided simultaneously on a single screen.

[0105] In one example, the electronic device (100) can automatically play multiple personalized UI images sequentially. For example, the electronic device (100) can automatically play multiple personalized UI images sequentially according to priority.

[0106] Meanwhile, in Fig. 3, the order is mapped for all steps for convenience of explanation, but it is of course not necessarily limited to the order of steps that are not related to the order or can be performed in parallel.

[0107] FIG. 4 is a flowchart illustrating a method for controlling an electronic device according to one or more embodiments.

[0108] According to FIG. 4, in operation 410, the electronic device (100) can obtain information about food stored in the storage.

[0109] For example, information about food stored in a storage unit may include at least one of information about the type of food stored in the storage unit, information about the time of receipt of food stored in the storage unit, or information about the expiration date of food stored in the storage unit.

[0110] In operation 420, the electronic device (100) can obtain cooking information using the stored food based on information about the food stored in the storage.

[0111] For example, the electronic device (100) may identify a dish that can be prepared using food with a shelf life within a critical period, based on the shelf life information of each food item stored in the storage unit. For example, the electronic device (100) may identify multiple dishes that can be prepared using food with a shelf life within a critical period.

[0112] For example, the electronic device (100) can calculate a score for each of a plurality of dishes that can be cooked with food whose shelf life is within a critical period, and identify priorities for the plurality of dishes based on the calculated scores.

[0113] For example, the electronic device (100) may calculate a first score corresponding to a first dish among a plurality of cookable dishes based on the total number of foods required for the first dish, the number of foods available in the storage among the total required foods, and the number of foods stored in the storage whose expiration dates are within a critical period. In addition, the electronic device (100) may calculate a second score corresponding to a second dish among a plurality of cookable dishes based on the total number of foods required for the second dish, the number of foods available in the storage among the total required foods, and the number of foods stored in the storage whose expiration dates are within a critical period.

[0114] In operation 430, the electronic device (100) may obtain a prompt based on the obtained cooking information.

[0115] For example, the electronic device (100) can compare a first score corresponding to a first dish and a second score corresponding to a second dish and obtain a prompt for the dish corresponding to a relatively larger score.

[0116] For example, the electronic device (100) may obtain a prompt for at least one dish having a preset score or higher. For example, if both the first score and the second score are above the preset score, the electronic device (100) may obtain a prompt for each of the first and second dishes.

[0117] In one example, when prompts for a first dish and a second dish are obtained, the electronic device (100) can identify priorities for the first dish and the second dish based on a first score corresponding to the first dish and a second score corresponding to the second dish, and determine the display order of the personalized background images obtained based on the prompts based on the identified priorities.

[0118] In one example, the electronic device (100) may obtain a prompt for at least one of the first dish and the second dish based on a preset number of dishes for obtaining a prompt. For example, the preset number of dishes may be set during manufacturing, or may be set or changed by the user. For example, if the preset number of dishes for generating a prompt is 1, only a prompt for a dish with a relatively high score among the first and second dishes may be obtained. For example, if the preset number of dishes for generating a prompt is 2, a prompt for each of the first and second dishes may be obtained. In one example, the preset number of dishes for obtaining a prompt may be automatically updated based on the user's usage history. For example, if the preset number is 2, prompts for each of the first and second dishes are generated and a background image is generated, but if the user always prefers only the background image based on the dish with the highest score, the preset number may be updated to 1.

[0119] In one example, the number of dishes for which prompts are obtained may be determined based on the scores of each of the multiple dishes. For example, if the score difference between the first and second dishes is greater than or equal to a threshold, only the prompt corresponding to the first dish may be obtained. For example, if the score difference between the first and second dishes is less than or equal to a threshold, the prompts corresponding to the first and second dishes may be obtained. The threshold may be set during manufacturing, or may be set or changed by the user. Alternatively, the threshold may be automatically updated based on the user's usage history. For example, if the score difference between the first and second dishes is less than a threshold, prompts for both dishes are generated and background images are generated, but the user always prefers only the background image based on the dish with the highest score, the threshold may be updated to decrease.

[0120] Although the above-described embodiment described scores being calculated for two dishes, it is of course possible to calculate scores for three or more dishes. For example, in operation 420, information on three or more dishes may be obtained based on information on food stored in the storage. An embodiment in which a prompt is obtained when three or more dishes are obtained may be identical or similar to an embodiment in which a prompt is obtained when two dishes are obtained. For example, depending on the embodiment, a prompt may be obtained only for the dish with the highest score among three dishes, a prompt may be obtained only for two dishes excluding the dish with the lowest score, or a prompt may be obtained for all three dishes.

[0121] According to one embodiment, the user's context and the context of the electronic device (100) may be additionally reflected when calculating a score for a dish. For example, contexts such as the user's preference, the current time, and the current weather may be reflected as weights. For example, a weight may be reflected for a dish type based on at least one of the user's preference, the current time, or the current weather. For example, among a plurality of dishes, a dish with a higher user preference may be given a higher weight. For example, among a plurality of dishes, a dish appropriate for the current time (e.g., breakfast time) may be given a higher weight. For example, among a plurality of dishes, a dish appropriate for the current weather (e.g., rainy and cold weather) may be given a higher weight.

[0122] In operation 440, the electronic device (100) may input a prompt, UI layout information, and panel color information of the refrigerator into the learned artificial intelligence model to obtain a personalized background image.

[0123] In operation 450, the electronic device (100) can provide the acquired personalized background image through the display (110) of the refrigerator.

[0124] Meanwhile, in Fig. 4, the order is mapped for all steps for convenience of explanation, but it is of course not necessarily limited to the order of steps that are not related to the order or can be performed in parallel.

[0125] FIG. 5 is a flowchart illustrating a method for controlling an electronic device according to one or more embodiments.

[0126] According to FIG. 5, in operation 510, the electronic device (100) can obtain information about food stored in a storage.

[0127] In operation 520, the electronic device (100) can identify the expiration date information of food stored in the storage.

[0128] In operation 530, the electronic device (100) may obtain a prompt for food whose expiration date is within a threshold period based on the identified expiration date information and reminder information for each remaining expiration date of the food.

[0129] For example, reminder information for each remaining shelf life of a food product may include status information, type information, and style information of a reminder image for each remaining shelf life of the food product.

[0130] In one example, the electronic device (100) may generate a prompt based on food text and expiration date information corresponding to food information. For example, a prompt related to an expiration date reminder may be generated by applying food text such as "tomato" and remaining expiration date information such as "2 days" to a preset rule. For example, the electronic device (100) may generate a prompt in the format "a {Type} of {Status} {Food Text} {Style}."

[0131] In operation 540, the electronic device (100) can input the prompt, UI layout information, and panel color information of the refrigerator into the learned artificial intelligence model to obtain a personalized background image.

[0132] In operation 550, the electronic device (100) can provide the acquired personalized background image through the display (110) of the refrigerator.

[0133] Meanwhile, in Fig. 5, the order is mapped for all steps for convenience of explanation, but it is of course not necessarily limited to the order of steps that are not related to the order or can be performed in parallel.

[0134] FIG. 6 is a diagram illustrating a method for obtaining a background image according to one or more embodiments.

[0135] According to one embodiment illustrated in FIG. 6, the electronic device (100) can obtain a background image based on user input.

[0136] According to one embodiment illustrated in FIG. 6, the electronic device (100) may include a prompt generator (610), a generative AI model (620), and an upscaler (630). In one example, at least one of the prompt generator (610), the generative AI model (620), and the upscaler (630) may be implemented by at least one software, at least one hardware, and / or a combination thereof. For example, the prompt generator (610) may be implemented to utilize a preset rule, a predefined algorithm, and / or an AI model. For example, at least one of the prompt generator (610), the generative AI model (620), and the upscaler (630) may be included within the electronic device (100), but may be distributed to at least one external device (e.g., a server) in one example.

[0137] In one example, the electronic device (100) may obtain a prompt based on user input using a prompt generator (610). In one example, the prompt generator (610) may generate a prompt using preset rules and / or preset algorithms.

[0138] According to one example, when the electronic device (100) receives a user input of the text "beach," it can input it to the prompt generator (610). The prompt generator (610) can generate a prompt called "paradise cosmic Beach" based on a preset rule. For example, the prompt generator (610) can generate a prompt according to a preset rule when the input text matches a specific pattern. The specific pattern can be related to a specific keyword, a form of sentence structure, etc. For example, the prompt generator (610) can generate a prompt according to a preset template.

[0139] According to an example, the electronic device (100) can input a prompt generated by the prompt generator (610) into the generative AI model (620) to obtain a background image. For example, the generative AI model (620) can be implemented as a Stable diffusion model. In this case, the Stable diffusion model can generate a small-sized background image. Accordingly, the background image generated by the generative AI model (620) can be input into the background image generated by the upscaler (630) to obtain an upscaled final background image. For example, the size of the background image generated by the generative AI model (620) can be a resolution of 384*672, and the size of the background image upscaled through the upscaler (630) can be a resolution of 1080*1920. However, the resolution values ​​are only examples and are not limited thereto. According to an example, the upscaler (630) can be implemented as an AI model for super resolution processing. However, it is not limited to this, and the upscaler (630) may perform upscaling processing using algorithms such as Nearest neighbor, Bicubic, Lanczos, and Spline.

[0140] FIG. 7 is a diagram illustrating a method for obtaining a background image according to one or more embodiments.

[0141] According to one embodiment illustrated in FIG. 7, the electronic device (100) may include a generative AI model (620) and an upscaler (630). According to one example, the implementation examples of the generative AI model (620) and the upscaler (630) are the same / similar to the embodiment illustrated in FIG. 6, and thus a detailed description thereof will be omitted.

[0142] According to one embodiment, the electronic device (100) can obtain a background image by inputting information received through user input into a generative AI model (620).

[0143] For example, information received through user input may include content images and style information. For example, the style information may include image styles and text styles. The image style may be style information in the form of an image, and the text style may be style information in the form of text. For example, the content image may be selected from a list including pre-stored images, or content stored through a user application may be selected. For example, the style information may be input as text or selected from a list. For example, the style information may include various style information such as abstract, impressionism, minimalist composition, cubism, 3D rendering, pastel colors, whimsical look, pop art, renaissane, gothiec art, futuristic look, illustration, graffiti painting, oil painting, art nouveau, sketch, dadaist collage, rococo painting, hyper realism, single continuous line art sketch, line drawing, etc.

[0144] For example, when a content image and style information stored in a gallery application are selected based on user input, the electronic device (100) may input the selected content image and style information into a generative AI model (620) to obtain a background image. For example, the background image obtained through the generative AI model (620) may be an image in which the content image has been style-converted based on the style information.

[0145] FIG. 8 is a diagram illustrating a method for obtaining a background image according to one or more embodiments.

[0146] According to an embodiment illustrated in FIG. 8, the electronic device (100) may include a prompt generator (610), a generative AI model (620), an upscaler (630), a cuisine generator (810), and an adapter (820). In one example, implementation examples of the prompt generator (610), the generative AI model (620), and the upscaler (630) are the same / similar to the embodiment illustrated in FIG. 6, and thus a detailed description thereof will be omitted.

[0147] In one example, the electronic device (100) can obtain cooking information using the cooking generator (810). For example, the cooking generator (810) can obtain cooking information based on a food list. For example, if the food list includes lettuce, egg, and tomato, the cooking information "sandwitch" can be obtained. In one example, the cooking generator (610) can generate cooking information using preset rules and / or preset algorithms.

[0148] In one example, the electronic device (100) can input a cooking text corresponding to the cooking information obtained through the cooking generator (810), for example, “sandwitch,” into the prompt generator (610). The prompt generator (610) can generate a prompt called “realistic photo of sandwich” based on a preset rule.

[0149] According to an example, the electronic device (100) can obtain a background image by inputting the prompt, object layout information, and color palette information generated by the prompt generator (610) into the generative AI model (620). For example, the electronic device (100) can convert the object layout information and the color palette information into a data string form that can be input into the generative AI model (620) using the adapter (820) and input the converted object layout information and the color palette information into the generative AI model (620).

[0150] For example, object layout information may include information about the structure of the UI screen, the position, size, spacing, etc. of UI elements. For example, color palette information may include color information such as primary colors, secondary colors, and contrast colors. For example, if the electronic device (100) is implemented as a refrigerator, the electronic device (100) may input color palette information based on the panel color of the refrigerator into the AI ​​model (620).

[0151] According to one example, the electronic device (100) can identify a dish that can be cooked with food whose expiration date is within a critical period, for example, 3 days, based on the expiration date information of each food stored in the storage.

[0152] For example, the cooking generator (810) can identify dishes that can be cooked with food that has an expiration date of less than 3 days by eliminating dishes that do not contain any food that has an expiration date of less than 3 days from the entire list of dishes that can be cooked based on the list of foods stored in the refrigerator.

[0153] For example, the cooking generator (810) can calculate a score for each dish that can be prepared using food with an expiration date of 3 days or less and identify a preset number of dishes in descending order of highest scores. For example, the electronic device (100) can calculate a score for each dish based on the following mathematical expression (1).

[0154]

[0155] TI(Total Ingredients) (or TF(Total Food)) may be the total number of food required for cooking, AI(Available Ingredients) (or AF(Available Food)) may be the number of food required for cooking that is currently available in the refrigerator, and DI(Desirable Ingredients) (or DF(Desirable Food)) may be the number of products required for cooking whose expiration date is within the critical period. According to mathematical expression 1, the cooking score may be proportional to TI and AI and inversely proportional to DI.

[0156] In one example, the prompt generator (610) can generate a prompt in a preset format based on preset rules. For example, the prompt generator (610) can generate a prompt in a format such as "a realistic photo of {Cuisine Text}."

[0157] FIG. 9 is a diagram illustrating a method for obtaining a background image according to one or more embodiments.

[0158] According to an embodiment illustrated in FIG. 9, the electronic device (100) may include a prompt generator (610), a generative AI model (620), an upscaler (630), and an adapter (820). According to an example, implementation examples of the prompt generator (610), the generative AI model (620), the upscaler (630), and the adapter (820) are the same / similar to the embodiments illustrated in FIGS. 6 to 8, and thus detailed descriptions thereof will be omitted.

[0159] In one example, the electronic device (100) may input food text and expiration date information corresponding to food information into the prompt generator (610). For example, food text such as "tomato" and remaining expiration date information such as "2 days" may be input into the prompt generator (610). The prompt generator (610) may generate a prompt related to an expiration date reminder based on a preset rule. For example, the prompt generator (610) may generate a prompt in a format such as "a {Type} of {Status} {Food Text} {Style}."

[0160] For example, the prompt generator (610) may generate a prompt related to an expiration date reminder based on a rule as shown in Table 1 below.

[0161] Expiration date: staustypestyle<1 day terrified drawing in animation style2~3 days depressed drawing in animation style4~7 days bored drawing in animation style>7 days freshrealistic photo

[0162] According to an example, the electronic device (100) can input a prompt, a food image, object layout information, and color palette information generated by the prompt generator (610) into the generative AI model (620) to obtain a background image. For example, the electronic device (100) can convert the object layout information and the color palette information into a data string form that can be input into the generative AI model (620) using the adapter (820) and input the converted data into the generative AI model (620). For example, the electronic device (100) can input a prompt generated based on Table 1, for example, "a drawing of depressed tomato in animation style", a tomato image, object layout information, and color palette information into the generative AI model (620) to obtain a background image.

[0163] FIG. 10 is a diagram illustrating a learning method of a generative AI model according to one or more embodiments.

[0164] According to one embodiment, the generative AI model can be implemented as a Diffusion Model, which is a type of image generative model, as illustrated in FIG. 10. For example, the generative AI model can be implemented as a Stable diffusion model. The Stable diffusion model is a type of deep learning model and can be a latent diffusion model that generates an AI image from text. For example, the Stable diffusion model can first compress the image into a latent space rather than a high-dimensional image space.

[0165] For example, the Stable diffusion model can learn through forward diffusion and reverse diffusion.

[0166] The forward diffusion process can gradually add noise (e.g., random noise) to the training images, gradually changing them into featureless, noisy images.

[0167] The backward diffusion process can recover the original image starting from a noisy image. In one example, the backward diffusion process can be trained to predict the added noise through the AI ​​model (1010). For example, the AI ​​model (1010) can actually predict the added noise (E real ) and noise prediction (E pred ) can be trained to reduce the difference. For example, a U-Net model called a noise predictor can be used, but is not limited thereto.

[0168] For example, the AI ​​model (1010) can be trained through conditioning, i.e., providing conditions. This is because, if trained without conditioning, images can be generated randomly. Accordingly, the AI ​​model (1010) can be trained through text conditioning related to the training images. For example, text captions corresponding to the training images can be input to the AI ​​model (1010) for training. For example, the text captions can be tokenized through a text encoder within the AI ​​model (1010) and then converted into an embedding vector for training.

[0169] For example, in addition to text captions corresponding to learning images, additional information such as object layout information, color palette information, and / or food images may be used for conditioning, but is not limited thereto.

[0170] FIG. 11 is a diagram illustrating a method for generating a background image of a generative AI model according to one or more embodiments.

[0171] According to an embodiment illustrated in FIG. 11, the electronic device (100) may input a noise image (1121) and conditioning information (1122) into a generative AI model (1110) to obtain a background image (1141). In one example, the conditioning information (1122) may include text caption information. In another example, the conditioning information (1122) may optionally include at least one of a food image, object layout information, or color palette information in addition to the text caption information.

[0172] For example, the electronic device (100) may obtain a background image (1141) by repeatedly inputting a noise image (1121) and conditioning information (1122) into a generative AI model (1110) and inputting the prediction noise output from the generative AI model (1110) together with the conditioning information (1122) into the generative AI model (1110). The generative AI model (1110) may obtain the background image (1141) based on the difference between the noise image (1121) and the final prediction noise (1131). For example, the generative AI model (1110) may obtain the background image (1141) by subtracting the final prediction noise (1131) from the noise image (1121).

[0173] As an example, a generative AI model (1110) may be implemented to include a text encoder, an image information generator, and an image decoder.

[0174] A text encoder can tokenize the input text caption and then provide the text embedding, which is converted into an embedding vector, to an image information generator.

[0175] An image information generator can generate an information array using a U-Net and a scheduler, which progressively processes or spreads information in the latent space, given a multidimensional array of text embeddings and noise. For example, a U-Net may include a series of layers, each operating on the output of the previous layer.

[0176] The image decoder can generate a background image (1141) based on an information array output from the image information generator.

[0177] For example, the electronic device (100) can reflect features corresponding to food images among the conditioning information (1122) by concatenating them into image features within the generative AI model (1110).

[0178] In one example, the electronic device (100) can reflect a prompt generated by the prompt generator (610), for example, a text caption, by cross-attention with an image feature within the generative AI model (1110).

[0179] For example, the electronic device (100) may add feature information corresponding to at least one of object layout information or color palette information obtained through the adapter (820) to the image features in the generative AI model (1110) and reflect the same.

[0180] The method by which various other conditioning information (1122) is reflected in the generative AI model (1110) may be identical / similar to the specific operation method of the Stable diffusion model.

[0181] FIGS. 12A and 12B are drawings for explaining a method of providing a UI screen according to one or more embodiments.

[0182] FIGS. 12A to 12C are drawings showing examples of UI screens provided through the front display of a refrigerator when an electronic device (100) is implemented as a refrigerator according to one embodiment.

[0183] According to FIG. 12a, the UI screen (1210) may include an image preview (1211), a food selection menu (1212), and an image creation button (1213).

[0184] For example, the food selection menu (1212) may include multiple items for selecting food information and expiration date information.

[0185] For example, when at least one food, for example, eggs and tomatoes, is selected through the food selection menu (1212), images of dishes using the selected foods may be provided through the image preview (1211) region. A “region” is a term referring to a portion of an image and means at least one pixel block or a set of pixel blocks. For example, when tomato egg soup and tomato egg sandwich are identified as dishes that can be prepared with the selected foods, the electronic device (100) may obtain a first dish image corresponding to tomato egg soup and a second dish image corresponding to tomato egg sandwich through generative AI and provide them through the image preview (1211) region. For example, the first dish image and the second dish image may be provided sequentially or simultaneously on one screen.

[0186] For example, when a specific background image displayed in the image preview (1211) area is selected using the image creation button (1213), the electronic device (100) can identify the corresponding cooking image as a personalized background image and provide it.

[0187] Accordingly, users can identify foods that are nearing their expiration date and receive personalized background image information on recipes that can be prepared using the identified foods.

[0188] However, although FIG. 12a describes an embodiment in which a user directly selects a desired cooking image as a background image, it is of course possible for the electronic device (100) to automatically generate and provide a background image based on the expiration date information of the food without user intervention. For example, the electronic device (100) can generate multiple cooking images and provide them as background images in an automatic slide format.

[0189] According to FIG. 12b, the UI screen (1220) may include an image preview (1221) and a next image view button (1222).

[0190] For example, the electronic device (100) may generate an image of food with an imminent expiration date in a specific style and provide a reminder image indicating that the expiration date is approaching in the image preview (1221) area. For example, the reminder image may be a reminder image indicating that the expiration date of a tomato is approaching, as illustrated in FIG. 12B. When the View Next Image button (1222) is selected, the electronic device (100) may provide another image of food with an imminent expiration date.

[0191] According to FIG. 12c, the UI screen (1230) may include an image preview (1231), a style selection menu (1232), a food selection menu (1233), and an image creation button (1234).

[0192] For example, the style selection menu (1232) may include multiple items for selecting style information to be used in creating a background image. For example, the style information may include style information such as Animation, Photographic, Digital Art, and Analog Film.

[0193] For example, the food selection menu (1233) may include multiple items containing food information and / or cooking information. For example, the food selection menu (1232) may include multiple items sequentially listed from the most recently received food information through refrigerator entry / exit recognition.

[0194] For example, when at least one food and / or dish, for example a sandwich, is selected through the food selection menu (1233), a dish image generated based on the selected style information may be provided through the image preview (1231) area. For example, multiple sandwich images may be provided sequentially or simultaneously on a single screen.

[0195] For example, when a specific background image displayed in the image preview (1231) area is selected using the image creation button (1234), the electronic device (100) can identify the corresponding cooking image as a personalized background image and provide it.

[0196] FIG. 13 is a drawing for explaining a method of providing a UI screen according to one or more embodiments.

[0197] FIG. 13 is a drawing showing an example of a UI screen provided through the front display of a refrigerator when an electronic device (100) is implemented as a refrigerator according to one embodiment.

[0198] A UI screen (1310) according to an embodiment illustrated in FIG. 13 may include an image preview (1311), a category selection menu (1312), a style selection menu (1313), a color tone selection menu (1314), and an image creation button (1315). In addition, the UI screen may include a text input window for entering at least one of a category, a style, or a color tone as text.

[0199] The category selection menu (1312) may provide multiple categories of information for the user to select. For example, the multiple categories of information may include, but are not limited to, at least one of patterns, landscape, landmarks, flowers, animals, food, interior, or nature.

[0200] The style selection menu (1313) may provide multiple style information that the user can select. For example, the multiple style information may include at least one of abstract, impressionism, minimal composition, cubism, 3D rendering, pastel colors, whimsical look, pop art, renaissane, gothiec art, futuristic look, illustration, graffiti painting, oil painting, art nouveau, sketch, dadaist collage, rococo painting, hyper realism, single continuous line art sketch, line drawing, or needle felted, but is not limited thereto.

[0201] The color tone selection menu (1314) may provide multiple color tone information items that the user can select. For example, the multiple color tone information items may include, but are not limited to, at least one of redish, bluish, greenish, or color pick. For example, color tone information items may be recommended based on colors related to the display (110) (e.g., panel color).

[0202] For example, the electronic device (100) may generate a plurality of background images corresponding to information selected through at least one of a category selection menu (1312), a style selection menu (1313), or a color tone selection menu (1314) and provide the generated background images in an image preview (1311) area.

[0203] For example, when a specific background image displayed in the image preview (1311) area is selected using the image creation button (1315), the electronic device (100) can identify the background image as a personalized background image and provide it.

[0204] According to one embodiment, the electronic device (100) may obtain a background image based on user information, preference information, and / or context information in addition to category information, style information, and color tone information. For example, user information may include anniversary information, calendar personal schedule information, etc. Accordingly, the electronic device (100) may generate a birthday-related image when it is a birthday and generate a year-end-related image when it is the end of the year.

[0205] In one example, user preference information may include at least one of content preference, visual preference, and digital action preference.

[0206] Content preference is a concept that represents a user's preference for a specific type of content and can vary depending on the user's tastes and interests. For example, content preference may include at least one of the following: genre preference, media type preference, language and style preference, depth of information preference, character preference, or content length preference.

[0207] Visual preferences can indicate a user's visual preferences. For example, visual preferences can include a user's preferences for visual content, such as design style, color, layout, image style, font style, and font size. For example, a user can directly select category information, style information, and color tone information through the UI screen (1310), but the face, category information, style information, and color tone information can also be automatically selected based on the user's preference information.

[0208] Digital action preferences can indicate how a user prefers certain actions or activities in a digital environment. For example, digital actions can include behaviors across various digital platforms, such as application usage habits, online activities (e.g., online shopping, online gaming), digital media consumption, social media activity, and digital service use. For example, digital action preferences can be identified based on information such as whether a user frequently uses a specific type of app, actively engages in a specific type of social media activity, actively consumes media through streaming services or online video platforms, or actively plays online games.

[0209] FIGS. 14a and 14b are drawings for explaining a method of providing a UI screen according to one or more embodiments.

[0210] FIGS. 14a and 14b are drawings showing examples of UI screens provided through the front display of a refrigerator when an electronic device (100) is implemented as a refrigerator according to one embodiment.

[0211] A UI screen (1410) according to an embodiment illustrated in FIG. 14a may include an image preview (1411), an application menu (1412), a wallpaper on / ff button (1413), and a clock style selection menu (1414). The clock style menu (1414) may include information on multiple clock styles.

[0212] For example, when a wallpaper application is selected from among multiple applications included in an application menu (1412) in the UI screen (1410) illustrated in FIG. 14a, a wallpaper on / off button (1413) may be provided.

[0213] For example, when the wallpaper on / off button (1413) is selected in the UI screen (1410) illustrated in FIG. 14a, a UI screen (1420) including an image preview (1421), a style selection menu (1422), a keyword selection menu (1423), and a clock style menu (1414) may be provided, as illustrated in FIG. 14b.

[0214] For example, the keyword selection menu (1423) may include multiple keyword information used to generate a background image. For example, the multiple keyword information may include, but is not limited to, at least one of "nature," "landscape," or "landmarks."

[0215] According to one example, the electronic device (100) can generate a background image based on style information selected through the style selection menu (1422), keyword information selected through the keyword selection menu (1423), and context information. For example, the electronic device (100) can generate multiple background images based on style information, keyword information, regional information, and weather information.

[0216] According to the various embodiments described above, a generative AI model can be used to provide personalized UI images. For example, a generative AI model can be used to provide a personalized background image on a refrigerator display.

[0217] Meanwhile, the methods according to the various embodiments of the present disclosure described above can be implemented only with a software upgrade or a hardware upgrade for an existing electronic device and / or server.

[0218] Additionally, the various embodiments of the present disclosure described above can also be performed through an embedded server provided in an electronic device, or an external server of the electronic device.

[0219] Meanwhile, according to a temporary example of the present disclosure, the various embodiments described above can be implemented as software including instructions stored in a machine-readable storage medium that can be read by a machine (e.g., a computer). The device is a device that can call instructions stored from the storage medium and operate according to the called instructions, and may include an electronic device (e.g., electronic device (A)) according to the disclosed embodiments. When an instruction is executed by a processor, the processor can perform a function corresponding to the instruction directly or by using other components under the control of the processor. The instruction may include code generated or executed by a compiler or interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' means that the storage medium does not contain a signal and is tangible, but does not distinguish between data being stored semi-permanently or temporarily in the storage medium.

[0220] Furthermore, according to one embodiment of the present disclosure, the method according to the various embodiments described above may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or online through an application store (e.g., Play Store™). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0221] In addition, each of the components (e.g., modules or programs) according to the various embodiments described above may be composed of a single or multiple entities, and some of the corresponding sub-components described above may be omitted, or other sub-components may be further included in various embodiments. Alternatively or additionally, some components (e.g., modules or programs) may be integrated into a single entity, which may perform the same or similar functions as those performed by each of the corresponding components prior to integration. Operations performed by modules, programs or other components according to various embodiments may be executed sequentially, in parallel, iteratively or heuristically, or at least some operations may be executed in a different order, omitted, or other operations may be added.

[0222] Although the preferred embodiments of the present disclosure have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above, and various modifications may be made by a person having ordinary skill in the art to which the present disclosure pertains without departing from the gist of the present disclosure as claimed in the claims, and such modifications should not be understood individually from the technical idea or prospect of the present disclosure.

Claims

1. In electronic devices, display; A memory storing one or more instructions; and comprising one or more processors; The one or more processors, by executing the one or more instructions, Obtain a text prompt based on information about items stored in the warehouse, By inputting the above text prompt, UI layout information and color information related to the electronic device into the learned artificial intelligence model, a personalized UI (User Interface) image is obtained, An electronic device that provides the acquired personalized UI image through the display.

2. In paragraph 1, The above electronic device, It is implemented as a refrigerator, The one or more processors, by executing the one or more instructions, Obtaining the text prompt based on information about the food stored in the above storage, The above text prompt, the UI layout information, and the panel color information of the refrigerator are input into the learned artificial intelligence model to obtain a personalized background image. Information about food stored in the above storage facility: An electronic device including at least one of cooking information using food stored in the storage, information on the time of receipt of food stored in the storage, or expiration date information of food stored in the storage.

3. In paragraph 2, The one or more processors, by executing the one or more instructions, Identify the expiration date information for each food stored in the above storage, Based on the above-identified expiration date information, identify dishes that can be prepared with food whose expiration date is within the critical period, An electronic device that obtains the text prompt based on the identified cookable dish.

4. In paragraph 3, The one or more processors, by executing the one or more instructions, For the identified first dish that can be cooked, a first score corresponding to the first dish is calculated based on the total number of foods required for the first dish, the number of foods available in the storage among the total foods required, and the number of foods stored in the storage whose shelf life is within a critical period. For the identified second dish that can be cooked, a second score corresponding to the second dish is calculated based on the total number of foods required for the second dish, the number of foods available in the storage among the total foods required, and the number of foods stored in the storage whose shelf life is within a critical period. An electronic device that obtains the text prompt based on a dish corresponding to a relatively larger score among the first score and the second score.

5. In paragraph 2, The one or more processors, by executing the one or more instructions, Identify the expiration date information for each food stored in the above storage, Obtain the above text prompt for food whose expiration date is within the critical period based on the identified expiration date information and reminder information for each remaining expiration date of the food. Reminder information for the remaining shelf life of the above food products is: An electronic device including status information, type information and style information of a reminder image for each remaining shelf life of food.

6. In paragraph 2, The one or more processors, by executing the one or more instructions, An electronic device that identifies a priority for the stored food based on information about the time of receipt of the food stored in the storage, and obtains the text prompt based on the identified priority.

7. In paragraph 1, The one or more processors, by executing the one or more instructions, An electronic device that obtains the personalized UI image by inputting the text prompt, the UI layout information, color information and context information related to the electronic device into a learned artificial intelligence model.

8. In paragraph 1, The one or more processors, by executing the one or more instructions, An electronic device that obtains a text prompt related to an image of an item stored in the storage unit based on information about the item stored in the storage unit and a preset rule.

9. In paragraph 1, The above learned artificial intelligence model is, It is implemented as a stable diffusion model, The above stable diffusion model is, When the above text prompt, the UI layout information, the color information and the noise image are input, noise is predicted based on the input data, An electronic device that obtains the personalized UI image based on the difference between the noise image and the predicted noise.

10. In paragraph 1, The one or more processors, by executing the one or more instructions, Provides a UI screen for selecting at least one of the categories, styles, and color tones. The color information is obtained based on the color tone selected through the above UI screen, An electronic device that obtains the personalized UI image by inputting the category information and style information selected through the UI screen into the learned artificial intelligence model together with the text prompt, the UI layout information, and the color information.

11. In a method for controlling an electronic device, A step of obtaining a text prompt based on information about items stored in a warehouse; A step of obtaining a personalized UI (User Interface) image by inputting the text prompt, UI layout information and color information related to the electronic device into a learned artificial intelligence model; and A control method comprising: a step of providing the acquired personalized UI image; 12. In paragraph 11, The above electronic device, It is implemented as a refrigerator, The steps to obtain the above text prompt are: Obtaining the text prompt based on information about the food stored in the above storage, The step of obtaining the above personalized image is: The above text prompt, the UI layout information, and the panel color information of the refrigerator are input into the learned artificial intelligence model to obtain a personalized background image. Information about food stored in the above storage facility: A control method comprising at least one of cooking information using food stored in the storage, information on the time of receipt of food stored in the storage, or information on the expiration date of food stored in the storage.

13. In paragraph 12, The steps to obtain the above text prompt are: A step of identifying the expiration date information of each food stored in the above storage; A step of identifying a dish that can be cooked with food whose expiration date is within a critical period based on the identified expiration date information; and A control method comprising: obtaining the text prompt based on the identified cookable dish; 14. In paragraph 13, The steps to obtain the above text prompt are: For the identified first dish that can be cooked, a step of calculating a first score corresponding to the first dish based on the total number of foods required for the first dish, the number of foods available in the storage among the total foods required, and the number of foods stored in the storage whose shelf life is within a critical period; For the identified second dish that can be cooked, a step of calculating a second score corresponding to the second dish based on the total number of foods required for the second dish, the number of foods available in the storage among the total foods required, and the number of foods stored in the storage whose shelf life is within a critical period; and A control method, comprising: a step of obtaining the text prompt based on a dish corresponding to a relatively larger score among the first score and the second score; 15. A non-transitory computer-readable medium storing computer instructions that, when executed by a processor of an electronic device, cause the electronic device to perform an operation, The above actions are, A step of obtaining a text prompt based on information about items stored in a warehouse; A step of obtaining a personalized UI (User Interface) image by inputting the text prompt, UI layout information and color information related to the electronic device into a learned artificial intelligence model; and A non-transitory computer-readable medium comprising: a step of providing the acquired personalized UI image;

Images