Image generation method and electronic device using ai on basis of personalized db

Abstract

According to various embodiments of the present invention, instructions stored in a memory (130), when executed by a processor, may cause an electronic device to: recognize an object included in an image displayed on a display; if the recognized object is a person, retrieve the recognized person from a face clustering database (DB) through a face clustering process; determine whether face correction of the recognized person within the displayed image is necessary; if the face correction of the recognized person within the displayed image is not necessary, update coordinate system rotation information of the person in a personalized DB on the basis of a face cluster identifier retrieved from the face clustering DB; if the face correction of the recognized person within the displayed image is necessary, provide, from the personalized DB, a recommended face list corresponding to the recognized person, according to a user request; generate a prompt requesting face generation of the recognized person on the basis of the displayed image and a face selected from the recommended face list on the basis of a user input; and according to the prompt, correct a face generated by means of generative AI to the face of the recognized person within the displayed image and provide same. Various embodiments are possible.

Description

Image generation method using AI based on personalized database and electronic device thereof

[0001] Various embodiments of the present disclosure disclose an image generation method using AI based on a personalized DB and an electronic device thereof.

[0002] With the advancement of digital technology, various types of electronic devices, such as mobile terminals, personal digital assistants (PDAs), electronic notebooks, smartphones, tablet PCs (personal computers), and wearable devices, are becoming widely used. These electronic devices are constantly undergoing improvements in their hardware and / or software to support and enhance their functionality.

[0003] Meanwhile, conventional image (or video) editing features (or applications) focus on enabling users to create engaging content and share it with others on social media. For example, image editing features may include offering a variety of stickers and filters, a user content sharing platform, and image or video editing technology.

[0004] Image generation methods using generative AI (artificial intelligence) can produce images of random shapes, depending on the performance of the generator engine. In particular, when it comes to human faces, generative AI can sometimes produce faces of completely different races or people, or their facial expressions can appear strange compared to the original.

[0005] In one embodiment, when the image quality of a person included in an image (or video) is poor due to blinking, turning the head, etc., a method and device for generating a face of a person using a well-made person photo stored in an already established personalized database by using a generative AI can be disclosed to obtain a more natural generation result.

[0006] An electronic device (101) according to an embodiment of the present disclosure includes a display (160), a memory (130) for storing instructions, and a processor (120), wherein the instructions, when executed by the processor, cause the electronic device to recognize an object included in an image displayed on the display, and if the recognized object is a person, search for the recognized person from a face clustering database (DB) through a face clustering process, determine whether face correction of the recognized person in the displayed image is necessary, and if face correction of the recognized person in the displayed image is not necessary, update coordinate system rotation information of the person in a personalized database (DB) based on a face cluster identifier searched from the face clustering DB, and if face correction of the recognized person in the displayed image is necessary, provide a recommended face list corresponding to the recognized person from the personalized DB according to a user request, and generate a prompt for requesting face generation of the recognized person based on a face selected from the recommended face list based on a user input and the image, and by the prompt The face generated by generative AI can be provided by correcting it to the face of the person recognized in the image.

[0007] An operating method of an electronic device (101) according to an embodiment of the present disclosure may include an operation of recognizing an object included in an image displayed on a display (160) of the electronic device, an operation of searching a face clustering database (DB) through a face clustering process of the recognized person when the recognized object is a person, an operation of determining whether face correction of the recognized person in the displayed image is required, an operation of updating coordinate system rotation information of the person in a personalized database (DB) based on a face cluster identifier retrieved from the face clustering DB when face correction of the recognized person in the displayed image is not required, an operation of providing a list of recommended faces corresponding to the recognized person from the personalized DB in response to a user request when face correction of the recognized person in the displayed image is required, an operation of generating a prompt requesting face generation of the recognized person based on a face selected from the recommended face list based on a user input and the displayed image, and an operation of correcting and providing a face generated by a generative AI by the prompt to the face of the recognized person in the displayed image.

[0008] According to one embodiment, the frequency of shooting of each item (e.g., clothing, shoes, accessories) included in the image may be calculated based on tag information and color information of the item, and items with a high shooting frequency may be provided with priority.

[0009] According to one embodiment, by providing the generative AI with facial information or items that match an object (e.g., face, upper clothing, shoes, etc.) selected by the user in the image among the facial information or items stored in the personalized DB, along with the image, the quality of the object generated by the generative AI can be improved.

[0010] In one embodiment, when outpainting a certain portion of an image (e.g., a part of the user's body, clothes, hairstyle), information stored in a personalized DB can be selected and used for outpainting.

[0011] In one embodiment, when replacing a face with a specific clustering ID, including a face within an image, the face replacement can be performed only within faces clustered (or grouped) with the same clustering ID.

[0012] According to one embodiment, when it is difficult to perform clustering of faces included in an image, an electronic device can provide a list of recommended faces of high-priority people based on the results of face clustering and information stored in a personalized DB.

[0013] According to one embodiment, when multiple people are recognized in an image, the faces of the people can be generated through generative AI by searching for a face cluster identifier for each person from a face clustering DB, and using information stored in a personalized DB corresponding to each person.

[0014] FIG. 1 is a block diagram of an electronic device within a network environment according to one embodiment.

[0015] FIG. 2 is a diagram illustrating a configuration for constructing and utilizing various databases in an electronic device according to one embodiment.

[0016] FIG. 3 is a flowchart illustrating an operating method of an electronic device according to one embodiment.

[0017] FIG. 4 is a diagram illustrating an example of generating a person's face using a personalized DB in an electronic device according to one embodiment.

[0018] FIG. 5 is a flowchart illustrating a method of recognizing a face in an electronic device according to one embodiment, updating a personalized DB, and generating a face using the personalized DB.

[0019] FIGS. 6A and 6B are drawings illustrating an example of synthesizing a selected area of ​​an image using information stored in a personalized DB in an electronic device according to one embodiment.

[0020] FIG. 7 is a flowchart illustrating a method for analyzing items in an image and updating a personalized DB in an electronic device according to one embodiment.

[0021] FIG. 8 is a flowchart illustrating a method for updating a representative image DB and a personalized DB in an electronic device according to one embodiment.

[0022] FIG. 9 is a diagram illustrating an example of in-painting using a personalized DB in an electronic device according to one embodiment.

[0023] FIG. 1 is a block diagram of an electronic device (101) within a network environment (100) according to various embodiments.

[0024] Referring to FIG. 1, in a network environment (100), an electronic device (101) may communicate with an electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of an electronic device (104) or a server (108) via a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0025] The processor (120) may, for example, execute software (e.g., a program (140)) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) and perform various data processing or operations. According to one embodiment, as at least a part of the data processing or operations, the processor (120) may store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the commands or data stored in the volatile memory (132), and store result data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or an auxiliary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together with the main processor (121). For example, when the electronic device (101) includes the main processor (121) and the auxiliary processor (123), the auxiliary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a given function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0026] The auxiliary processor (123) may control at least a portion of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0027] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).

[0028] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0029] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0030] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0031] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0032] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0033] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0034] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0035] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0036] The haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. According to one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0037] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0038] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0039] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0040] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0041] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0042] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas, for example, by the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device via the at least one selected antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0043] According to various embodiments, the antenna module (197) may form a mmWave antenna module. In one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.

[0044] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0045] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server utilizing machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0046] Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to the embodiments of this document are not limited to the aforementioned devices.

[0047] The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another component (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0048] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0049] Various embodiments of the present document may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0050] According to one embodiment, the method according to various embodiments disclosed in this document may be provided as a computer program product. The computer program product may be traded between sellers and buyers as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or may be provided through an application store (e.g., Play Store). TM ) or directly between two user devices (e.g., smart phones), online distribution (e.g., downloading or uploading). In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored or temporarily created in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0051] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and placed in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0052] FIG. 2 is a diagram illustrating a configuration for constructing and utilizing various databases in an electronic device according to one embodiment.

[0053] Referring to FIG. 2, an electronic device (e.g., the electronic device (101) of FIG. 1) according to an embodiment may include a face clustering database (DB) (210), a personalized database (DB) (220), a representative image database (DB) (230), an information acquisition unit (251), a data analysis unit (253), and a data processing unit (255). The face clustering DB (210), the personalized DB (220), and the representative image DB (230) may be included in a memory of the electronic device (101) (e.g., the memory (130) of FIG. 1). In addition, although the drawing and description illustrate that three databases are included, one database may include the contents of all three databases. For example, the personalized DB (220) may include the face clustering DB (210) and the representative image DB (230). Additionally, the information acquisition unit (251), data analysis unit (253), and data processing unit (255) may be included in the processor of the electronic device (101) (e.g., the processor (120) of FIG. 1).

[0054] The face clustering DB (210) may store a face cluster identifier for each person and a storage location (or face location) of a face image corresponding to the face cluster identifier. For example, the information acquisition unit (251) may acquire an image from an electronic device (101), an external electronic device (e.g., a computer) having the same user account as the electronic device (101), or a cloud server (e.g., the server (108) of FIG. 1). The image may include an image that the user directly took (or saved) or an image downloaded from an external source. The information acquisition unit (251) may detect a person through object recognition included in the acquired image, and may classify the person into a user, another user 1 (e.g., spouse), another user 2 (e.g., child 1), and another user 3 (e.g., child 2) through face clustering. Face clustering may mean grouping by person by analyzing facial feature points (e.g., eyes, nose, mouth, facial contour, face size, etc.). The information acquisition unit (251) can grant (or allocate) a first face cluster identifier to a user, a second face cluster identifier to another user 1, a third face cluster identifier to another user 2, and a fourth face cluster identifier to another user 3.

[0055] For example, the face clustering DB (210) may include a storage location of an image including a user corresponding to the first face cluster identifier, a storage location of an image including another user 1 corresponding to the second face cluster identifier, a storage location of an image including another user 2 corresponding to the third face cluster identifier, and a storage location of an image including another user 3 corresponding to the fourth face cluster identifier.

[0056] The above-described recognized object can be classified into a person, an animal, an object, etc. Since the present invention describes a configuration for personalizing a user's image, the description will be made only when the recognized object is a person. The image may include only the person's face, or may include body parts of the person, such as the person's upper body, lower body, or shoes. The personalized DB (220) may store at least one of face information, upper body information, lower body information, or shoe information corresponding to a face cluster identifier. The face information may include at least one of face size information, area information, 3D coordinate system rotation information, and clarity (or reliability). The face size information may refer to the size of the face within the image, and the face area information may refer to the area (or range, coordinates) occupied by the face within the image. The 3D coordinate system rotation information indicates the angle or degree of rotation of the face, and may indicate how much the face is tilted based on the x, y, and z axes, such as yaw, pitch, and roll. The above sharpness indicates how clear the face is. The higher the sharpness, the clearer the eyes, nose, and mouth of the face are, which can be more advantageous for generative AI (artificial intelligence) when generating a person's face. The above sharpness can be included as a number from 1 to 100 or as a percentage.

[0057] The above-mentioned top information, the above-mentioned bottom information, or the above-mentioned shoe information may be items, and may include at least one of tag information, coordinate area information, posture information, color information, or reliability information corresponding to each item. For example, the above-mentioned top information may include at least one of tag information (e.g., type of top clothing), coordinate area information (e.g., location of the top within the image, coordinates), posture information (e.g., degree of tilt of the top clothing), color information (e.g., main color of the top clothing), or reliability information (e.g., indicating how clearly the photo was taken). In the above-mentioned top clothing, the type of clothing may indicate the length of the clothing (e.g., whether it is short-sleeved or long-sleeved), the shape of the clothing (e.g., round neck or V-neck), or the pattern of the clothing (e.g., polka-dot pattern, stripes, no pattern). The above-described lower garment information may include at least one of tag information (e.g., type of lower garment), coordinate area information (e.g., location of lower garment within the image, coordinates), pose information (e.g., angle indicating how much the lower garment is tilted), color information (e.g., main color of the lower garment), or confidence (e.g., indicating how clearly the photo was taken).

[0058] In the above bottoms, the clothing type may mean pants (e.g., shorts, long pants), skirts (e.g., short skirts, long skirts), dresses, etc. The shoe information may include at least one of tag information (e.g., shoe type), coordinate area information (e.g., location of shoes included in the image, coordinates), posture information (e.g., angle indicating how much the shoes are tilted), color information (e.g., main color of the shoes), or reliability (e.g., indicating how clearly the photo was taken). The shoe type may mean shoes (e.g., low heels, high heels), sneakers, and sandals. The personalized DB (220) may store frequencies (e.g., how often they are detected in an image) for upper clothing information, lower clothing information, or shoe information.

[0059] The representative image database (230) may include representative images corresponding to face cluster identifiers. These representative images may represent the highest-quality photos of each person. These representative images may be included solely for the face, or may be included separately for each body part, such as clothing or shoe type.

[0060] The information acquisition unit (251) can acquire an image, perform face clustering using the acquired image, and assign a face cluster identifier.

[0061] The data analysis unit (253) can extract at least one of face information, upper body information, lower body information, or shoe information from the image. The data analysis unit (253) can extract at least one of face size information, area information, 3D coordinate rotation information, and sharpness from the image and store the extracted information in the personalized DB (220). For example, the data analysis unit (253) can identify an item of a person in the image and analyze the identified item to obtain coordinate area information corresponding to the item area. The data analysis unit (253) can extract detailed information of the item including at least one of tag information, posture information, color information, or reliability of the identified item. The data analysis unit (253) can search whether an item corresponding to the detailed information of the extracted item is stored in the personalized DB (220), and if an item corresponding to the detailed information of the extracted item is not searched in the personalized DB (220), the coordinate area information and detailed information of the identified item can be stored in the personalized DB (220).

[0062] When an item corresponding to the detailed information of the extracted item is searched from the personalized DB (220), the data analysis unit (253) can compare the reliability of the identified item with the reliability of the item searched from the personalized DB. When the reliability of the item searched from the personalized DB (220) is higher than the reliability of the identified item as a result of the comparison, the data analysis unit (253) can maintain a representative image corresponding to the searched item, and when the reliability of the identified item is higher than the reliability of the item searched from the personalized DB as a result of the comparison, the data analysis unit (253) can update a representative image stored in the representative image DB (230) corresponding to the identified item.

[0063] The data processing unit (255) can generate a prompt for requesting image generation using generative AI, and synthesize the image generated from the generative AI based on the generated prompt onto the original image. For example, the data processing unit (255) can identify (or determine) whether the face of a person in an image needs to be corrected, and if face correction is necessary, can guide the user. The data processing unit (255) can provide a list of recommended faces corresponding to the person from the personalized DB (220) according to a user request, and generate a prompt for requesting the generation of the face of the recognized person based on a face selected based on user input from the list of recommended faces and the displayed image. At this time, the data processing unit (255) can generate the prompt including the coordinate information of the selected face, the displayed image, and the face of the recognized person. Alternatively, the data processing unit (255) can generate the prompt including a masking area including the selected face and the recognized person. The data processing unit (255) can correct (or synthesize) the face generated by the generative AI by the above prompt and provide it as the face of the person recognized in the image.

[0064] An electronic device (101) according to an embodiment of the present disclosure includes a display (160), a memory (130) for storing instructions, and a processor (120), wherein the instructions, when executed by the processor, cause the electronic device to recognize an object included in an image displayed on the display, and if the recognized object is a person, search for the recognized person from a face clustering database (DB) through a face clustering process, determine whether face correction of the recognized person in the displayed image is necessary, and if face correction of the recognized person in the displayed image is not necessary, update coordinate system rotation information of the person in a personalized database (DB) based on a face cluster identifier searched from the face clustering DB, and if face correction of the recognized person in the displayed image is necessary, provide a recommended face list corresponding to the recognized person from the personalized DB according to a user request, and generate a prompt for requesting face generation of the recognized person based on a face selected from the recommended face list based on a user input and the image, and by the prompt A face generated by generative AI can be provided by correcting it to the face of a person recognized in the image. The coordinate rotation information of the person can include coordinate values, yaw, pitch, and roll values ​​of the detected face in the image.

[0065] The above face clustering DB may store a face cluster identifier assigned to each person and a storage location of a face image corresponding to the face cluster identifier. The personalized DB may store at least one of face information, upper body information, lower body information, or shoe information corresponding to the face cluster identifier, and the face information may include at least one of size information, area information, 3D coordinate rotation information, and sharpness corresponding to the face.

[0066] The above instructions, when executed by the processor, may cause the electronic device to extract face size information and three-dimensional coordinate rotation information corresponding to the recognized person, and search for face information matching the extracted face size information and three-dimensional coordinate rotation information in the personalized DB.

[0067] The above personalized DB may store at least one of tag information, coordinate area information, detail information, color information, or reliability corresponding to an item such as upper body information, lower body information, or shoe information.

[0068] The instructions, when executed by the processor, may cause the electronic device to identify an item of the recognized person within the displayed image, analyze the identified item to obtain coordinate area information corresponding to an item area, and extract detailed information of the item including at least one of tag information, detail information, color information, or reliability of the identified item.

[0069] The above instructions, when executed by the processor, may cause the electronic device to search whether an item corresponding to the detailed information of the extracted item is stored in the personalized DB, and if an item corresponding to the detailed information of the extracted item is not found in the personalized DB, store coordinate area information and detailed information of the identified item in the personalized DB.

[0070] The instructions, when executed by the processor, may cause the electronic device to compare the reliability of the identified item with the reliability of the item searched for in the personalized DB when an item corresponding to the detailed information of the extracted item is searched for in the personalized DB, and, as a result of the comparison, if the reliability of the item searched for in the personalized DB is higher than the reliability of the identified item, maintain a representative image corresponding to the searched item, and, as a result of the comparison, if the reliability of the identified item is higher than the reliability of the item searched for in the personalized DB, update a representative image stored in a representative image database (DB) corresponding to the identified item.

[0071] The instructions, when executed by the processor, may cause the electronic device to provide a representative image list including a representative image corresponding to each person from a representative image database (DB) if the recognized person is not searched for from the face clustering database, select a representative image from the provided representative image list based on a user input, and generate a prompt requesting face generation of the recognized person based on the selected representative image and the image.

[0072] The above representative image DB may store a representative face image corresponding to the above face cluster identifier and a representative item image for each person item.

[0073] The instructions, when executed by the processor, may cause the electronic device to generate a prompt including facial coordinate information of the selected face, the displayed image, and the recognized person, or to generate a prompt including a masking area including the selected face and the recognized person.

[0074] Figure 3 is a flowchart (300) illustrating an operating method of an electronic device according to one embodiment.

[0075] Referring to FIG. 3, in operation 301, a processor (e.g., the processor 120 of FIG. 1) of an electronic device (e.g., the electronic device 101 of FIG. 1) according to an embodiment may recognize a face from an image. The image may be an image displayed on a display (e.g., the display module 160 of FIG. 1), and may be acquired from the electronic device (101), an external electronic device (e.g., a computer) having the same user account as the electronic device (101), or a cloud server (e.g., the server 108 of FIG. 1). The image may include an image directly captured (or stored) by a user, or an image downloaded from an external source. The processor (120) may recognize a face through object recognition from the image. The object may include at least one of a person, an animal, or an object. If the recognized object is a person, the processor (120) may recognize the face.

[0076] In operation 303, the processor (120) may search (or identify) a face clustering DB (e.g., the face clustering DB (210) of FIG. 2) through face clustering. The face clustering may mean analyzing facial features (e.g., eyes, nose, mouth, facial contour, face size, etc.) and grouping them by person. The face clustering DB (210) may store a face cluster identifier for each person and a storage location (or face location) of a face image corresponding to the face cluster identifier. For example, the face clustering DB (210) may include a storage location of an image including a user corresponding to the first face cluster identifier, a storage location of an image including another user 1 corresponding to the second face cluster identifier, a storage location of an image including another user 2 corresponding to the third face cluster identifier, and a storage location of an image including another user 3 corresponding to the fourth face cluster identifier. The processor (120) can search the face clustering DB (210) to see if there is a face cluster identifier (ID) corresponding to the face recognized in operation 301.

[0077] Although the drawing describes that operation 303 is performed before operation 305, operation 305 may be performed first to determine whether facial correction of a person included in the image is necessary, and if facial correction is necessary, operation 303 may be performed. The description provided to help understanding of the invention does not limit the invention.

[0078] In operation 305, the processor (120) may determine whether facial correction is required. Whether the face requires correction may be due to factors such as the eyes being closed, a portion of the face being obscured, the face being of poor clarity, or the face being rotated so that it cannot be seen from the front. The processor (120) may determine whether the face of the person included in the image requires correction through facial feature point analysis. If facial correction is required, the processor (120) may perform operation 307, and if facial correction is not required, the processor (120) may perform operation 306.

[0079] If face correction is not required, in operation 306, the processor (120) may update a personalized DB (e.g., personalized DB (220) of FIG. 2) based on a face cluster identifier (ID). The face cluster identifier may be the one retrieved in operation 303. The personalized DB (220) may store at least one of face information, upper body information, lower body information, or shoe information corresponding to the face cluster identifier. The face information may include at least one of face size information, face area information, three-dimensional coordinate system rotation information, and clarity (or reliability). The face size information may refer to a size of a face within an image, and the face area information may refer to an area (or range, coordinates) occupied by a face within an image. The three-dimensional coordinate system rotation information indicates an angle or degree of rotation of the face, and may indicate how much the face is tilted with respect to the x, y, and z axes, such as yaw, pitch, and roll. The above sharpness indicates how clear the face is, and the higher the sharpness, the clearer the eyes, nose, and mouth of the face, which may be more advantageous when the generative AI generates the face of a person. The above-mentioned upper body information, lower body information, or shoe information may be items, and may include at least one of tag information, coordinate area information, posture information, color information, or reliability information corresponding to each item. The processor (120) may extract (or identify) facial information from the image, and store (or add) the extracted facial information in the personalized DB (220) corresponding to the face cluster identifier. The personalized DB (220) may store the frequency (e.g., how often it is detected in the image) of the upper body information, lower body information, or shoe information.

[0080] If face correction is required, in operation 307, the processor (120) may detect whether face replacement is requested. If face correction is required, the processor (120) may inform the user that face replacement is required. For example, the processor (120) may provide a user interface that includes a message indicating that face replacement is required. The user interface may include at least one of text, an image, a video, or audio. The processor (120) may display the user interface on the display module (160) or output a voice related to the user interface to a speaker (e.g., the audio output module (155) of FIG. 1). If face replacement is requested, the processor (120) may perform operation 309, and if face replacement is not requested, the process may be terminated. If the face is covered or the face is not at a frontal angle, it may be difficult to utilize it as personalized information, and thus the process may be terminated.

[0081] When a face replacement is requested, in operation 309, the processor (120) may provide a recommended face list from the personalized DB (220). The recommended face list may include a face image that is most similar to the face information of the person included in the image among the face information stored in the personalized DB (220). For example, since the face information includes face size information, area information, 3D coordinate rotation information, and sharpness, the processor (120) may include in the recommended face list face information that has high sharpness and is most similar to the face size information, area information, or 3D coordinate rotation information. The processor (120) may display a face image that has a high similarity to the face information of the person included in the image at the top of the recommended face list.

[0082] In operation 311, the processor (120) may select a face (or face image) based on a user input. The processor (120) may receive a user selection of any one of the face images included in the recommended face list.

[0083] In operation 313, the processor (120) may request face generation based on the image and the selected face. The generative AI may be included in the electronic device (101) or may be included in an external server (e.g., server (108) of FIG. 1). The processor (120) may generate a prompt including the image, facial coordinate information of a person included in the image, and the selected face (e.g., face image and face information). Alternatively, the processor (120) may generate a prompt including a masking area including the recognized person and the selected face. The processor (120) may transmit the generated prompt to the generative AI to request face generation. The processor (120) may transmit the prompt to the server (108) via a communication module (e.g., communication module (190) of FIG. 1) to request face generation.

[0084] In operation 315, the processor (120) can synthesize the generated face into an image. The processor (120) can obtain (or receive) the generated face from the generative AI and replace it with the generated face based on the facial coordinate information of the person included in the image.

[0085] FIG. 4 is a diagram illustrating an example of generating a person's face using a personalized DB in an electronic device according to one embodiment.

[0086] Referring to FIG. 4, a processor (e.g., a processor (120) of FIG. 1) of an electronic device (e.g., an electronic device (101) of FIG. 1) according to an embodiment may provide a first user interface (410) including an image (401). The image (401) may be displayed on a display (e.g., a display module (160) of FIG. 1). The processor (120) may recognize an object from the image (401) to recognize a face of a person (411). The processor (120) may search for a face cluster identifier corresponding to the face of the person (411) from a face clustering database (DB) through a face clustering process. The processor (120) may obtain a first face cluster identifier corresponding to the face of the person (411) as a result of the search. The processor (120) may determine whether face correction of the person (411) is necessary. When looking at the face of the person (411), the person (411) has his / her eyes closed, so the processor (120) may determine that the face of the person (411) requires facial correction.

[0087] When the face of the person (411) requires facial correction, the processor (120) may inform the user that a face replacement is required. For example, the processor (120) may provide a user interface that includes a notification that a face replacement is required. When a user requests a face replacement, the processor (120) may search for a facial image corresponding to the face of the person (411) from a personalized DB (e.g., the personalized DB (220) of FIG. 2). The processor (120) may extract (or identify) facial information of the person (411) in the image (401) and search for facial information corresponding to the extracted facial information of the person (411) from the personalized DB (220). For example, since the face information of a person (411) includes face size information, area information, 3D coordinate rotation information, and sharpness, the processor (120) can search for face information that has high sharpness and is most similar to the face information of the person (411) and the face size information, area information, or 3D coordinate rotation information from the personalized DB (220). The processor (120) can provide a recommended face list of face images corresponding to the searched face information. The second user interface (430) can include a first face image (431) and a second face image (433) having similar face information to the face information of the person (411) as a face recommendation list.

[0088] The third user interface (450) illustrates an example of receiving a user's selection of a first face image (451) from the recommended face list. The processor (120) may generate a prompt requesting face generation of a person (411) based on the image (401) and the first face image (451) selected by the user (e.g., a face image including face information). The processor (120) may generate a prompt including the image (401), face coordinate information of the person (411) included in the image (401), and the selected face image (e.g., a face image and face information). Alternatively, the processor (120) may generate a prompt including a masking area including the recognized person (411) and the selected face. The processor (120) may transmit the generated prompt to a generative AI to request face generation. The processor (120) may obtain a face generated by the generative AI based on the prompt. The processor (120) can synthesize the face of a person (411) in the image (401) with a face generated by the generative AI. The fourth user interface (471) may illustrate an example in which the face of a person (411) in the image (401) is replaced (or corrected) with a face (471) generated by the generative AI.

[0089] FIG. 5 is a flowchart (500) illustrating a method of recognizing a face in an electronic device according to one embodiment, updating a personalized DB, and generating a face using the personalized DB.

[0090] Referring to FIG. 5, in operation 501, a processor (e.g., the processor (120) of FIG. 1) of an electronic device (e.g., the electronic device (101) of FIG. 1) according to an embodiment may recognize an object from an image. The image may be an image displayed on a display (e.g., the display module (160) of FIG. 1), and may be obtained from the electronic device (101), an external electronic device (e.g., a computer) having the same user account as the electronic device (101), or a cloud server (e.g., the server (108) of FIG. 1). The image may include an image directly captured (or stored) by a user, or an image downloaded from an external source. The object may include at least one of a person, an animal, or an object.

[0091] In operation 503, the processor (120) can determine (or identify) whether a face is detected (or recognized) in the image. The processor (120) can recognize a person through object recognition from the image. If the recognized object is a person, the processor (120) can recognize the face of the person. If a face is detected in the image, the processor (120) can perform operation 505, and if a face is not detected in the image, the processor (120) can perform operation 506.

[0092] If a face is not detected in the image, in operation 506, the processor (120) may extract item information and update a personalized DB (e.g., personalized DB (220) of FIG. 2). An item may refer to a body part of a person. For example, the item may refer to upper garments (or upper body), lower garments (or lower body), or shoes. The item information may include at least one of tag information, coordinate area information, posture information, color information, or reliability corresponding to each item. For example, if the item included in the image is an upper garment, the processor (120) may extract upper garment information including at least one of tag information (e.g., type of upper garment), coordinate area information (e.g., location of the upper garment within the image, coordinates), posture information (e.g., angle indicating how tilted the upper garment is), color information (e.g., main color of the upper garment), or reliability (e.g., indicating how clearly the photo was taken). According to one embodiment, if the object recognized in operation 501 is not a person, the processor (120) may terminate operation 503 because no face was detected.

[0093] If a face is detected in the image, in operation 505, the processor (120) may search (or identify) a face clustering DB (e.g., the face clustering DB (210) of FIG. 2) through face clustering. The face clustering may mean analyzing facial features (e.g., eyes, nose, mouth, facial contour, face size, etc.) and grouping them by person. The face clustering DB (210) may store a face cluster identifier for each person and a storage location (or face location) of a face image corresponding to the face cluster identifier. The processor (120) may search for a face cluster identifier corresponding to a face of a person included in the image from the face clustering DB (210).

[0094] In operation 507, the processor (120) can determine whether a face cluster identifier (ID) has been acquired. If a face cluster identifier corresponding to the face of the person included in the image is retrieved from the face clustering DB (210), the processor (120) can perform operation 509, and if a face cluster identifier corresponding to the face of the person included in the image is not retrieved, the processor (120) can perform operation 519.

[0095] If a face cluster identifier corresponding to a face of a person included in the image is not searched, in operation 519, the processor (120) may provide a representative image from a representative image DB (e.g., the representative image DB (230) of FIG. 2). The representative image DB (230) may include a representative image corresponding to the face cluster identifier. The representative image may mean a photo with the highest quality for each person. The representative image may be included only for the face, or may be included separately for each body part, such as a type of clothing or a type of shoe. The processor (120) may provide a user with a representative image list including one or more representative images stored in the representative image DB (230).

[0096] In operation 521, the processor (120) may select a representative image based on user input. The processor (120) may receive a selection of a representative image from a list of representative images from the user. If the representative image is selected by the user, the processor (120) may return to operation 509.

[0097] If a face cluster identifier corresponding to a face of a person included in the image is searched, in operation 509, the processor (120) may determine whether face correction is required. Whether the face needs correction may be due to the eyes being closed, a part of the face being covered, the face having low clarity, or the face being rotated so that it cannot be seen from the front. The processor (120) may determine whether the face of the person included in the image needs correction through facial feature point analysis. If face correction is required, the processor (120) may perform operation 511, and if face correction is not required, the processor (120) may perform operation 523.

[0098] Although the drawing describes that operation 505 is performed before operation 509, operation 509 may be performed first to determine whether facial correction of a person included in the image is necessary, and if facial correction is necessary, operation 505 may be performed. The description provided to help understanding of the invention does not limit the invention.

[0099] If facial correction is not required, in operation 523, the processor (120) may update a personalized DB (e.g., personalized DB (220) of FIG. 2) based on a facial cluster identifier (ID). The facial cluster identifier may be retrieved in operation 505. The personalized DB (220) may store at least one of face information, upper body information, lower body information, or shoe information corresponding to the facial cluster identifier. The facial information may include at least one of face size information, area information, 3D coordinate system rotation information, and clarity (or reliability). The upper body information, the lower body information, or the shoe information may be items, and may include at least one of tag information, coordinate area information, posture information, color information, or reliability corresponding to each item. The processor (120) may extract (or identify) facial information from the image, and store (or add) the extracted facial information corresponding to the facial cluster identifier in the personalized DB (220). The personalized DB (220) can store the frequency (e.g., how often it is detected in an image) of upper body information, lower body information, or shoe information.

[0100] If face correction is required, in operation 511, the processor (120) may detect whether face replacement is requested. If face correction is required, the processor (120) may guide the user that face replacement is required. For example, the processor (120) may provide a user interface that includes a message indicating that face replacement is required. The user interface may include at least one of text, an image, a video, or audio. The processor (120) may display the user interface on the display module (160) or output a voice related to the user interface to a speaker (e.g., the audio output module (155) of FIG. 1). If face replacement is requested, the processor (120) may perform operation 513, and if face replacement is not requested, the process may be terminated.

[0101] When a face replacement is requested, in operation 513, the processor (120) may provide a recommended face list from the personalized DB (220). The recommended face list may include a face image that is most similar to the face information of the person included in the image among the face information stored in the personalized DB (220). For example, since the face information includes face size information, area information, 3D coordinate system rotation information, and sharpness, the processor (120) may include in the recommended face list face information that has high sharpness and is most similar to the face size information, area information, or 3D coordinate system rotation information. The processor (120) may display a face image that has a high similarity to the face information of the person included in the image at the top of the recommended face list.

[0102] In operation 515, the processor (120) may select a face (or face image) based on a user input. The processor (120) may receive a user selection of any one of the face images included in the recommended face list.

[0103] In operation 517, the processor (120) can generate and synthesize a face using generative AI. The processor (120) can request face generation based on the image and the selected face. The generative AI may be included in the electronic device (101) or may be included in an external server (e.g., server (108) of FIG. 1). The processor (120) can generate a prompt including the image, facial coordinate information of a person included in the image, and the selected face (e.g., face image and face information). Alternatively, the processor (120) can generate a prompt including a masking area including the recognized person and the selected face. The processor (120) can transmit the generated prompt to the generative AI to request face generation. The processor (120) can transmit the prompt to the server (108) via a communication module (e.g., communication module (190) of FIG. 1) to request face generation. The processor (120) can synthesize the generated face into an image. The processor (120) can obtain (or receive) a face generated from the generative AI and replace it with the generated face based on the facial coordinate information of the person included in the image.

[0104] FIGS. 6A and 6B are drawings illustrating an example of synthesizing a selected area of ​​an image using information stored in a personalized DB in an electronic device according to one embodiment.

[0105] Referring to FIG. 6A, a processor (e.g., a processor (120) of FIG. 1) of an electronic device (e.g., an electronic device (101) of FIG. 1) according to an embodiment may display a first user interface (610) including an image (601) on a display (e.g., a display module (160) of FIG. 1). The image (601) may be stored in a memory (e.g., a memory (130) of FIG. 1) or a cloud server (e.g., a server (108) of FIG. 1). The image (601) may be captured by a user or downloaded from an external source. The first user interface (610) may include a person (611) in the image (601). When a request for image generation (621) through a generative AI is selected in the second user interface (620), the processor (120) may recognize an object from the image (601) to recognize the face of the person (611). The third user interface (630) may be divided into a face area (631) and an upper body area (633).

[0106] Referring to FIG. 6B, the fourth user interface (64) may illustrate an example of selecting the upper body area (643) among the face area (641) and the upper body area (643) as an area for which an image is desired to be generated through generative AI. The processor (120) may search for item information corresponding to the upper body area (643) in a personalized DB (e.g., the personalized DB (220) of FIG. 2). The personalized DB (220) may store at least one of face information, upper body information, lower body information, or shoe information corresponding to a face cluster identifier. The upper body information may include at least one of tag information, coordinate area information, posture information, color information, or reliability for the upper body area. The lower body information may include at least one of tag information, coordinate area information, posture information, color information, or reliability for the lower body area. The shoe information may include at least one of tag information, coordinate area information, posture information, color information, or reliability for the shoe area. The personalized DB (220) can store the frequency (e.g., how often it is detected in an image) of upper body information, lower body information, or shoe information.

[0107] The processor (120) can extract (or identify) item details (e.g., tag information, coordinate area information, detail information, color information, or reliability) corresponding to the upper area (643) from the image (601), and search for upper information corresponding to the extracted item details in the personalized DB (220). The fifth user interface (650) can include a recommended item list including first item information (or image) (651) and second item information (653). The recommended item list can include an image of upper information corresponding to the extracted item details. The processor (120) can display an upper image (e.g., a frequently worn upper image) with a high frequency at the top of the recommended item list based on the frequency with which the item is detected in the image.

[0108] When the second item information (653) is selected in the fifth user interface (650), the processor (120) may generate a prompt including an image (601), an upper body area (643), and second item information (653). The prompt may be a request for generating an image that replaces the upper body area (643) with the second item information (653). The processor (120) may transmit the generated prompt to a generative AI, and the generative AI may generate an image. The generative AI may generate an image for the upper body area (643) by referring to the color, pattern, texture, and design of the clothes, which are the second item information (653). The sixth user interface (660) may include an image (661) generated by the generative AI.

[0109] FIG. 7 is a flowchart (700) illustrating a method for analyzing items within an image and updating a personalized DB in an electronic device according to one embodiment. FIG. 7 may be a method performed after operation 501 of FIG. 5.

[0110] Referring to FIG. 7, in operation 701, a processor (e.g., processor (120) of FIG. 1) of an electronic device (e.g., electronic device (101) of FIG. 1) according to an embodiment may determine whether an upper body is detected from an image. Each time a new image is acquired, the processor (120) may recognize an object from the image, and if the recognized object is a person, determine whether the person includes an upper body. If an upper body is detected, the processor (120) may perform operation 703, and if an upper body is not detected, the processor (120) may perform operation 711.

[0111] If the upper body is detected, in operation 703, the processor (120) may analyze the upper body area. The upper body area may refer to the area from the person's shoulder to the waist. The processor (120) may analyze the upper body area of ​​the person in the image.

[0112] In operation 705, the processor (120) may obtain coordinate information corresponding to an area of ​​the image from the image. The coordinate information may indicate an area (or location, coordinates) (e.g., coordinate area information) occupied by the area of ​​the image.

[0113] In operation 707, the processor (120) may extract detailed information of the upper region from the image. The detailed information may include at least one of tag information, posture information, color information, or reliability information regarding the upper garment. The tag information may indicate the type of the upper garment, the posture information may indicate how much the upper garment is tilted, and the color information may indicate the main color of the upper garment. The reliability may indicate how clearly the upper garment was photographed. The type of garment in the upper garment may indicate the length of the garment (e.g., short sleeves or long sleeves), the shape of the garment (e.g., round neck or V-neck), or the pattern of the garment (e.g., polka dots, stripes, no pattern).

[0114] In operation 709, the processor (120) may update a personalized DB (e.g., the personalized DB (220) of FIG. 2). The processor (120) may add (or store, update) at least one of coordinate information, detailed information, or an image of the upper garment corresponding to the upper garment area, the upper garment area, or the shoe area to the personalized DB (220). A method of updating the personalized DB (220) will be described in detail with reference to FIG. 8.

[0115] Although the drawing illustrates that operation 701 is performed before operation 711 or operation 721, operation 711 or operation 721 may be performed before operation 711. This is merely an implementation issue and the invention is not limited by the description.

[0116] If the upper body is not detected, in operation 711, the processor (120) may determine whether the lower body is detected from the image. Each time a new image is acquired, the processor (120) may recognize an object from the image, and if the recognized object is a person, determine whether the person includes a lower body. If the lower body is detected, the processor (120) may perform operation 713, and if the lower body is not detected, the processor (120) may perform operation 721.

[0117] If the lower body is detected, in operation 713, the processor (120) may analyze the upper body area. The upper body area may refer to the area from the person's buttocks to the feet (or ankles). The processor (120) may analyze the lower body area of ​​the person in the image.

[0118] In operation 715, the processor (120) may obtain coordinate information corresponding to the lower region from the image. The coordinate information may indicate an area (or location, coordinates) (e.g., coordinate area information) occupied by the lower region in the image.

[0119] In operation 717, the processor (120) may extract detailed information of the lower body area from the image. The detailed information may include at least one of tag information, posture information, color information, or reliability information regarding the lower body. The tag information may indicate the type of the lower body, the posture information may indicate how tilted the lower body is, and the color information may indicate the main color of the lower body. The reliability may indicate how clearly the lower body is photographed. The type of clothing regarding the lower body may mean pants (e.g., shorts, long pants), skirts (e.g., short skirts, long skirts), and dresses. After extracting the detailed information of the lower body area from the image, the processor (120) may return to operation 709.

[0120] If the lower body is not detected, in operation 721, the processor (120) may determine whether shoes are detected from the image. Each time a new image is acquired, the processor (120) may recognize an object from the image, and if the recognized object is a person, determine whether the person includes shoes. If shoes are detected, the processor (120) may perform operation 723, and if shoes are not detected, the processor may terminate the process.

[0121] If shoes are detected, in operation 723, the processor (120) may analyze the shoes. The processor (120) may analyze the shoes of the person in the image.

[0122] In operation 725, the processor (120) may obtain coordinate information corresponding to a shoe area from the image. The coordinate information may indicate an area (or location, coordinates) (e.g., coordinate area information) occupied by the shoe area in the image.

[0123] In operation 727, the processor (120) may extract detailed information of the shoe area from the image. The detailed information may include at least one of tag information, posture information, color information, or reliability information regarding the shoe. The tag information may indicate the type of shoe, the posture information may indicate how much the shoe is tilted, and the color information may indicate the main color of the shoe. The reliability may indicate how clearly the shoe was captured. The shoe type may mean a shoe (e.g., low heel, high heel), a sneaker, or a sandal. After extracting the detailed information of the shoe area from the image, the processor (120) may return to operation 709.

[0124] FIG. 8 is a flowchart (800) illustrating a method for updating a representative image DB and a personalized DB in an electronic device according to one embodiment. FIG. 8 may be a detailed description of operation 709 of FIG. 7.

[0125] Referring to FIG. 8, in operation 801, a processor (e.g., the processor 120 of FIG. 1) of an electronic device (e.g., the electronic device 101 of FIG. 1) according to an embodiment may obtain item information from an image. The item information may mean at least one of upper body information, lower body information, or shoe information. Operation 801 may be performed when an upper body, lower body, or shoes of a person are detected in the image through object recognition of the image each time a new image is obtained. The upper body information may include at least one of tag information (e.g., the type of upper clothing), coordinate area information (e.g., the location and coordinates of the upper clothing within the image), posture information (e.g., the degree to which the upper clothing is tilted), color information (e.g., the main color of the upper clothing), or reliability (e.g., indicating how clearly the photo was taken). In the above example, the type of clothing can refer to the length of the clothing (e.g., short sleeves or long sleeves), the shape of the clothing (e.g., round neck or V-neck), or the pattern of the clothing (e.g., polka dots, stripes, no pattern).

[0126] The above-described lower body information may include at least one of tag information (e.g., type of lower body clothing), coordinate area information (e.g., location of the lower body within the image, coordinates), pose information (e.g., angle indicating how tilted the lower body is), color information (e.g., main color of the lower body clothing), or reliability (e.g., indicating how clearly the photo is taken). In the lower body, the type of clothing may mean pants (e.g., shorts, long pants), skirts (e.g., short skirts, long skirts), dresses, etc. The above-described shoe information may include at least one of tag information (e.g., type of shoe), coordinate area information (e.g., location of the shoe within the image, coordinates), pose information (e.g., angle indicating how tilted the shoe is), color information (e.g., main color of the shoe), or reliability (e.g., indicating how clearly the photo is taken). The above-described shoe type may mean shoes (e.g., low heels, high heels), sneakers, and sandals.

[0127] In operation 803, the processor (120) can compare the acquired item information with the stored item information in a personalized DB (e.g., the personalized DB (220) of FIG. 2). The processor (120) can compare the acquired item information with the stored item information.

[0128] In operation 805, the processor (120) can determine whether the acquired item information is new. The processor (120) can determine whether the item information acquired in operation 801 is not stored in the personalized DB (220). If the acquired item information is new, the processor (120) can perform operation 807, and if the acquired item information is not new, the processor (120) can perform operation 811.

[0129] If the acquired item information is new, in operation 807, the processor (120) can add the acquired item information to the personalized DB (220). If the acquired item information is different from the item information stored in the personalized DB (220), the processor (120) can view it as new and add the acquired item information.

[0130] If the acquired item information is not new, in operation 811, the processor (120) can determine whether the acquired item information has a high reliability. The reliability may be a degree (or a numerical value) indicating how clearly the item was photographed. The higher the reliability, the better the quality of the image can be generated by the generative AI when generating the image. If the reliability of the acquired item information is higher than the reliability of the item information stored in the personalized DB (220), the processor (120) can perform operation 813, and if the reliability of the acquired item information is lower than the reliability of the item information stored in the personalized DB (220), the processor (120) can perform operation 815.

[0131] According to one embodiment, the processor (120) may count the frequency of the item information (e.g., frequently worn clothes) if the acquired item information is not new (e.g., if the acquired item information is identical to the stored item information). For example, if the acquired item information is a white and black striped t-shirt, the processor (120) may identify how many items identical to (or similar to) the acquired item information are detected in the image and store the frequency in the personalized DB (220).

[0132] If the reliability of the acquired item information is higher than the reliability of the item information stored in the personalized DB (220), in operation 813, the processor (120) may update the representative image DB (e.g., the representative image DB (230) of FIG. 2). The processor (120) may change the representative image of the item stored in the representative image DB (230) to the image of the acquired item information.

[0133] If the reliability of the acquired item information is lower than the reliability of the item information stored in the personalized DB (220), in operation 815, the processor (!20) may maintain the representative image DB (230). The processor (120) may maintain the representative image of the item stored in the representative image DB (230) without changing it.

[0134] FIG. 9 is a diagram illustrating an example of in-painting using a personalized DB in an electronic device according to one embodiment.

[0135] Referring to FIG. 9, a processor (e.g., a processor (120) of FIG. 1) of an electronic device (e.g., an electronic device (101) of FIG. 1) according to an embodiment may display a first user interface (910) including an image (911) on a display (e.g., a display module (160) of FIG. 1). The image (911) may be stored in a memory (e.g., a memory (130) of FIG. 1) or a cloud server (e.g., a server (108) of FIG. 1). The image (911) may be captured by a user or downloaded from an external source. The processor (120) may analyze the image (911) to recognize a person (913) and an animal (915) as objects. The second user interface (930) may include a list of recommended shoes (931) to replace an animal area on which in-painting should be performed when an animal (933) is selected. The processor (120) can retrieve information about lower garments or shoes stored in correspondence to the face clustering identifier of the person (913) from a personalized DB (e.g., personalized DB (220) of FIG. 2). The recommended shoe list (931) may include shoe images retrieved from the personalized DB (220). The processor (120) can display shoe images with a high frequency (e.g., frequently worn shoes) at the top of the recommended shoe list based on the frequency with which items are detected in the images.

[0136] The processor (120) may generate a prompt based on an image (911), a coordinate area (933) in which the image (911) is painted, and a third shoe image (e.g., an image including shoe information) selected from the recommended shoe list. The third shoe image may be a representative shoe image obtained from the first image (901). The prompt may be an image generation request that replaces the coordinate area (933) with the third shoe image (935). The processor (120) may transmit the generated prompt to a generative AI, and an image may be generated by the generative AI. The generative AI may generate an image for the coordinate area (933) by referring to the color, pattern, texture, and design of the shoe, which is the third shoe image (935). The third user interface (950) may include an image (951) generated by the generative AI. The generated image (951) may be a result image of in-painting an area including an animal (915).

[0137] An operating method of an electronic device (101) according to an embodiment of the present disclosure may include an operation of recognizing an object included in an image displayed on a display (160) of the electronic device, an operation of searching a face clustering database (DB) through a face clustering process of the recognized person when the recognized object is a person, an operation of determining whether face correction of the recognized person in the displayed image is required, an operation of updating coordinate system rotation information of the person in a personalized database (DB) based on a face cluster identifier retrieved from the face clustering DB when face correction of the recognized person in the displayed image is not required, an operation of providing a list of recommended faces corresponding to the recognized person from the personalized DB in response to a user request when face correction of the recognized person in the displayed image is required, an operation of generating a prompt requesting face generation of the recognized person based on a face selected from the recommended face list based on a user input and the displayed image, and an operation of correcting and providing a face generated by a generative AI by the prompt to the face of the recognized person in the displayed image. The coordinate rotation information of the above person may include coordinate values, yaw, pitch, and roll values ​​on the image of the detected face.

[0138] In the above face clustering DB, a face cluster identifier assigned to each person and a storage location of a face image corresponding to the face cluster identifier are stored, and in the personalized DB, at least one of face information, upper body information, lower body information, or shoe information corresponding to the face cluster identifier is stored, and the face information may include at least one of size information, area information, 3D coordinate rotation information, and sharpness corresponding to the face.

[0139] The method may include an operation of extracting face size information and three-dimensional coordinate system rotation information corresponding to the recognized person, and an operation of searching for face information matching the extracted face size information and three-dimensional coordinate system rotation information in the personalized DB.

[0140] The above personalized DB may store at least one of tag information, coordinate area information, detail information, color information, or reliability corresponding to an item such as upper body information, lower body information, or shoe information.

[0141] The method may include an operation of identifying an item of the recognized person in the displayed image, an operation of analyzing the identified item to obtain coordinate area information corresponding to the item area, and an operation of extracting detailed information of the item including at least one of tag information, detail information, color information, or reliability of the identified item.

[0142] The method may further include an operation of searching whether an item corresponding to the detailed information of the extracted item is stored in the personalized DB, and an operation of storing coordinate area information and detailed information of the identified item in the personalized DB when an item corresponding to the detailed information of the extracted item is not searched in the personalized DB.

[0143] The method may further include, when an item corresponding to detailed information of the extracted item is searched from the personalized DB, an operation of comparing the reliability of the identified item with the reliability of the item searched from the personalized DB; when the reliability of the item searched from the personalized DB is higher than the reliability of the identified item as a result of the comparison; an operation of maintaining a representative image corresponding to the searched item; and when the reliability of the identified item is higher than the reliability of the item searched from the personalized DB as a result of the comparison; an operation of updating a representative image stored in a representative image database (DB) corresponding to the identified item.

[0144] The method may further include, if the recognized person is not retrieved from the face clustering database, providing a representative image list including a representative image corresponding to each person from a representative image database (DB), selecting a representative image from the provided representative image list based on a user input, and generating a prompt requesting face generation of the recognized person based on the selected representative image and the image.

[0145] The above representative image DB may store a representative face image corresponding to the face cluster identifier and a representative item image for each person item.

[0146] The above generating operation may include an operation of generating the prompt including the facial coordinate information of the selected face, the displayed image, and the recognized person, or an operation of generating the prompt including a masking area including the selected face and the recognized person.

[0147] The various embodiments of the present invention disclosed in this specification and drawings are merely specific examples presented to facilitate easy explanation of the technical content of the present invention and aid understanding thereof, and are not intended to limit the scope of the present invention. Therefore, the scope of the present invention should be interpreted to include all modifications or variations derived based on the technical concept of the present invention, in addition to the embodiments disclosed herein.

Claims

1. In an electronic device (101), Display (160), Memory (130) for storing instructions; and A processor (120) is included, and the instructions, when executed by the processor, cause the electronic device to: Recognize objects contained in the image displayed on the above display, If the above recognized object is a person, the face clustering database (DB) is searched through the face clustering process of the recognized person, Determine whether facial correction of the recognized person in the image displayed above is required, If the face correction of the recognized person in the image displayed above is not required, the coordinate rotation information of the person is updated in the personalized database (DB) based on the face cluster identifier retrieved from the face clustering DB, If facial correction of the recognized person in the image displayed above is required, a list of recommended faces corresponding to the recognized person is provided from the personalized DB at the user's request, Generate a prompt requesting the generation of a face of the recognized person based on a face selected from the above recommended face list based on user input and the above displayed image, An electronic device that provides a face generated by a generative AI by the above prompt, corrected to the face of a person recognized in the image displayed above.

2. In paragraph 1, In the above face clustering DB, a face cluster identifier assigned to each person and a storage location of a face image corresponding to the face cluster identifier are stored, An electronic device in which at least one of face information, upper body information, lower body information, or shoe information corresponding to the face cluster identifier is stored in the personalized DB, and the face information includes at least one of size information, area information, 3D coordinate rotation information, and clarity corresponding to the face.

3. In the second paragraph, when the instructions are executed by the processor, the electronic device, Extracting the size information of the face corresponding to the above-mentioned recognized person and the three-dimensional coordinate rotation information, An electronic device that searches for facial information matching the extracted facial size information and three-dimensional coordinate rotation information in the personalized DB.

4. In paragraph 2, An electronic device in which at least one of tag information, coordinate area information, detail information, color information, or reliability is stored in the above personalized DB corresponding to an item such as upper body information, lower body information, or shoe information.

5. In the fourth paragraph, when the instructions are executed by the processor, the electronic device, Identify the item of the recognized person in the image displayed above, By analyzing the above identified items, coordinate area information corresponding to the item area is obtained, An electronic device for extracting detailed information of an item, including at least one of tag information, detail information, color information, or reliability of the identified item.

6. In the fifth paragraph, when the instructions are executed by the processor, the electronic device, Search whether an item corresponding to the detailed information of the above extracted item is stored in the personalized DB, An electronic device that stores coordinate area information and detailed information of the identified item in the personalized DB when an item corresponding to the detailed information of the extracted item is not searched in the personalized DB.

7. In the sixth paragraph, when the instructions are executed by the processor, the electronic device, When an item corresponding to the detailed information of the extracted item is searched in the personalized DB, the reliability of the identified item is compared with the reliability of the item searched in the personalized DB, As a result of the above comparison, if the reliability of the item searched from the personalized DB is higher than the reliability of the identified item, the representative image corresponding to the searched item is maintained, An electronic device that updates a representative image stored in a representative image database (DB) corresponding to the identified item when, as a result of the comparison, the reliability of the identified item is higher than the reliability of the item searched in the personalized DB.

8. In the first paragraph, when the instructions are executed by the processor, the electronic device, If the above recognized person is not retrieved from the face clustering database, a representative image list including a representative image corresponding to each person is provided from a representative image database (DB). Select a representative image based on user input from the representative image list provided above, An electronic device that generates a prompt requesting the generation of a face of the recognized person based on the selected representative image and the image.

9. In paragraph 8, An electronic device in which a representative face image corresponding to the face cluster identifier and a representative item image for each person are stored in the representative image DB.

10. In the first paragraph, when the instructions are executed by the processor, the electronic device, Generate the prompt including the selected face, the displayed image, and the facial coordinate information of the recognized person, or An electronic device that generates the prompt including the selected face and the masking area including the recognized person.

11. In the operating method of an electronic device (101), An operation of recognizing an object included in an image displayed on a display (160) of the electronic device; If the above recognized object is a person, an operation of searching from a face clustering database (DB) through a face clustering process of the above recognized person; An action for determining whether facial correction of the recognized person in the image displayed above is required; If facial correction of the recognized person in the image displayed above is not required, an operation of updating coordinate rotation information of the person in a personalized database (DB) based on a face cluster identifier retrieved from the face clustering DB; An operation of providing a list of recommended faces corresponding to the recognized person from the personalized DB at the user's request when facial correction of the recognized person in the image displayed above is required; An action of generating a prompt requesting the generation of a face of the recognized person based on a face selected from the above recommended face list based on user input and the displayed image; and A method including an action of providing a face generated by a generative AI by the above prompt by correcting it to the face of a person recognized in the displayed image.

12. In paragraph 11, In the above face clustering DB, a face cluster identifier assigned to each person and a storage location of a face image corresponding to the face cluster identifier are stored, A method in which at least one of face information, upper body information, lower body information, or shoe information corresponding to the face cluster identifier is stored in the personalized DB, and the face information includes at least one of size information, area information, 3D coordinate rotation information, and sharpness corresponding to the face.

13. In paragraph 12, An operation of extracting face size information and three-dimensional coordinate rotation information corresponding to the recognized person; and A method including an operation of searching for face information matching the size information of the extracted face and the three-dimensional coordinate system rotation information in the personalized DB.

14. In paragraph 12, A method in which at least one of tag information, coordinate area information, detail information, color information, or reliability is stored in the above personalized DB corresponding to an item such as upper body information, lower body information, or shoe information.

15. In paragraph 14, An action of identifying an item of the recognized person within the image displayed above; An operation of analyzing the above-mentioned identified item to obtain coordinate area information corresponding to the item area; and A method comprising an action of extracting detailed information of an item including at least one of tag information, detail information, color information, or reliability of the identified item.

Images