Electronic device for providing generative service with improved specification performance and control method therefor

Abstract

An electronic device is disclosed. The present electronic device comprises: a memory in which a first encoder and a second encoder obtained by additionally learning a first original encoder and a second original encoder included in a generative neural network model are stored; and one or more processors that are connected to the memory and control the electronic device, wherein the processor inputs a prompt to the first encoder to obtain a first embedding vector in which the prompt is projected onto a multi-dimensional space, inputs an input image to the second encoder to obtain a second embedding vector in which the input image is projected onto the multi-dimensional space, and obtains a final image generated using a decoder included in the generative neural network model, on the basis of the first embedding vector and the second embedding vector, wherein the first original encoder and the second original encoder are trained on the basis of a first sample prompt of a first layer and a first sample input image corresponding to the first sample prompt, and the first encoder and the second encoder may be obtained by additionally learning the first original encoder and the second original encoder on the basis of a second sample prompt of a second layer lower than the first layer and a second sample input image corresponding to the second sample prompt.

Description

Electronic device and control method thereof for providing a generative service with improved specific performance

[0001] The present disclosure relates to an electronic device and a control method thereof, and more particularly, to an electronic device and a control method thereof that provide a generative service with improved specific performance.

[0002] Advances in electronic technology have led to the development of electronic devices offering a variety of functions. In particular, generative artificial intelligence (AI) has recently been developed, enhancing user convenience.

[0003] Generative AI is an AI technology that uses existing content such as text, audio, and images to create new similar content. It is also called generative AI.

[0004] However, conventional generative artificial intelligence is insufficient in terms of expressing specific objects.

[0005] Accordingly, methods have been developed to increase the specificity performance by additionally learning the decoder included in generative artificial intelligence, such as Dreambooth, LoRA, HyperNetworks, and Textual Inversion, but there is a problem that the generalization performance is reduced and a significant learning cost is incurred.

[0006] In general, a generative neural network model can represent input data of different types in a common embedding space. For example, a generative neural network model can represent different types of input data, such as images, text, and voice, in a common embedding space, and can be expressed as projecting the input data into the embedding space. Here, a generative neural network model may include encoders for each type. For example, a generative neural network model may include a first encoder that receives text and projects it into the embedding space, and a second encoder that receives an image and projects it into the embedding space.

[0007] The present disclosure maintains the characteristic expression ability (generalization) of the existing generalization learning embedding space and adds personal expression data, and this is done through the method described below.

[0008] According to one embodiment of the present disclosure for achieving the above object, an electronic device includes a memory storing a first encoder and a second encoder obtained by further learning a first original encoder and a second original encoder included in a generative neural network model, and one or more processors connected to the memory and controlling the electronic device, wherein the processor inputs a prompt to the first encoder to obtain a first embedding vector in which the prompt is projected into a multidimensional space, inputs an input image to the second encoder to obtain a second embedding vector in which the input image is projected into the multidimensional space, and obtains a final image generated using a decoder included in the generative neural network model based on the first embedding vector and the second embedding vector, wherein the first original encoder and the second original encoder are trained based on a first sample prompt of a first layer and a first sample input image corresponding to the first sample prompt, and the first encoder and the second encoder are connected to a second layer which is a lower layer of the first layer. The first original encoder and the second original encoder can be further trained based on a second sample prompt of the layer and a second sample input image corresponding to the second sample prompt.

[0009] Additionally, the additional learning may be a method of fixing one of the first original encoder and the second original encoder and performing additional learning on the remaining one of the first original encoder and the second original encoder.

[0010] And, one of the first encoder and the second encoder may be identical to one of the first original encoder and the second original encoder, and the remaining of the first encoder and the second encoder may be additionally learned encoders.

[0011] Additionally, the decoder may be trained to generate an image based on an output of at least one of the first original encoder and the second original encoder.

[0012] And, further comprising a user interface, wherein the processor can receive a user command corresponding to a portion of the input image through the user interface, update the second embedding vector based on the user command, and obtain the final image based on the first embedding vector and the updated second embedding vector.

[0013] Additionally, the processor can input the user command to the first encoder to obtain a third embedding vector in which the user command is projected into the multidimensional space, and update the second embedding vector based on the third embedding vector.

[0014] And, the processor can obtain a fourth embedding vector by geometrically averaging the first embedding vector and the second embedding vector for each corresponding element, and obtain the final image generated by inputting the fourth embedding vector into the decoder.

[0015] In addition, the processor further includes a communication interface, and the processor controls the communication interface to transmit the first embedding vector and the second embedding vector to a server, and can receive the final image from the server through the communication interface.

[0016] And, the decoder may have a capacity greater than the sum of the capacity of the first original encoder and the capacity of the second original encoder or the sum of the capacity of the first encoder and the capacity of the second encoder.

[0017] Meanwhile, according to one embodiment of the present disclosure, a control method of an electronic device includes a step of inputting a prompt to a first encoder obtained by additionally learning a first original encoder and a second original encoder included in a generative neural network model to obtain a first embedding vector in which the prompt is projected into a multidimensional space, a step of inputting an input image to a second encoder obtained by additionally learning the first original encoder and the second original encoder to obtain a second embedding vector in which the input image is projected into the multidimensional space, and a step of obtaining a final image generated using a decoder included in the generative neural network model based on the first embedding vector and the second embedding vector, wherein the first original encoder and the second original encoder are trained based on a first sample prompt of a first layer and a first sample input image corresponding to the first sample prompt, and the first encoder and the second encoder are trained based on a second sample prompt of a second layer which is a lower layer of the first layer and the second sample It can be obtained by further learning the first original encoder and the second original encoder based on a second sample input image corresponding to the prompt.

[0018] Additionally, the additional learning may be a method of fixing one of the first original encoder and the second original encoder and performing additional learning on the remaining one of the first original encoder and the second original encoder.

[0019] And, one of the first encoder and the second encoder may be identical to one of the first original encoder and the second original encoder, and the remaining of the first encoder and the second encoder may be additionally learned encoders.

[0020] Additionally, the decoder may be trained to generate an image based on an output of at least one of the first original encoder and the second original encoder.

[0021] And, the step of receiving a user command corresponding to a portion of the input image and the step of updating the second embedding vector based on the user command are further included, and the step of obtaining the final image can obtain the final image based on the first embedding vector and the updated second embedding vector.

[0022] Additionally, the updating step may input the user command into the first encoder to obtain a third embedding vector in which the user command is projected into the multidimensional space, and update the second embedding vector based on the third embedding vector.

[0023] And, the step of obtaining the final image may obtain the fourth embedding vector by geometrically averaging the first embedding vector and the second embedding vector for each corresponding element, and obtain the final image generated by inputting the fourth embedding vector into the decoder.

[0024] Additionally, the step of obtaining the final image may include transmitting the first embedding vector and the second embedding vector to a server and receiving the final image from the server.

[0025] And, the decoder may have a capacity greater than the sum of the capacity of the first original encoder and the capacity of the second original encoder or the sum of the capacity of the first encoder and the capacity of the second encoder.

[0026] Meanwhile, according to one embodiment of the present disclosure, a non-transitory computer-readable recording medium storing a program for executing an operating method of an electronic device includes a step of inputting a prompt into a first encoder obtained by additionally learning a first original encoder and a second original encoder included in a generative neural network model to obtain a first embedding vector in which the prompt is projected into a multidimensional space, a step of inputting an input image into a second encoder obtained by additionally learning the first original encoder and the second original encoder to obtain a second embedding vector in which the input image is projected into the multidimensional space, and a step of obtaining a final image generated using a decoder included in the generative neural network model based on the first embedding vector and the second embedding vector, wherein the first original encoder and the second original encoder are trained based on a first sample prompt of a first layer and a first sample input image corresponding to the first sample prompt, and the first encoder and the second encoder are trained based on the The first original encoder and the second original encoder can be additionally trained based on a second sample prompt of a second layer, which is a lower layer of the first layer, and a second sample input image corresponding to the second sample prompt.

[0027] FIG. 1 is a block diagram showing the configuration of an electronic device according to one embodiment of the present disclosure.

[0028] FIG. 2 is a block diagram showing a detailed configuration of an electronic device according to an embodiment of the present disclosure.

[0029] FIG. 3 is a diagram illustrating a generative neural network model according to an embodiment of the present disclosure.

[0030] FIGS. 4 to 7 are drawings for explaining operations according to additional learning according to one embodiment of the present disclosure.

[0031] FIG. 8 is a diagram illustrating a method for processing embedding vectors output from a first encoder and a second encoder according to an embodiment of the present disclosure.

[0032] FIGS. 9 and 10 are drawings for explaining a method for specifying a portion of an input image according to an embodiment of the present disclosure.

[0033] FIG. 11 is a block diagram showing the configuration of an electronic system according to one embodiment of the present disclosure.

[0034] FIG. 12 is a flowchart for explaining a control method of an electronic device according to an embodiment of the present disclosure.

[0035] The purpose of the present disclosure is to provide an electronic device and a control method thereof for providing a generative service with improved specificity performance while reducing learning costs.

[0036] It should be understood that the various embodiments and terms used in this document are not intended to limit the technical features described in this document to specific embodiments, but rather to include various modifications, equivalents, or substitutes of the embodiments.

[0037] In connection with the description of the drawings, similar reference numerals may be used for similar or related components.

[0038] The singular form of a noun corresponding to an item may include one or more items, unless the context clearly indicates otherwise.

[0039] In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" may include any one of the items listed together in that phrase, or all possible combinations thereof.

[0040] Terms such as "first," "second," or "first" or "second" may be used simply to distinguish one component from another and do not qualify the components in any other respect (e.g., importance or order).

[0041] When a component (e.g., a first component) is referred to as being “coupled” or “connected” to another component (e.g., a second component), with or without the terms “functionally” or “communicatively,” it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0042] The terms “include” or “have” are intended to specify the presence of a feature, number, step, operation, component, part or combination thereof described in this document, but do not preclude the presence or addition of one or more other features, numbers, steps, operations, components, parts or combinations thereof.

[0043] When a component is said to be “connected,” “coupled,” “supported,” or “in contact with” another component, this includes not only cases where the components are directly connected, coupled, supported, or in contact, but also cases where the components are indirectly connected, coupled, supported, or in contact through a third component.

[0044] When we say that a component is “on” another component, this includes not only cases where the component is in contact with the other component, but also cases where there is another component between the two components.

[0045] The term “and / or” includes any combination of a plurality of related described elements or any one of a plurality of related described elements.

[0046] The operating principle and embodiments of the present invention will be described with reference to the attached drawings below.

[0047] FIG. 1 is a block diagram showing the configuration of an electronic device (100) according to one embodiment of the present disclosure.

[0048] The electronic device (100) is a device that provides a generative service and may be implemented as a TV, desktop PC, laptop, video wall, large format display (LFD), digital signage, digital information display (DID), projector display, smartphone, tablet PC, etc. For example, the electronic device (100) may be a device that provides a prompt and a final image generated from an input image. Here, the prompt is a type of instruction message and may be text.

[0049] However, it is not limited thereto, and the electronic device (100) may be any device that provides a creation service.

[0050] According to FIG. 1, the electronic device (100) includes a memory (110) and a processor (120). However, the present invention is not limited thereto, and the electronic device (100) may be implemented in a form in which some components are excluded.

[0051] Memory (110) may refer to hardware that stores information such as data in an electrical or magnetic form so that a processor (120) or the like can access it. To this end, memory (110) may be implemented as at least one piece of hardware from among non-volatile memory, volatile memory, flash memory, hard disk drive (HDD), solid state drive (SSD), RAM, ROM, etc.

[0052] The memory (110) may store at least one instruction required for the operation of the electronic device (100) or the processor (120). Here, the instruction is a code unit that instructs the operation of the electronic device (100) or the processor (120), and may be written in machine language, which is a language that a computer can understand. Alternatively, the memory (110) may store EDID and DPCD for the display (120).

[0053] The memory (110) may store data, which is information in bit or byte units that can represent characters, numbers, images, etc. For example, the memory (110) may store a first encoder and a second encoder. Here, the first encoder and the second encoder may be encoders obtained by additionally learning the first original encoder and the second original encoder included in the generative neural network model. Alternatively, the memory (110) may further store a decoder included in the generative neural network model.

[0054] The memory (110) is accessed by the processor (120), and reading / writing / modifying / deleting / updating instructions, instruction sets, or data can be performed by the processor (120).

[0055] The processor (120) controls the overall operation of the electronic device (100). Specifically, the processor (120) is connected to each component of the electronic device (100) and can control the overall operation of the electronic device (100). For example, the processor (120) is connected to components such as a memory (110), a user interface (not shown), and the like and can control the operation of the electronic device (100).

[0056] The processor (120) may be implemented with one or more processors. In this case, the one or more processors may include one or more of a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an APU (Accelerated Processing Unit), a MIC (Many Integrated Core), a DSP (Digital Signal Processor), an NPU (Neural Processing Unit), a hardware accelerator, or a machine learning accelerator. The one or more processors may control one or any combination of other components of the electronic device (100) and perform operations related to communication or data processing. The one or more processors may execute one or more programs or instructions stored in the memory (110). For example, the one or more processors may perform a method according to an embodiment of the present disclosure by executing one or more instructions stored in the memory (110).

[0057] When a method according to an embodiment of the present disclosure includes a plurality of operations, the plurality of operations may be performed by one processor or by a plurality of processors. For example, when a first operation, a second operation, and a third operation are performed by a method according to an embodiment, the first operation, the second operation, and the third operation may all be performed by a first processor, or the first operation and the second operation may be performed by a first processor (e.g., a general-purpose processor) and the third operation may be performed by a second processor (e.g., an AI-specific processor). For example, a process of quantizing a neural network model according to an embodiment of the present disclosure may be performed by a general-purpose processor, and a process of learning or inferring the quantized neural network model may be performed by an AI-specific processor.

[0058] One or more processors may be implemented as a single core processor including one core, or may be implemented as one or more multicore processors including multiple cores (e.g., homogeneous multicore or heterogeneous multicore). When one or more processors are implemented as a multicore processor, each of the multiple cores included in the multicore processor may include internal processor memory, such as cache memory or on-chip memory, and a common cache shared by the multiple cores may be included in the multicore processor. In addition, each of the multiple cores (or some of the multiple cores) included in the multicore processor may independently read and execute a program instruction for implementing a method according to an embodiment of the present disclosure, or all (or some) of the multiple cores may be linked to read and execute a program instruction for implementing a method according to an embodiment of the present disclosure.

[0059] When a method according to an embodiment of the present disclosure includes a plurality of operations, the plurality of operations may be performed by one core among the plurality of cores included in a multi-core processor, or may be performed by the plurality of cores. For example, when a first operation, a second operation, and a third operation are performed by a method according to an embodiment, the first operation, the second operation, and the third operation may all be performed by a first core included in the multi-core processor, or the first operation and the second operation may be performed by a first core included in the multi-core processor, and the third operation may be performed by a second core included in the multi-core processor.

[0060] In embodiments of the present disclosure, one or more processors may refer to a system on a chip (SoC) in which one or more processors and other electronic components are integrated, a single-core processor, a multi-core processor, or a core included in a single-core processor or a multi-core processor, wherein the core may be implemented as a CPU, a GPU, an APU, a MIC, a DSP, an NPU, a hardware accelerator, or a machine learning accelerator, but the embodiments of the present disclosure are not limited thereto. However, for convenience of explanation, the operation of the electronic device (100) is described below using the expression processor (120).

[0061] The processor (120) can input a prompt to a first encoder to obtain a first embedding vector in which the prompt is projected into a multidimensional space, input an input image to a second encoder to obtain a second embedding vector in which the input image is projected into a multidimensional space, and obtain a final image generated using a decoder included in a generative neural network model based on the first embedding vector and the second embedding vector.

[0062] Here, the first original encoder and the second original encoder can be trained based on the first sample prompt of the first layer and the first sample input image corresponding to the first sample prompt. For example, when the first sample prompt is input, the first original encoder outputs an embedding vector in which the first sample prompt is projected into a multidimensional space, and when the second original encoder inputs the first sample input image, the second original encoder outputs an embedding vector in which the first sample input image is projected into a multidimensional space, and the first original encoder and the second original encoder can be trained so that the values ​​of the two embedding vectors become identical.

[0063] The first encoder and the second encoder can be obtained by further learning the first original encoder and the second original encoder based on the second sample prompt of the second layer, which is a lower layer of the first layer, and the second sample input image corresponding to the second sample prompt. For example, if the first original encoder and the second original encoder have learned the first sample prompt, "Western man sitting on a chair", and the first sample input image corresponding to the first sample prompt, the first encoder and the second encoder can be obtained by further learning the second sample prompt, "Western man Tom sitting on a chair", and the second sample input image corresponding to the second sample prompt. The additional learning method can be the same as the learning method described above. For example, when a second sample prompt is input, the first original encoder outputs an embedding vector in which the second sample prompt is projected into a multidimensional space, and when a second sample input image is input, the second original encoder outputs an embedding vector in which the second sample input image is projected into a multidimensional space, and the first original encoder and the second original encoder can be trained so that the values ​​of the two embedding vectors become the same.

[0064] Here, hierarchy can refer to the degree of specificity of the content contained in the prompt. As described above, the prompt "Western man sitting on a chair" can be a higher-level prompt than the prompt "Western man Tom sitting on a chair." Hierarchy can also be referred to as depth. Alternatively, hierarchy can refer to the granularity of the object properties contained in the prompt. However, this is not limited to this, and any term that indicates the degree of specificity of the content contained in the prompt can be used instead of hierarchy.

[0065] Additional learning may be a method of fixing one of the first original encoder and the second original encoder, and further learning the remaining one of the first original encoder and the second original encoder. That is, when learning is completed, one of the first encoder and the second encoder may be identical to one of the first original encoder and the second original encoder, and the remaining one of the first encoder and the second encoder may be an additionally learned encoder. For example, if the second original encoder is additionally learned while the first original encoder is fixed, the first encoder may be identical to the first original encoder, and the second encoder may be an additionally learned second original encoder. Alternatively, if the first original encoder is further trained while the second original encoder is fixed, the first encoder may be the further trained first original encoder, and the second encoder may be identical to the second original encoder.

[0066] At least one of the above learning or additional learning may be performed on a server. However, this is not limited to the above, and at least one of the learning or additional learning may also be performed on an electronic device (100).

[0067] The decoder can be trained to generate an image based on the output of at least one of the first source encoder and the second source encoder.

[0068] That is, since the decoder is trained to generate images based on the first original encoder and the second original encoder before additional learning is performed, the output may change if the first original encoder and the second original encoder are changed. However, according to the present disclosure, since one of the first original encoder and the second original encoder is fixed during the additional learning process, the existing learning state can be maintained. Nevertheless, the specialization performance can be improved by additional learning for the remaining first original encoder and the second original encoder.

[0069] The electronic device (100) further includes a user interface, and the processor (120) can receive a user command corresponding to a portion of an input image through the user interface, update a second embedding vector based on the user command, and obtain a final image based on the first embedding vector and the updated second embedding vector.

[0070] For example, the processor (120) may input a user command to the first encoder to obtain a third embedding vector in which the user command is projected into a multidimensional space, and update the second embedding vector based on the third embedding vector. For example, the processor (120) may obtain an updated second embedding vector by multiplying the second embedding vector and the third embedding vector by corresponding elements.

[0071] The processor (120) can obtain a fourth embedding vector by geometrically averaging the first embedding vector and the second embedding vector for each corresponding element, and obtain a final image generated by inputting the fourth embedding vector to the decoder. In this case, the first embedding vector, the second embedding vector, and the fourth embedding vector may have the same vector size. For example, each of the first embedding vector, the second embedding vector, and the fourth embedding vector may be in the form of 1×n. Here, n may be a natural number.

[0072] However, the present invention is not limited thereto, and the processor (120) may obtain the fourth embedding vector from the first embedding vector and the second embedding vector through various methods. For example, the processor (120) may obtain the fourth embedding vector by taking the arithmetic mean of the first embedding vector and the second embedding vector for each corresponding element.

[0073] The electronic device (100) further includes a communication interface, and the processor (120) controls the communication interface to transmit the first embedding vector and the second embedding vector to the server, and can receive the final image from the server through the communication interface. That is, the server can store the decoder, obtain the final image, and provide it to the electronic device (100). Here, the decoder may have a capacity greater than the sum of the capacities of the first original encoder and the second original encoder, or the sum of the capacities of the first encoder and the second encoder. That is, since the electronic device (100) only needs to store the first encoder and the second encoder, the storage space of the electronic device (100) can be saved. In addition, since only the first original encoder and the second original encoder, which have relatively small capacities compared to the decoder, are additionally trained, the training cost can be reduced. In addition, since the information transmitted to the server is the first embedding vector and the second embedding vector, personal information can be protected.

[0074] However, this is not limited to this, and the electronic device (100) may also store the decoder. In this case, the processor (120) may directly use the decoder to obtain the final image. In addition, the processor (120) may control the communication interface to transmit the fourth embedding vector to the server, rather than the first and second embedding vectors.

[0075] The functions related to artificial intelligence according to the present disclosure can be operated through a processor (120) and a memory (110).

[0076] The processor (120) may be composed of one or more processors. In this case, the one or more processors may be a general-purpose processor such as a CPU, AP, DSP, etc., a graphics-only processor such as a GPU or VPU (Vision Processing Unit), or an artificial intelligence-only processor such as an NPU.

[0077] One or more processors are controlled to process input data according to predefined operating rules or artificial intelligence models stored in the memory (110). Alternatively, if one or more processors are dedicated artificial intelligence processors, the dedicated artificial intelligence processors may be designed with a hardware structure specialized for processing a specific artificial intelligence model. The predefined operating rules or artificial intelligence models are characterized by being created through learning.

[0078] Here, "created through learning" means that a basic artificial intelligence model is learned using a learning algorithm using a plurality of learning data, thereby creating a predefined set of operating rules or an artificial intelligence model set to perform a desired characteristic (or purpose). This learning may be performed on the device itself on which the artificial intelligence according to the present disclosure is performed, or may be performed through a separate server and / or system. Examples of learning algorithms include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

[0079] An artificial intelligence model may be composed of multiple neural network layers. Each of the multiple neural network layers has multiple weight values ​​and performs neural network operations by calculating the results of previous layers and the multiple weights. The multiple weights of the multiple neural network layers can be optimized based on the learning results of the artificial intelligence model. For example, the multiple weights may be updated during the learning process to reduce or minimize the loss or cost values ​​obtained by the artificial intelligence model.

[0080] Artificial neural networks may include deep neural networks (DNNs), such as, but not limited to, convolutional neural networks (CNNs), deep neural networks (DNNs), recurrent neural networks (RNNs), restricted boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), generative adversarial networks (GANs), or deep Q-networks.

[0081] FIG. 2 is a block diagram illustrating a detailed configuration of an electronic device (100) according to an embodiment of the present disclosure. The electronic device (100) may include a memory (110) and a processor (120). The electronic device (100) may further include a user interface (130), a communication interface (140), a display (150), a camera (160), a microphone (170), and a speaker (180). Among the components illustrated in FIG. 2, a detailed description of parts that overlap with the components illustrated in FIG. 1 will be omitted.

[0082] The user interface (130) may be implemented with buttons, a touch pad, a mouse, a keyboard, etc., or may be implemented with a touch screen capable of performing both display and operation input functions. Here, the buttons may be various types of buttons, such as mechanical buttons, touch pads, wheels, etc., formed in any area of ​​the front, side, or back of the main body of the electronic device (100).

[0083] The communication interface (140) is a configuration that performs communication with various types of external devices according to various types of communication methods. For example, the electronic device (100) can perform communication with a server through the communication interface (140).

[0084] The communication interface (140) may include a Wi-Fi module, a Bluetooth module, an infrared communication module, a wireless communication module, etc. Here, each communication module may be implemented in the form of at least one hardware chip.

[0085] Wi-Fi and Bluetooth modules communicate via Wi-Fi and Bluetooth, respectively. When using a Wi-Fi or Bluetooth module, connection information, such as the SSID and session key, is first transmitted and received. This information is then used to establish a communication connection before various other information can be transmitted and received. Infrared communication modules use infrared data association (IrDA) technology, which wirelessly transmits data over short distances using infrared light, which lies between visible light and millimeter waves.

[0086] In addition to the above-described communication method, the wireless communication module may include at least one communication chip that performs communication according to various wireless communication standards such as zigbee, 3G (3rd Generation), 3GPP (3rd Generation Partnership Project), LTE (Long Term Evolution), LTE-A (LTE Advanced), 4G (4th Generation), 5G (5th Generation), etc.

[0087] Alternatively, the communication interface (140) may include a wired communication interface such as HDMI, DP, Thunderbolt, USB, RGB, D-SUB, DVI, etc.

[0088] In addition, the communication interface (140) may include at least one of a LAN (Local Area Network) module, an Ethernet module, or a wired communication module that performs communication using a pair cable, a coaxial cable, or an optical fiber cable.

[0089] The display (150) is a component that displays content and can be implemented as a variety of displays such as an LCD (Liquid Crystal Display), an OLED (Organic Light Emitting Diodes) display, a PDP (Plasma Display Panel), etc. The display (150) may also include a driving circuit, a backlight unit, etc. that can be implemented as a form such as an a-si TFT, an LTPS (low temperature poly silicon) TFT, an OTFT (organic TFT), etc. Meanwhile, the display (150) may be implemented as a touch screen combined with a touch sensor, a flexible display, a 3D display, etc.

[0090] The camera (160) is configured to capture still images or moving images. The camera (160) can capture still images at a specific point in time, but can also capture still images continuously. The camera (160) can capture images in at least one direction of the electronic device (100).

[0091] The camera (160) includes a lens, a shutter, an aperture, a solid-state image sensor, an AFE (Analog Front End), and a TG (Timing Generator). The shutter controls the time at which light reflected from a subject enters the camera (160), and the aperture mechanically increases or decreases the size of the opening through which light enters to control the amount of light incident on the lens. When the solid-state image sensor accumulates light reflected from a subject as a photocharge, the image generated by the photocharge is output as an electrical signal. The TG outputs a timing signal for reading out pixel data of the solid-state image sensor, and the AFE samples and digitizes the electrical signal output from the solid-state image sensor.

[0092] The processor (120) can use an image captured by a camera (160) as an input image.

[0093] The microphone (170) is configured to receive sound and convert it into an audio signal. The microphone (170) is electrically connected to the processor (120) and can receive sound under the control of the processor (120).

[0094] For example, the microphone (170) may be formed as an integrated unit integrated into the upper side, front side, side side, etc. of the electronic device (100). Alternatively, the microphone (170) may be provided in a remote control, etc., separate from the electronic device (100). In this case, the remote control may receive sound through the microphone (170) and provide the received sound to the electronic device (100).

[0095] The microphone (170) may include various configurations such as a microphone that collects sound in analog form, an amplifier circuit that amplifies the collected sound, an A / D conversion circuit that samples the amplified sound and converts it into a digital signal, and a filter circuit that removes noise components from the converted digital signal.

[0096] Meanwhile, the microphone (170) may be implemented in the form of a sound sensor, and any method may be used as long as it has a configuration capable of collecting sound.

[0097] The processor (120) may receive a user's spoken voice through a microphone (170) and obtain a prompt from the user's spoken voice.

[0098] The speaker (180) is a component that outputs various audio data processed by the processor (120) as well as various notification sounds and voice messages.

[0099] As described above, the electronic device (100) can provide a generative service with improved specificity performance while reducing learning costs by additionally learning the first original encoder and the second original encoder.

[0100] Hereinafter, the operation of the electronic device (100) will be described in more detail with reference to FIGS. 3 to 11. For convenience of explanation, individual embodiments are described in FIGS. 3 to 11. However, the individual embodiments of FIGS. 3 to 11 may be implemented in any combination or with the order changed.

[0101] FIG. 3 is a diagram illustrating a generative neural network model according to an embodiment of the present disclosure.

[0102] A generative neural network model may include an encoder and a decoder. For example, as illustrated in FIG. 3, a generative neural network model may include an encoder (Text Encoder, 310) and a decoder (Image Information Creator, 320), Image Decoder Creator, 330).

[0103] The encoder (310) may include a first encoder for projecting a prompt into a multidimensional space and a second encoder for projecting an input image into the multidimensional space. The first encoder may output a first embedding vector in which the prompt is projected into the multidimensional space, and the second encoder may output a second embedding vector in which the input image is projected into the multidimensional space.

[0104] The encoder (310) can output a fourth embedding vector based on the first embedding vector and the second embedding vector to the decoder (320, 330). For example, the encoder (310) can obtain a fourth embedding vector by geometrically averaging the first embedding vector and the second embedding vector for each corresponding element, and output the fourth embedding vector to the decoder (320, 330).

[0105] The decoder (320, 330) can generate a final image based on the fourth embedding vector and output the generated final image.

[0106] FIGS. 4 to 7 are drawings for explaining operations according to additional learning according to one embodiment of the present disclosure.

[0107] First, the first original encoder and the second original encoder can be trained based on the first sample prompt (text) of the first layer and the first sample input image (image) corresponding to the first sample prompt, as illustrated in FIG. 4. For example, the first original encoder receives the first sample prompt as input and outputs an embedding vector corresponding to the first sample prompt, and the second original encoder receives the first sample image as input and outputs an embedding vector corresponding to the first sample image, and the first original encoder and the second original encoder can be trained so that the two embedding vectors become identical.

[0108] However, since the first sample prompt is a first-layer prompt, the characterization performance may be relatively low. For example, as illustrated in Fig. 5, the first original encoder and the second original encoder may have learned the first sample prompt, such as "Western man", "Eastern man", etc., and the first sample input image corresponding to the first sample prompt. In this case, when the prompt "Tom" is input to the first original encoder or the input image corresponding to "Tom" is input to the second original encoder, the encoder can output information identifying whether it is a "Western man" or an "Eastern man", but cannot output information about "Tom".

[0109] On the other hand, the specificity performance can be further improved by using the first encoder and the second encoder obtained by additionally learning the first original encoder and the second original encoder.

[0110] First, the first original encoder and the second original encoder may have completed training in the same manner as in FIG. 4. Thereafter, the first original encoder and the second original encoder may be additionally trained to obtain the first encoder and the second encoder. For example, the first original encoder and the second original encoder may be additionally trained based on the second sample prompt (text) of the second layer, which is a lower layer of the first layer, and the second sample input image (image) corresponding to the second sample prompt, as illustrated in FIG. 6. For example, the first original encoder may receive the second sample prompt and output an embedding vector corresponding to the second sample prompt, and the second original encoder may receive the second sample image and output an embedding vector corresponding to the second sample image, and the first original encoder and the second original encoder may be trained so that the two embedding vectors become identical.

[0111] At this time, the additional learning may be a method of fixing one of the first original encoder and the second original encoder and additionally learning the remaining one of the first original encoder and the second original encoder. Accordingly, one of the first encoder and the second encoder may be identical to one of the first original encoder and the second original encoder, and the remaining one of the first encoder and the second encoder may be additionally learned encoders. For example, if the second original encoder is additionally learned while the first original encoder is fixed, the first encoder may be identical to the first original encoder, and the second encoder may be the additionally learned second original encoder. Alternatively, if the first original encoder is additionally learned while the second original encoder is fixed, the first encoder may be the additionally learned first original encoder, and the second encoder may be identical to the second original encoder.

[0112] For convenience of explanation, assuming that the first encoder is identical to the first original encoder, the second encoder may have learned second sample input images such as "Western man", "Eastern man", as well as "Jack", "John", etc., as illustrated in FIG. 7. In this case, when an input image (710) containing "Tom" is input to the second encoder, the second encoder may output information related to xx (e.g., nose) of "Jack" and yy (e.g., mouth) of "John" while being a "Western man".

[0113] Here, "Western man" or "Eastern man" may be anchors derived from the training results of the first and second original encoders. An anchor may be a point where the object's characteristics are projected into a multidimensional space.

[0114] According to the present disclosure, since one of the first and second original encoders is fixed during the additional learning process, anchors from the original learning process can be maintained. Subsequently, when additional learning is performed, the second original encoder, updated through the additional learning, can identify additional anchors, such as "Jack" and "John." In other words, by maintaining the existing anchors, the existing performance can be maintained even without additional learning of the decoder, and the additional anchors can improve the specificity performance.

[0115] FIG. 8 is a diagram illustrating a method for processing embedding vectors output from a first encoder and a second encoder according to an embodiment of the present disclosure.

[0116] The processor (120) can obtain a fourth embedding vector based on the first embedding vector and the second embedding vector. For example, as illustrated in FIG. 8, the processor (120) can obtain the fourth embedding vector by performing a geometric mean (810) on the first embedding vector and the second embedding vector for each corresponding element.

[0117] The processor (120) can operate so that the fourth embedding vector is processed in the decoder.

[0118] FIGS. 9 and 10 are drawings for explaining a method for specifying a portion of an input image according to an embodiment of the present disclosure.

[0119] The processor (120) may receive a user command corresponding to a portion of an input image and update a second embedding vector based on the user command. For example, as illustrated in FIG. 9, the processor (120) may input an input image to a second encoder to obtain a second embedding vector, receive a user command corresponding to a portion of the input image, input the user command to a first encoder to obtain a third embedding vector, and perform a multiplication operation (910) on the second embedding vector and the third embedding vector for each corresponding element to obtain an updated second embedding vector. For example, when a user command to use the person on the left is received in an input image such as FIG. 10, the first encoder outputs a third embedding vector with a high weight in the left area according to the user command, and when the second embedding vector corresponding to the input image and the third embedding vector are multiplied by corresponding elements, an updated second embedding vector in which the person on the left (1010) is more prominent than the person on the right (1020) can be obtained.

[0120] The processor (120) may input a prompt to the first encoder to obtain a first embedding vector, and may obtain a fourth embedding vector by performing a geometric mean (920) on the first embedding vector and the updated second embedding vector for each corresponding element. In the above-described example, the fourth embedding vector may include information about the person (1010) on the left as main information.

[0121] In Fig. 9, for convenience of explanation, the first encoder is illustrated as having two first encoders, but the first encoder may be implemented as a single first encoder. That is, when a prompt and an input image are received, the processor (120) may input the prompt to the first encoder to obtain a first embedding vector, and input the input image to the second encoder to obtain a second embedding vector. The processor (120) may store the first embedding vector in the memory (110), and when a user command corresponding to a portion of the input image is received, the processor may input the user command to the first encoder to obtain a third embedding vector. When operating in this manner, the first encoder may operate as a single first encoder.

[0122] FIG. 11 is a block diagram showing the configuration of an electronic system according to one embodiment of the present disclosure.

[0123] The electronic system may include an electronic device (100) and a server (200).

[0124] The electronic device (100) includes a first encoder and a second encoder, and can transmit a first embedding vector and a second embedding vector output from the first encoder and the second encoder, respectively, to the server (200). Alternatively, the electronic device (100) can also transmit a fourth embedding vector obtained by geometrically averaging the first embedding vector and the second embedding vector for each corresponding element to the server (200).

[0125] The server (200) stores the decoder and can receive the first embedding vector and the second embedding vector or the fourth embedding vector from the electronic device (100). When the first embedding vector and the second embedding vector are received, the server (200) can obtain a fourth embedding vector by geometrically averaging the first embedding vector and the second embedding vector for each corresponding element, and input the fourth embedding vector to the decoder to obtain a final image. Alternatively, when the fourth embedding vector is received, the server (200) can input the fourth embedding vector to the decoder to obtain a final image. The server (200) can transmit the final image to the electronic device (100).

[0126] Through this operation, the electronic device (100) can save storage space by storing only the first encoder and the second encoder, which have relatively small capacities compared to the decoder. In addition, the electronic device (100) provides the first embedding vector and the second embedding vector to the server (200) or the fourth embedding vector to the server (200), and since the first embedding vector, the second embedding vector, and the fourth embedding vector are processed information, personal information protection is possible.

[0127] FIG. 12 is a flowchart for explaining a control method of an electronic device according to an embodiment of the present disclosure.

[0128] First, the prompt is input into the first encoder obtained by additionally learning the first original encoder and the second original encoder included in the generative neural network model to obtain a first embedding vector in which the prompt is projected into a multidimensional space (S1210). Then, the input image is input into the second encoder obtained by additionally learning the first original encoder and the second original encoder to obtain a second embedding vector in which the input image is projected into a multidimensional space (S1220). Then, the final image generated using the decoder included in the generative neural network model based on the first embedding vector and the second embedding vector is obtained (S1230). Here, the first original encoder and the second original encoder are trained based on the first sample prompt of the first layer and the first sample input image corresponding to the first sample prompt, and the first encoder and the second encoder are obtained by further training the first original encoder and the second original encoder based on the second sample prompt of the second layer, which is a lower layer of the first layer, and the second sample input image corresponding to the second sample prompt.

[0129] Additionally, the additional learning may be a method of fixing one of the first original encoder and the second original encoder and additionally learning the remaining one of the first original encoder and the second original encoder.

[0130] And, one of the first encoder and the second encoder may be identical to one of the first original encoder and the second original encoder, and the remaining of the first encoder and the second encoder may be additionally trained encoders.

[0131] Additionally, the decoder can be trained to generate an image based on the output of at least one of the first source encoder and the second source encoder.

[0132] And, the step of receiving a user command corresponding to a part of an input image and the step of updating a second embedding vector based on the user command are further included, and the step of obtaining a final image (S1230) can obtain a final image based on the first embedding vector and the updated second embedding vector.

[0133] Additionally, the updating step may input a user command to the first encoder to obtain a third embedding vector in which the user command is projected into a multidimensional space, and update the second embedding vector based on the third embedding vector.

[0134] And, the step of obtaining the final image (S1230) obtains a fourth embedding vector by performing a geometric mean on the first embedding vector and the second embedding vector for each corresponding element, and the fourth embedding vector is input to the decoder to obtain the final image generated.

[0135] Additionally, the step of obtaining the final image (S1230) may transmit the first embedding vector and the second embedding vector to the server and receive the final image from the server.

[0136] And, the decoder may have a capacity greater than the sum of the capacity of the first source encoder and the capacity of the second source encoder or the sum of the capacity of the first encoder and the capacity of the second encoder.

[0137] According to various embodiments of the present disclosure as described above, an electronic device can provide a generative service with improved specificity performance while reducing learning costs by additionally learning a first original encoder and a second original encoder.

[0138] Meanwhile, according to a temporary example of the present disclosure, the various embodiments described above can be implemented as software including instructions stored in a machine-readable storage medium that can be read by a machine (e.g., a computer). The device is a device that can call instructions stored from the storage medium and operate according to the called instructions, and may include an electronic device (e.g., electronic device (A)) according to the disclosed embodiments. When an instruction is executed by a processor, the processor can perform a function corresponding to the instruction directly or by using other components under the control of the processor. The instruction may include code generated or executed by a compiler or interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' means that the storage medium does not contain a signal and is tangible, but does not distinguish between data being stored semi-permanently or temporarily in the storage medium.

[0139] Furthermore, according to one embodiment of the present disclosure, the method according to the various embodiments described above may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or online through an application store (e.g., Play Store™). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0140] Furthermore, according to one embodiment of the present disclosure, the various embodiments described above may be implemented in a computer-readable recording medium or a similar device using software, hardware, or a combination thereof. In some cases, the embodiments described herein may be implemented by the processor itself. In a software implementation, embodiments such as the procedures and functions described herein may be implemented as separate software. Each software may perform one or more functions and operations described herein.

[0141] Meanwhile, computer instructions for performing processing operations of a device according to the various embodiments described above may be stored in a non-transitory computer-readable medium. The computer instructions stored in such a non-transitory computer-readable medium, when executed by a processor of a specific device, cause the specific device to perform processing operations in the device according to the various embodiments described above. A non-transitory computer-readable medium refers to a medium that permanently stores data and can be read by a device, rather than a medium that stores data for a short period of time, such as a register, cache, or memory. Specific examples of non-transitory computer-readable media may include a CD, DVD, hard disk, Blu-ray disk, USB, memory card, or ROM.

[0142] In addition, each of the components (e.g., modules or programs) according to the various embodiments described above may be composed of a single or multiple entities, and some of the corresponding sub-components described above may be omitted, or other sub-components may be further included in various embodiments. Alternatively or additionally, some components (e.g., modules or programs) may be integrated into a single entity, which may perform the same or similar functions as those performed by each of the corresponding components prior to integration. Operations performed by modules, programs or other components according to various embodiments may be executed sequentially, in parallel, iteratively or heuristically, or at least some operations may be executed in a different order, omitted, or other operations may be added.

[0143] Although the preferred embodiments of the present disclosure have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above, and various modifications may be made by a person having ordinary skill in the art to which the present disclosure pertains without departing from the gist of the present disclosure as claimed in the claims, and such modifications should not be understood individually from the technical idea or prospect of the present disclosure.

Claims

1. In electronic devices, A memory storing a first encoder and a second encoder obtained by additionally learning the first original encoder and the second original encoder included in the generative neural network model; and one or more processors connected to the memory and controlling the electronic device; The above processor, Inputting a prompt into the first encoder to obtain a first embedding vector in which the prompt is projected into a multidimensional space, Inputting an input image into the second encoder to obtain a second embedding vector in which the input image is projected onto the multidimensional space, Obtaining a final image generated using a decoder included in the generative neural network model based on the first embedding vector and the second embedding vector; The first original encoder and the second original encoder, It is learned based on the first sample prompt of the first layer and the first sample input image corresponding to the first sample prompt, The above first encoder and the above second encoder, An electronic device obtained by further learning the first original encoder and the second original encoder based on a second sample prompt of a second layer which is a lower layer of the first layer and a second sample input image corresponding to the second sample prompt.

2. In paragraph 1, The above additional learning is, An electronic device, comprising a method of fixing one of the first original encoder and the second original encoder and further training the remaining one of the first original encoder and the second original encoder.

3. In paragraph 2, One of the first encoder and the second encoder, Same as one of the first original encoder and the second original encoder, The remainder of the first encoder and the second encoder, An electronic device, wherein the remaining of the first original encoder and the second original encoder are additionally learned encoders.

4. In paragraph 1, The above decoder, An electronic device trained to generate an image based on the output of at least one of the first original encoder and the second original encoder.

5. In paragraph 1, It further includes a user interface; The above processor, Receive a user command corresponding to a portion of the input image through the user interface, Update the second embedding vector based on the user command, An electronic device that obtains the final image based on the first embedding vector and the updated second embedding vector.

6. In paragraph 5, The above processor, Inputting the user command into the first encoder to obtain a third embedding vector in which the user command is projected into the multidimensional space, An electronic device that updates the second embedding vector based on the third embedding vector.

7. In paragraph 1, The above processor, The first embedding vector and the second embedding vector are geometrically averaged for each corresponding element to obtain a fourth embedding vector, An electronic device that obtains the final image generated by inputting the fourth embedding vector into the decoder.

8. In paragraph 1, further comprising a communication interface; The above processor, Controlling the communication interface to transmit the first embedding vector and the second embedding vector to the server, An electronic device that receives the final image from the server via the communication interface.

9. In paragraph 1, The above decoder, An electronic device having a capacity greater than the sum of the capacity of the first original encoder and the capacity of the second original encoder or the sum of the capacity of the first encoder and the capacity of the second encoder.

10. In a method for controlling an electronic device, A step of inputting a prompt into a first encoder obtained by additionally learning a first original encoder and a second original encoder included in a generative neural network model to obtain a first embedding vector in which the prompt is projected into a multidimensional space; A step of inputting an input image to a second encoder obtained by additionally learning the first original encoder and the second original encoder to obtain a second embedding vector in which the input image is projected onto the multidimensional space; and A step of obtaining a final image generated using a decoder included in the generative neural network model based on the first embedding vector and the second embedding vector; The first original encoder and the second original encoder, It is learned based on the first sample prompt of the first layer and the first sample input image corresponding to the first sample prompt, The above first encoder and the above second encoder, A control method obtained by further learning the first original encoder and the second original encoder based on a second sample prompt of a second layer which is a lower layer of the first layer and a second sample input image corresponding to the second sample prompt.

11. In paragraph 10, The above additional learning is, A control method comprising: fixing one of the first original encoder and the second original encoder, and further training the remaining one of the first original encoder and the second original encoder.

12. In paragraph 11, One of the first encoder and the second encoder, Same as one of the first original encoder and the second original encoder, The remainder of the first encoder and the second encoder, A control method, wherein the remaining of the first original encoder and the second original encoder are additionally learned encoders.

13. In paragraph 10, The above decoder, A control method, wherein the control method is trained to generate an image based on the output of at least one of the first original encoder and the second original encoder.

14. In paragraph 10, A step of receiving a user command corresponding to a portion of the input image; and Further comprising a step of updating the second embedding vector based on the user command; The steps of obtaining the above final image are: A control method for obtaining the final image based on the first embedding vector and the updated second embedding vector.

15. In paragraph 14, The above updating steps are: Inputting the user command into the first encoder to obtain a third embedding vector in which the user command is projected into the multidimensional space, A control method for updating the second embedding vector based on the third embedding vector.

Images