Electronic device and operating method thereof

Abstract

An electronic device includes: a memory storing one or more instructions; and a processor configured to execute the one or more instructions to: receive a selection of a synthesis area position in an input image, obtain a target object image to be located in the synthesis area of ​​the input image by using one or more neural networks, by using at least one object detected in the input image, generate a synthesized image by combining the input image with the target object image by using the one or more neural networks, and control a display of the electronic device to display the synthesized image.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of Korean Patent Application No. 10-2019-0097647 filed on August 9, 2019, in the Korean Intellectual Property Office, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates to an electronic device and an operating method thereof, and more particularly, to an electronic device for combining two different images and an operating method thereof. Background Art

[0004] The user can add the desired object image to the background image to obtain a more satisfactory image by using an image / video editing tool such as Photoshop. For example, the user can combine two different images using various existing technical functions of the image editing tool.

[0005] Figure 1 is a view illustrating a related art method of combining two or more images.

[0006] For example, refer to Figure 1 , a user initially prepares a target image to be combined with an input image (operation 10), crops a desired object image from the target image (operation 20), pastes the object image onto the input image, removes the background of the object by using, for example, a layer mask (operation 30), edits details to achieve naturalness by using a brush or selection tool (operation 40), and thereby generates a composite image (operation 50).

[0007] As described above, the prior art method using image editing tools requires many tools and a lot of time to combine two different images.

[0008] That is, a desired target image to be combined with an input image needs to be obtained by, for example, searching or capturing.

[0009] Furthermore, the background area to be removed from the target image must be manually selected using a brush tool, and a layer mask must be used to hide the background area. When the boundary between the background area to be removed and the object area to be used is unclear, it can take considerable time to completely remove the background area, and users who are unfamiliar with using layer masks may encounter difficulties.

[0010] In addition, when the user is not satisfied with the synthesized image generated using the prior art method, the user needs to start over from the selection of the target image and perform all operations again. Summary of the Invention

[0011] Provided are an electronic device for conveniently providing an image synthesis function and an operating method of the electronic device.

[0012] Provided are an electronic device for generating and providing candidate images suitable for an input image on behalf of a user, and an operating method of the electronic device.

[0013] Provided are an electronic device for changing a style of a candidate image to be suitable for an input image by using features of a synthesis position selected by a user in the input image, and an operating method of the electronic device.

[0014] Provided are an electronic device that updates candidate images suitable for an input image according to user control, and a method for operating the electronic device, the electronic device allowing the user to more conveniently select a desired composite image.

[0015] Additional aspects will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the presented embodiments.

[0016] According to one aspect of the present disclosure, an electronic device includes: a memory storing one or more instructions; and a processor configured to execute the one or more instructions stored in the memory to: receive a selection of a synthesis area position in an input image, obtain a target object image to be located in the synthesis area by using one or more neural networks, and generate a synthetic image by combining the input image with the obtained target object image by using one or more neural networks.

[0017] The processor may be further configured to execute one or more instructions to detect one or more objects in the input image, and obtain, by using the first neural network, an object category corresponding to each of the one or more objects in the input image, and position information of each of the one or more objects. Furthermore, based on the position information of the synthesized region and the position information of the one or more objects, obtain distance information from the synthesized region to each of the one or more objects.

[0018] The processor may be further configured to execute one or more instructions to obtain a class vector of the target object based on the obtained one or more object categories by using a second neural network.

[0019] The processor may be further configured to execute one or more instructions to obtain a class vector for each of the one or more objects by using a second neural network, and obtain a class vector of a target object adjacent to the obtained class vector of the one or more objects based on distance information from the synthesis area to each of the one or more objects by using a word embedding lookup table generated based on the second neural network.

[0020] The processor may be further configured to execute the one or more instructions to obtain the class vector of the target object by assigning a higher weight to the class vector of the object that is closer to the location of the synthesized region.

[0021] The processor may be further configured to execute one or more instructions to generate a target object image corresponding to the obtained class vector of the target object by using a third neural network.

[0022] The processor may be further configured to execute the one or more instructions to extract synthetic region image features corresponding to the image of the synthetic region by using a fourth neural network, and to generate a target object image corresponding to the class vector of the target object by reflecting the extracted synthetic region image features by using a third neural network.

[0023] The processor may be further configured to execute one or more instructions to combine the target object image with the input image by reflecting the extracted synthetic region image features using a fifth neural network.

[0024] The processor may be further configured to execute one or more instructions to display the composite image on the display, generate an updated target object image according to user input for controlling the target object image included in the displayed composite image, and display the updated target object image.

[0025] The processor can also be configured to execute one or more instructions to output a scroll bar to control the update of the target object image, obtain an updated class vector adjacent to the class vector corresponding to the target object based on user input for controlling the output scroll bar, obtain an updated target object image based on the updated class vector, generate a composite image by combining the input image with the updated target object image, and display the composite image.

[0026] According to one aspect of the present disclosure, an operating method of an electronic device includes: receiving a selection of a position of a synthesis area in an input image to be combined with a target object image, obtaining a target object image to be placed in the synthesis area by using one or more objects detected in the input image, and generating a synthesized image by combining the input image with the obtained target object using one or more neural networks.

[0027] According to one aspect of the present disclosure, a computer-readable recording medium has recorded thereon a computer program for executing an operating method of an electronic device, the operating method including: receiving a selection of a position of a synthesis area in an input image to be combined with a target object image, obtaining a target object image to be located in the synthesis area by using one or more objects detected in the input image, and synthesizing an image by combining the input image with the obtained target object image using one or more neural networks. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The above and other aspects, features and advantages of certain embodiments of the present disclosure will become more apparent from the following description taken in conjunction with the accompanying drawings, in which:

[0029] Figure 1 is a diagram illustrating a prior art method of combining two or more images;

[0030] Figure 2 An example of an electronic device according to an embodiment is shown;

[0031] Figure 3 is a block diagram of an electronic device according to an embodiment;

[0032] Figure 4 is a detailed block diagram of an electronic device according to an embodiment;

[0033] Figure 5 is a flowchart of an example of a method of operating an electronic device according to an embodiment;

[0034] Figure 6 A method for executing the Figure 5 Example configurations of electronic devices for operation shown;

[0035] Figure 7 is a flowchart of an example of a method of operating an electronic device according to an embodiment;

[0036] Figure 8 A method for executing the Figure 7 Example configurations of electronic devices for operation shown;

[0037] Figure 9 is a flowchart of an example of a method of operating an electronic device according to an embodiment;

[0038] Figure 10 A method for executing the Figure 9 Example configurations of electronic devices for operation shown;

[0039] Figure 11 is a block diagram of an example of an object detection module according to an embodiment;

[0040] Figure 12 shows a network architecture of a region-based convolutional neural network (R-CNN) according to an object detection method using a combination of region proposals and CNN according to an embodiment;

[0041] Figure 13 Shown according to the embodiment Figure 11 Examples of object information and object lists shown;

[0042] Figure 14 shows an example of an embedded module according to an embodiment;

[0043] Figure 15 shows examples of word embeddings according to an embodiment;

[0044] Figure 16 An example of a continuous bag of words (CBOW) method of Word2Vec according to an embodiment is shown;

[0045] Figure 17 is a diagram illustrating a word embedding lookup table according to an embodiment;

[0046] Figure 18 An example of a method for obtaining a target object class vector by referring to a word embedding lookup table by a target object class vector extraction module according to an embodiment is shown;

[0047] Figure 19 An example of a method for obtaining a target object class vector by referring to a word embedding lookup table by a target object class vector extraction module according to an embodiment is shown;

[0048] Figure 20 shows an example of a feature extraction module according to an embodiment;

[0049] Figure 21 shows the network architecture of a CNN according to an embodiment;

[0050] Figure 22 shows an example of an image generation module according to an embodiment;

[0051] Figure 23 shows the architecture of a generative adversarial network (GAN) according to an embodiment;

[0052] Figure 24A is a diagram illustrating a method of training an image generation model according to an embodiment;

[0053] Figure 24B is a diagram illustrating a method of training an image generation model according to an embodiment;

[0054] Figure 25 shows an example of an image generation module further using a style vector according to an embodiment;

[0055] Figure 26 is a view illustrating an operation of an image generation module using a style vector according to an embodiment;

[0056] Figure 27 is a diagram showing an example of an image synthesis module according to an embodiment;

[0057] Figure 28A shows an example of an image segmentation module according to an embodiment;

[0058] Figure 28B shows an example of an image segmentation module according to an embodiment;

[0059] Figure 29 shows an example of a semantic segmentation model according to an embodiment;

[0060] Figure 30 is a diagram illustrating an object control module according to an embodiment;

[0061] Figure 31 is a view showing an example of displaying a scroll bar to control a target object according to an embodiment;

[0062] Figure 32 is a diagram illustrating an operation performed by an embedding module to update a target object class vector according to an embodiment;

[0063] Figure 33 is a block diagram for describing the configuration of a processor with respect to training and processing of a neural network, respectively, according to an embodiment;

[0064] Figure 34A shows an example of a data learner installed in a server and a data processor installed in a user's electronic device according to an embodiment; and

[0065] Figure 34B An example of a data learner and a data processor installed in a server according to an embodiment is shown. DETAILED DESCRIPTION

[0066] The terms used herein will now be briefly described.

[0067] Although many of the terms used herein have been selected from currently widely used general terms while taking into account the functions obtained according to the embodiments, these terms may be replaced by other terms based on the intention of one of ordinary skill in the art, custom, the emergence of new technologies, etc. In certain cases, terms arbitrarily selected by the applicant may be used, and in such cases, the meanings of these terms may be described herein. Therefore, it should be noted that the terms used herein are to be interpreted based on their actual meanings and the contents of the embodiments described herein, rather than being interpreted based solely on the names of the terms.

[0068] It will be understood that when used herein, the terms "comprise," "comprising," "include," and / or "includes" specify the presence of the elements described, but do not exclude the presence or addition of one or more elements, unless otherwise specified. As used herein, the term "unit" or "module" refers to an entity for performing at least one function or operation, and can be implemented as hardware, software, or a combination of hardware and software.

[0069] Here, the expression "at least one of a, b or c" means only a; b; c; a and b; a and c; b and c; all of a, b and c or variations thereof.

[0070] Hereinafter, certain embodiments will be described with reference to the accompanying drawings. However, the embodiments may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. In the drawings, for the sake of clarity, parts not relevant to the description are not illustrated, and like reference numerals represent like elements throughout.

[0071] As used herein, the term "user" refers to a person who controls a function or operation of the image display apparatus by using a controller, and may include an observer, a manager, or an installation engineer.

[0072] Reference Figure 2 In an input image 200 to be combined with a target object image, electronic device 100 receives input image 200 for image synthesis and selection of a synthesis area, for example, via touch input from user 202 using a finger, stylus, or the like. Electronic device 100 combines two different images. For example, of the two different images, the image serving as a background is referred to as the input image, and the image including the object to be combined with the input image is referred to as the target object image. However, the two different images may also be referred to as a first image and a second image.

[0073] According to an embodiment, by using one or more neural networks, the electronic device 100 can generate a target object image to be combined with the input image 200 based on the input image 200 and the selection of the synthesis area to be combined with the target object image, and generate and output the synthesized image 300 by combining the input image 200 with the target object image.

[0074] According to an embodiment, the electronic device 100 may detect one or more objects in the input image 200 by using one or more neural networks. For example, the electronic device 100 may detect one or more objects in the input image 200 by using one or more neural networks. Figure 2 Objects such as a boy, a bird, and a house are detected in the input image 200. Information about the objects detected in the input image 200 may include object categories of the objects and distance information from the synthesized region to the objects.

[0075] According to an embodiment, by using one or more neural networks, the electronic device 100 may generate a target object to be combined with the synthesis region of the input image 200 based on the object category of one or more objects detected in the input image 200 and the distance from the synthesis region to the object. Figure 2 Objects such as a boy, a bird, and a house are detected in the input image 200 shown, and a dog or a cat is generated as a target object to be combined with the synthesis region.

[0076] According to an embodiment, the electronic device 100 may generate a target object image corresponding to the generated target object by using one or more neural networks. The electronic device 100 may generate a target object image to be naturally combined with the input image 200 by reflecting the characteristics of the synthesized region of the input image 200 to generate the target object image. For example, when generating a target object image including a dog or a cat and when the characteristics of the synthesized region indicate an image of grass, the electronic device 100 may change the style of the target object image including the dog or cat to a texture or color suitable for grass.

[0077] According to an embodiment, the electronic device 100 may generate a synthesized image 300 by combining the generated target object image with the input image 200 using one or more neural networks. The electronic device 100 may naturally combine the input image 200 with the target object image to generate the synthesized image 300 by reflecting the characteristics of the synthesized region of the input image 200. For example, when the input image 200 includes an image of a dog or a cat when combined with the target object, and when the characteristics of the synthesized region indicate a grass image, the electronic device 100 may naturally generate an edge of the target object image including the dog or cat by using the grass characteristics.

[0078] According to an embodiment, the electronic device 100 may update the target object image according to a user input for controlling the target object image in the composite image 300, and may generate and output an updated composite image 400 by combining the updated target object image with the input image 200. For example, when the electronic device 100 outputs the composite image 300 including <Bulldog> as the target object, and when the user desires a dog of a type other than a Bulldog as the target object, the electronic device 100 may generate and output an updated composite image 400 including <Cocker Spaniel> as the target object.

[0079] As described above, the electronic device 100 may use one or more neural networks suitable for each of one or more operations performed to combine two or more images. A neural network is a statistical learning algorithm for implementing machine learning by simulating a human brain.

[0080] Figure 3 is a block diagram of an electronic device 100 according to an embodiment.

[0081] Reference Figure 3 , the electronic device 100 may include a memory 110 and one or more processors 120. However, the electronic device 100 may include a greater number of elements than shown, and is not limited to the above examples.

[0082] The memory 110 according to an embodiment may store programs for processing and control operations of the processor 120 , and store data input or output to or from the electronic device 100 .

[0083] The memory 110 may include at least one type of storage medium selected from the group consisting of a flash memory, a hard disk, a micro multimedia card, a memory card (e.g., a secure digital (SD) card or an extreme digital (XD) card), a random access memory (RAM), a static RAM (SRAM), a read-only memory (ROM), an electrically erasable programmable ROM (EEPROM), a programmable ROM (PROM), a magnetic memory, a magnetic disk, or an optical disk.

[0084] The processor 120 controls the overall operation of the electronic device 100. For example, the processor 120 may execute one or more instructions stored in the memory 110 to perform the functions of the electronic device 100.

[0085] The processor 120 according to an embodiment may combine two or more different images and output a synthesized image by using a learning model using one or more neural networks.

[0086] In one embodiment, processor 120 may execute one or more instructions stored in memory 110 to control various operations to be performed.

[0087] In one embodiment, the processor 120 may include an internal memory that stores one or more instructions and executes the one or more instructions stored in the internal memory to control various operations to be performed. That is, the processor 120 may execute at least one instruction or program stored in the memory 110 or the internal memory included in the processor 120 to perform a specific operation.

[0088] Despite Figure 3 1 and 2. One processor 120 is shown in FIG. 1 , but a plurality of processors may be included. In this case, at least one of the plurality of processors may be used to perform each operation performed by the electronic device 100 according to the embodiment.

[0089] According to one embodiment, the electronic device 100 may further include a neural network processor. The neural network processor may control specific operations to be performed by performing calculations via a neural network. Specifically, in one embodiment, the neural network processor may execute one or more instructions to perform calculations via a neural network.

[0090] According to an embodiment, the processor 120 may execute one or more instructions stored in the memory 110 to receive a selection of a position of a synthesis area in an input image to be combined with a target object image, obtain a target object image to be positioned in the synthesis area by using one or more objects detected in the input image, and generate a synthesized image by combining the input image with the obtained target object image by using one or more neural networks.

[0091] According to one embodiment, the processor 120 may execute one or more instructions stored in the memory 110 to detect one or more objects in the input image using a first neural network, obtain an object category corresponding to each of the one or more objects and position information of each of the one or more objects in the input image, and obtain distance information from the synthesis region to each of the one or more objects based on the position information of the synthesis region and the position information of each of the one or more objects. By obtaining the distance information from the synthesis region to each of the one or more objects, a target object suitable for the synthesis region of the input image can be obtained based on the relative position between the synthesis region of the input image and the object detected in the input image.

[0092] According to an embodiment, the processor 120 may execute one or more instructions to obtain a class vector of the target object based on the obtained one or more object categories by using the second neural network.

[0093] According to an embodiment, the processor 120 may execute one or more instructions to obtain a class vector for each of the one or more objects and obtain a class vector of a target object adjacent to the target object by using a second neural network. By using a word embedding lookup table generated based on the second neural network, based on the distance information from the synthesis area to each of the one or more objects, a class vector of the target object adjacent to the obtained class vector of the one or more objects is obtained. When the class vector of the target object adjacent to the obtained class vector of the one or more objects is obtained based on the distance information from the synthesis area to each of the one or more objects, the user does not need to manually search for the target and can use artificial intelligence (AI) to automatically obtain the object to be located in the synthesis area and the appropriate target object.

[0094] According to an embodiment, the processor 120 may execute one or more instructions to obtain a class vector of the target object by assigning a higher weight to a class vector of an object located closer to the synthesis area. By assigning a higher weight to a class vector of an object located closer to the synthesis area, a larger number of target objects that are more suitable for objects located closer to the synthesis area may be extracted.

[0095] According to an embodiment, the processor 120 may execute one or more instructions to generate a target object image corresponding to the obtained class vector of the target object by using a third neural network.

[0096] According to an embodiment, the processor 120 may execute one or more instructions to extract synthetic region image features corresponding to the synthetic region image using the fourth neural network, and generate a target object image corresponding to the class vector of the target object by reflecting the extracted synthetic region image features using the third neural network. By reflecting the synthetic region image features to generate the target object image, a target object image having a style more natural to the synthetic region can be obtained.

[0097] According to one embodiment, the processor 120 may execute one or more instructions to combine the target object image with the input image by reflecting the extracted synthetic region image features using the fifth neural network. By reflecting the synthetic region image features to combine the input image with the target object image, the edges of the target object image to be combined with the input image may be extracted so as to appear more natural to the user.

[0098] According to an embodiment, the processor 120 may execute one or more instructions to display a composite image on a display, generate an updated target object image based on a user input for controlling the target object image included in the displayed composite image, and display the updated target object image. By providing a composite image including an updated target object image based on the user input for controlling the target object image, when a user does not like the target object included in the initially generated composite image, a composite image including another target object image can be easily provided based on a simple user input.

[0099] According to an embodiment, the processor 120 may execute one or more instructions to output a scroll bar for controlling the updating of the target object image, obtain an updated class vector adjacent to the class vector corresponding to the target object based on a user input for controlling the output scroll bar, obtain an updated target object image based on the updated class vector, generate a composite image by combining the input image with the updated target object image, and display the composite image.

[0100] The processor 120 and the memory 110 are used to perform functions associated with the combination of two or more images using AI according to the present disclosure. The processor 120 may include one or more processors. In this case, the one or more processors may include: processors such as a central processing unit (CPU), an application processor (AP), and a digital signal processor (DSP); dedicated graphics processors such as a graphics processing unit (GPU) and a visual processing unit (VPU); or dedicated AI processors such as a neural processing unit (NPU). The one or more processors control the input data to be processed according to predefined operating rules or AI models stored in the memory. Alternatively, when the one or more processors are dedicated AI processors, the dedicated AI processor can be designed with a hardware structure dedicated to processing a specific AI model.

[0101] Establishing predefined operating rules or AI models through training. In this article, making through training means training a basic AI model based on a learning algorithm using multiple training data to make a predetermined operating rule or AI model configured to achieve the desired characteristics (or purpose). Training can be performed directly by a device having AI functions according to the present disclosure or by a separate server and / or system. The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited thereto.

[0102] The AI ​​model may include multiple neural network layers. Each of the multiple neural network layers has multiple weight values, and the neural network calculation is performed by calculating between the calculation result of the previous layer and the multiple weight values. The multiple weight values ​​of the multiple neural network layers can be optimized based on the results of training the AI ​​model. For example, the multiple weight values ​​can be updated to reduce or minimize the loss value or cost value obtained by the AI ​​model during the training process. The artificial neural network may include, for example, a CNN, a deep neural network (DNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recursive deep neural network network (BRDNN) or a deep Q network, but is not limited thereto.

[0103] According to an embodiment, the electronic device 100 can generate a composite image by combining an input image with one or more target object images using one or more neural networks, and display the composite image by using a video / audio signal output port or wireless communication, and transmit the generated composite image to an externally connected display device. For example, the electronic device 100 may include a device mainly for processing data and transmitting the processed data to an external display device (e.g., a set-top box).

[0104] Figure 4 is a detailed block diagram of the electronic device 100 according to an embodiment.

[0105] exist Figure 3 and Figure 4 In the present invention, similar reference numerals denote similar elements. Therefore, the above description of Figure 3 The description is provided to describe the electronic device 100 .

[0106] Reference Figure 4 In addition to the memory 110 and the processor 120, the electronic device 100 may also include a display 130, one or more antennas 155, a communicator 150, a detector 160, an input / output device 180, a video processor 135, an audio processor 145, an audio output device 140 and a user input device 190.

[0107] Above about Figure 3 A description of the memory 110 and the processor 120 is provided. Figure 4 The general will not repeat it.

[0108] The display 130 may display an image on a screen under the control of the processor 120. An image to be displayed on a screen may be received from the communicator 150, the inputter / outputter 180, or the memory 110.

[0109] According to an embodiment, the display 130 may display the composite image generated by the processor 120. The display 130 may also display the composite image including the target object image updated according to a user input for controlling the target object image included in the composite image.

[0110] The antenna 155 is used to receive signals from other devices or transmit signals to other devices. Although one antenna 155 is shown, multiple antennas may be included. Therefore, the electronic device 100 according to the present disclosure can support a multiple-input multiple-output (MIMO) system.

[0111] The communicator 150 may include one or more modules that enable wireless communication between the electronic device 100 and a wireless communication system or between the electronic device 100 and a network including another electronic device. For example, the communicator 150 may include a broadcast receiver module 151, a mobile communication module 152, a wireless Internet module 153, and a short-range wireless communication module 154. The communicator 150 may also be referred to as a transmitter / receiver.

[0112] The broadcast receiver module 151 receives broadcast signals and / or broadcast information from an external broadcast management server through a broadcast channel. The broadcast signals may include TV broadcast signals, radio broadcast signals, and data broadcast signals, and / or may include a combination of TV broadcast signals or radio broadcast signals and data broadcast signals.

[0113] The mobile communication module 152 transmits and receives wireless signals to and from at least one of a base station in a mobile communication network, an external device, or a server. The wireless signals may include voice call signals, video call signals, or various types of data associated with sending and receiving text / multimedia messages.

[0114] The wireless Internet module 153 refers to a module for wireless Internet access and can be provided as an embedded or external module. As a wireless Internet technology, for example, a wireless local area network (WLAN) (e.g., Wi-Fi), wireless broadband (Wibro), world interoperability for microwave access (WiMAX) or high-speed downlink packet access (HSDPA) can be used. Through the wireless Internet module 153, the electronic device 100 can establish a Wi-Fi peer-to-peer (P2P) connection to another device. Due to the Wi-Fi P2P connection, streaming services, data transmission / reception services or printing services based on a connection to a printer can be provided between devices.

[0115] The short-range wireless communication module 154 is a module for short-range wireless communication. As a short-range wireless communication technology, for example, Bluetooth, Radio Frequency IDentification (RFID), Infrared Data Association (IrDA), Ultra Wideband (UWB), or ZigBee can be used.

[0116] The communicator 150 according to an embodiment may receive a learning model from an external server using one or more neural networks.

[0117] The detector 160 detects a user's voice, a user's image, or a user's interaction, and may include a microphone 161 , a camera 162 , and a light receiver 163 .

[0118] The microphone 161 receives a voice uttered by a user and converts the received voice into an electric signal and outputs the electric signal to the processor 120 .

[0119] The camera 162 may receive images (eg, consecutive frames) corresponding to the user's motion (including gestures) within the camera recognition range.

[0120] The optical receiver 163 receives optical signals (including control signals) from the remote controller. The optical receiver 163 can receive optical signals corresponding to user input (e.g., touch, press, touch gesture, voice, or action) from the remote controller. The control signal can be extracted from the received optical signal under the control of the processor 120.

[0121] In an embodiment, a selection of a location in an input image for a composite region to be combined with a target object image may be received via one or more of microphone 161, camera 162, and optical receiver 163. When a user wishes to control the updating of a target object in an output composite image, a target object control command may be received via one or more of microphone 161, camera 162, and optical receiver 163. For example, the location selection or target object control command may include one or more of a user's speech detected by microphone 161, a user's gesture detected by camera 162, or a control signal received from a remote control by optical receiver 163. However, this is not limiting, and the user selection input may be received via the touch screen of display 130.

[0122] Under the control of the processor 120, the input / output 180 receives, for example, video data (e.g., moving images), audio data (e.g., voice or music), and additional information (e.g., electronic program guide (EPG)) from the outside of the electronic device 100. The input / output 180 may include one of a high-definition multimedia interface (HDMI) port 181, a component jack 182, a computer port 183, and a universal serial bus (USB) 184. The input / output 180 may include a combination of the HDMI port 181, the component jack 182, the computer port 183, and the USB port 184.

[0123] The memory 110 according to an embodiment may store programs for processing and control operations of the processor 120 and store data input to or output from the electronic device 100. The memory 110 may also store data required for the operation of the electronic device 100.

[0124] The programs stored in the memory 110 may be classified into a plurality of modules according to their functions. Specifically, the memory 110 may store one or more programs for performing specific operations by using a neural network. For example, the one or more programs stored in the memory 110 may include an object detection module 610, an embedding module 620, an image generation module 630, an image synthesis module 640, a feature extraction module 650, and an object control module 660.

[0125] Object detection module 610 may include one or more instructions for detecting one or more objects in an input image using one or more neural networks.

[0126] The embedding module 620 may generate a target object to be combined with the synthesized region of the input image based on one or more objects detected in the input image and the positions of the one or more objects using one or more neural networks.

[0127] The image generation module 630 can generate a target object image corresponding to the generated target object by using one or more neural networks. The image generation module 630 can generate a target object image to be naturally combined with the input image by reflecting the characteristics of the synthesis area of ​​the input image to generate the target object image.

[0128] The image synthesis module 640 can generate a synthesized image by combining the generated target object image with the input image using one or more neural networks. The image synthesis module 640 can naturally combine the input image with the target object image by generating a synthesized image reflecting features of a synthesis region of the input image.

[0129] The feature extraction module 650 may extract features from the synthesized region in the input image to be combined with the target object image by using one or more neural networks.

[0130] The object control module 660 may update the target object image according to a user input for controlling the target object image in the synthesized image, and generate and output a synthesized image by combining the updated target object image with the input image.

[0131] The processor 120 is used to control the overall operation of the electronic device 100 and the signal flow between the internal elements of the electronic device 100 and process data. When a user input is made or preset and pre-stored conditions are met, the processor 120 can execute an operating system (OS) and various applications stored in the memory 110.

[0132] The processor 120 may include an internal memory. In this case, at least one of the data, programs, or instructions stored in the memory 110 may be stored in the internal memory of the processor 120. For example, to perform an operation using a neural network or one or more instructions for the neural network to perform certain operations, the internal memory of the processor 120 may store one or more programs for performing specific operations.

[0133] According to an embodiment, the processor 120 may execute one or more instructions included in the object detection module 610, the embedding module 620, the image generation module 630, the image synthesis module 640, the feature extraction module 650 and the object control module 660 stored in the memory 110 to implement the image synthesis function of the electronic device 100 described herein.

[0134] The video processor 135 may process image data to be displayed on the display 130 and perform various image processing operations on the image data, such as decoding, rendering, scaling, noise filtering, frame rate conversion, and resolution conversion.

[0135] The audio processor 145 processes audio data. The audio processor 145 may perform various processing operations on the audio data, such as decoding, amplification, and noise filtering.

[0136] Under the control of the processor 120, the audio outputter 140 may output audio data included in a broadcast signal received through the tuner, audio data input through the communicator 150 or the inputter / outputter 180, or audio data stored in the memory 110. The audio outputter 140 may include at least one of a speaker 141, a headphone output terminal 142, or a Sony / Philips Digital Interface (S / PDIF) output terminal 143.

[0137] The user input 190 refers to a device used by a user to input data for controlling the electronic device 100. For example, the user input 190 may include a keypad, a dome switch, a touch pad, a jog wheel, or a micro switch, but is not limited thereto.

[0138] The user inputter 190 according to an embodiment may receive a command for receiving selection of an input image, receiving selection of a synthesis area to be combined with a target object image in an input image, or controlling a target object image in a synthesized image.

[0139] exist Figure 3 and Figure 4 In the block diagram of the electronic device 100 shown, the elements shown can be integrated, added, or omitted. For example, as described in more detail below, two or more elements can be combined into one element, or one element can be divided into two or more elements. The functions performed by the blocks are only for describing the embodiments, and the specific operations or devices are not limiting.

[0140] Figure 5 is a flowchart of an example of an operating method of the electronic device 100 according to an embodiment.

[0141] Reference Figure 5 , in operation S510, the electronic device 100 may receive a selection of a synthesis area to be combined with a target object image in an input image.

[0142] When the electronic device 100 combines two different images, the image used as the background is referred to as an input image, and the image including the target object is referred to as a target object image.

[0143] According to an embodiment, the electronic device 100 may receive an input image and receive a selection of a synthesis area in the input image to be combined with the target object image. For example, the user may input the location of a portion of the input image to be combined with the target object image into the electronic device 100 using various user interaction means such as touch, gesture, and pointer.

[0144] In operation S520, the electronic device 100 may detect one or more objects in the input image by using one or more neural networks.

[0145] The electronic device 100 may detect one or more objects different from the background by extracting features from the input image, classify categories of the detected one or more objects, and obtain position information of the detected objects.

[0146] The neural networks used when the electronic device 100 detects one or more objects in the input image may include, for example, two-stage methods such as fast region-based convolutional neural network (Fast R-CNN), region-based fully convolutional network (R_FCN) and feature pyramid network (FPN)-FRCN; or single-stage methods such as You Only Look Once (YOLO), Single Shot Multibox Detector (SSD) and Retina Net.

[0147] In operation S530, the electronic device 100 may acquire images of one or more target objects to be located in the synthesis area by using one or more neural networks based on the detected one or more objects.

[0148] The electronic device 100 may obtain class vectors corresponding to one or more objects detected in the input image, and obtain class vectors of one or more target objects to be located in the synthesized area taking into account the position of the synthesized area in the input image.

[0149] The neural network used by the electronic device 100 when obtaining the class vector of the target object by using one or more objects may include, for example, latent semantic analysis (LSA), Word2Vec, GloVe, and fastText.

[0150] The electronic device 100 may generate a target object image by using the obtained class vector of the target object.

[0151] The neural network used when the electronic device 100 generates the target object image by using the class vector of the target object may include, for example, GAN.

[0152] In operation S540, the electronic device 100 may obtain a synthesized image by combining the input image with the obtained one or more target object images using one or more neural networks.

[0153] The neural network used by the electronic device 100 when combining the input image with the obtained one or more target object images may include, for example, U-Net.

[0154] Figure 6 A method for executing the Figure 5 An example configuration of the electronic device 100 is shown for operation.

[0155] Reference Figure 6 , the electronic device 100 may include an object detection module 610 , an embedding module 620 , an image generation module 630 , and an image synthesis module 640 .

[0156] The object detection module 610 may comprise suitable logic, circuitry, interfaces, and / or code that may be operable to detect one or more objects in the input image 200 .

[0157] According to an embodiment, the object detection module 610 may receive the input image 200 and position information of a synthesis region in the input image 200 to be combined with the target object image.

[0158] According to an embodiment, the object detection module 610 may detect one or more objects in the input image 200 by using one or more neural networks, and provide an object list including distance information from the synthesized region to the detected objects to the embedding module 620. The object list may include information about the one or more detected objects, and the information about the one or more objects may include an object category of each object and distance information from the synthesized region to each object. For example, when the object detection module 610 detects objects such as a boy, a bird, and a house in the input image 200, the object list may include (boy, 3), (bird, 5), and (house, 20) as (object category, distance information, for example, measured in appropriate units (e.g., pixels, millimeters, centimeters, etc.)).

[0159] The embedding module 620 may comprise suitable logic, circuitry, interfaces and / or code for generating a class vector for one or more target objects to be located in a synthesized region of the input image 200 using one or more objects included in the object list received from the object detection module 610 .

[0160] According to one embodiment, the embedding module 620 may receive an object list from the object detection module 610 , the object list including information about one or more objects.

[0161] According to one embodiment, the embedding module 620 may obtain a class vector corresponding to one or more object categories included in the received object list by using one or more neural networks, and obtain one or more target object class vectors adjacent to the obtained object vector by using distance information of one or more objects included in the object list. For example, the embedding module 620 may output fifteen class vectors corresponding to dogs and three class vectors corresponding to cats as target object class vectors obtained using the object list received from the object detection module 610, such as (boy, 3), (bird, 5), and (house, 20).

[0162] The image generation module 630 may comprise suitable logic, circuitry, interfaces and / or code that may be operable to generate one or more target object images corresponding to the one or more target object class vectors received from the embedding module 620 .

[0163] According to one embodiment, the image generation module 630 may generate target object images corresponding to one or more target object class vectors received from the embedding module 620 by using one or more neural networks. For example, the image generation module 630 may generate fifteen dog images and three cat images by using fifteen class vectors corresponding to dogs and three class vectors corresponding to cats received from the embedding module 620.

[0164] The image synthesis module 640 may comprise suitable logic, circuitry, interfaces and / or code that may be operable to combine the input image 200 with the target object image received from the image generation module 630 .

[0165] According to an embodiment, the image synthesis module 640 may output a synthesized image generated by combining the input image 200 with the target object image received from the image generation module 630, by using one or more neural networks. For example, the image synthesis module 640 may generate the synthesized image 300 by combining the input image 200 with each of fifteen dog images and three cat images received from the image generation module 630.

[0166] Figure 7 is a flowchart of an example of an operating method of the electronic device 100 according to an embodiment.

[0167] Here we will briefly provide the above Figure 5 The description provided is related to describing Figure 7 operation.

[0168] Reference Figure 7 , in operation S710, the electronic device 100 may receive a selection of a synthesis area to be combined with a target object image in an input image.

[0169] In operation S720, the electronic device 100 may detect one or more objects in the input image by using one or more neural networks.

[0170] In operation S730, the electronic device 100 may extract image features from the synthesized region of the input image by using one or more neural networks.

[0171] The neural network used by the electronic device 100 when extracting image features from the synthesized area of ​​the input image may include, for example, CNN.

[0172] In operation S740, the electronic device 100 may obtain one or more target object images to be located in the synthesis area by reflecting features of the synthesis area by using one or more neural networks based on the detected one or more objects.

[0173] The electronic device 100 may obtain class vectors corresponding to one or more objects detected in the input image, and obtain one or more target object class vectors to be located in the synthesized area taking into account the position of the synthesized area in the input image.

[0174] The electronic device 100 may generate a target object image by using the obtained target object class vector, and in this case, may generate a target object image more suitable for the composite area by reflecting the characteristics of the composite area.

[0175] The neural network used when the electronic device 100 generates the target object image by using the target object class vector may include, for example, a conditional GAN.

[0176] In operation S750, the electronic device 100 may obtain a synthesized image by combining the input image with the obtained one or more target object images by using one or more neural networks based on the features of the synthesized area.

[0177] The electronic device 100 may obtain a composite image having a natural boundary between the target object and the input image by reflecting the features of the composite area to combine the input image with one or more target object images.

[0178] Figure 8 A method for executing the Figure 7 An example configuration of the electronic device 100 is shown for operation.

[0179] Reference Figure 8 In addition to the object detection module 610 , the embedding module 620 , the image generation module 630 , and the image synthesis module 640 , the electronic device 100 may further include a feature extraction module 650 .

[0180] Above and Figure 6 The description of the object detection module 610 and the embedding module 620 is provided in conjunction with each other and will not be repeated here.

[0181] The feature extraction module 650 may comprise suitable logic, circuitry, interfaces and / or code that may be operable to receive a composite region of the input image 200 and extract features from the composite region of the input image 200 .

[0182] According to an embodiment, the feature extraction module 650 may extract one or more features from the synthesized region of the input image 200 using one or more neural networks. The one or more features may include image color or texture. The feature extraction module 650 may configure a style vector using the extracted features of the synthesized region and provide the configured style vector to at least one of the image generation module 630 or the image synthesis module 640.

[0183] The image generation module 630 may comprise suitable logic, circuitry, interfaces and / or code that may be operable to reflect a style vector corresponding to features of a synthesized region to generate one or more target object images corresponding to one or more target object class vectors received from the embedding module 620 .

[0184] According to an embodiment, the image generation module 630 may reflect the features of the synthesized region using one or more neural networks to generate a target object image corresponding to one or more target object class vectors received from the embedding module 620. For example, the image generation module 630 may generate fifteen dog images and three cat images by using fifteen class vectors corresponding to dogs and three class vectors corresponding to cats received from the embedding module 620. In this case, the dog images and cat images may be generated by reflecting the image features of the synthesized region (e.g., image color or texture information).

[0185] The image synthesis module 640 may comprise suitable logic, circuitry, interfaces and / or code that may be operable to reflect the style vector corresponding to the features of the synthesis region to combine the input image 200 with one or more target object images received from the image generation module 630 .

[0186] According to an embodiment, the image synthesis module 640 may combine the input image 200 with one or more target object images received from the image generation module 630 by reflecting image features of the synthesis region using one or more neural networks. For example, the image synthesis module 640 may naturally process the edges of a dog or a cat by considering image features of the synthesis region (e.g., color or texture of the synthesis region) to combine the input image 200 with each of the fifteen dog images and three cat images received from the image generation module 630.

[0187] Figure 9 is a flowchart of an example of an operating method of the electronic device 100 according to an embodiment.

[0188] Reference Figure 9 , in operation S910, the electronic device 100 may receive a user input for controlling a target object included in a composite image.

[0189] In operation S920, the electronic device 100 may update the target object according to the user input and obtain an image of the updated target object.

[0190] The electronic device 100 may obtain a class vector adjacent to the class vector of the target object currently included in the synthesized image to update the target object and generate a target object image corresponding to the updated class vector of the target object. For example, if a poodle class vector is obtained as a class vector adjacent to the <bulldog> class vector, the electronic device 100 may obtain an image corresponding to the poodle.

[0191] In operation S930, the electronic device 100 may combine the input image with the updated target object image using one or more neural networks to obtain a composite image. For example, the electronic device 100 may output a composite image generated by combining the input image with a poodle image corresponding to the updated target object image.

[0192] Figure 10 A method for executing the Figure 9 An example configuration of the electronic device 100 is shown for operation.

[0193] Reference Figure 10 In addition to the object detection module 610 , the embedding module 620 , the image generation module 630 , the image synthesis module 640 and the feature extraction module 650 , the electronic device 100 may further include an object control module 660 .

[0194] The object detection module 610, the embedding module 620, the image generation module 630, the image synthesis module 640 and the feature extraction module 650 are substantially the same as those described above, and redundant descriptions will be omitted.

[0195] The object control module 660 may comprise suitable logic, circuitry, interfaces and / or code that may be operable to receive user input indicating an update of a target object in the composite image (i.e., a target object control command) and control the embedding module 620 and the image generation module 630 in accordance with the target object control command.

[0196] Depending on the embodiment, the object control module 660 may instruct the embedding module 620 to generate a class vector for a target object adjacent to the class vector of the target object currently included in the composite image. For example, when the electronic device 100 outputs a composite image generated by combining an input image with an image of a "pit bull" as a target object, the user may wish to use a different type of dog as the target object. In this case, the object control module 660 may receive a control command from the user to change or update the target object in the composite image.

[0197] The embedding module 620 may generate a target object class vector adjacent to the class vector of the target object currently included in the synthesized image, and transmit the generated target object class vector to the image generation module 630 under the control of the object control module 660. For example, the embedding module 620 may obtain a poodle class vector as a class vector adjacent to the <bulldog> class vector, and transmit the poodle class vector to the image generation module 630.

[0198] The image generation module 630 may generate an updated target object image corresponding to the updated target object class vector received from the embedding module 620. For example, the image generation module 630 may generate a <bulldog> image corresponding to the <bulldog> class vector as the updated target object image.

[0199] The image synthesis module 640 may generate a synthesized image by combining the input image with the updated target object image received from the image generation module 630. For example, the image synthesis module 640 may output a synthesized image generated by combining the input image with a <bulldog> image as the updated target object image.

[0200] Now refer to Figures 11 to 32 Describe each module of the electronic device in detail.

[0201] Object Detection Module

[0202] Figure 11 is a block diagram of an example of the object detection module 610 according to an embodiment.

[0203] Reference Figure 11 , the object detection module 610 includes an object detection model 611 and a distance calculation module 612 .

[0204] The object detection model 611 may detect one or more objects in the input image 200 by using one or more neural networks, and output object information 1100 including object categories and object locations corresponding to the detected one or more objects.

[0205] Object detection includes object localization, which determines the location of objects in a given image, and object classification, which determines the category to which one or more objects belong. Therefore, the object detection model 611 may include three stages: information region selection, which selects information regions; feature extraction, which extracts features from each information region; and classification, which classifies each information region by applying a classifier to the extracted features. Depending on the detection method, localization performance can be improved through post-processing such as bounding box regression.

[0206] Now refer to Figure 12 Describes an example of an object detection model.

[0207] Figure 12 A network architecture of R-CNN according to an object detection method using a combination of region proposal and CNN according to an embodiment is shown.

[0208] Reference Figure 12, the object detection model 611 may include a region proposal module 1210 , a CNN 1220 , a classifier module 1230 , and a bounding box regression module 1240 .

[0209] The region proposal module 1210 extracts information regions from the input image 200. A certain number of information regions, for example, 2,000 information regions, may be extracted. R-CNN uses selection-search as one of the region proposal algorithms.

[0210] CNN 1220 extracts fixed-length feature vectors from the regions generated by region proposal module 1210. CNN 1220 (e.g., AlexNet or VGGNet) receives input of a specific size, so each rectangular image region given by the region proposal algorithm needs to be warped to a specific size, regardless of its size and aspect ratio. CNN 1220 receives the warped region and extracts the results of the previous layers of the classifier module 1230.

[0211] The classifier module 1230 receives a fixed-length feature vector as input and performs classification.

[0212] The bounding box regression module 1240 receives a fixed-length feature vector as input and calculates the four coordinates (x, y, w, h) of the representative box. The position of the object can be specified by the four coordinates (x, y, w, h) of the representative box.

[0213] That is, R-CNN performs object detection by performing object localization through region proposal extraction and performing object classification using the extracted features. Bounding box regression can be performed to reduce localization errors.

[0214] To train the object detection model 611, the classification layer (e.g., output layer) of the pre-trained CNN is newly changed to the "number of object categories + background" for object detection, and weight initialization is performed only on the corresponding part so as to appropriately modify the pre-trained CNN for target detection.

[0215] In order to train a linear support vector machine (SVM) for each category, when positive samples (regions where the final result value of the object of each category is output from the region proposal algorithm is equal to or greater than a specific threshold) and negative samples (regions where the final result value of the object of each category is output from the region proposal algorithm is less than a specific threshold) are selected for each category, and positive samples and negative samples are configured for the linear SVM of each category to be trained, a fixed-length feature vector can be extracted using a fine-tuned CNN, and the fixed-length feature vector can be used as input to train the linear SVM of each category.

[0216] For example, the object detection model 611 may detect one or more objects in the input image 200. The object information 1100 includes information about one or more objects, and may represent information about each object in terms of (object class, location).

[0217] Figure 12 Only an example for implementing the object detection module is shown, and the object detection module may be implemented in a simpler configuration.

[0218] Figure 13 Shown according to the embodiment Figure 11 Examples of object information 1100 and object list 1110 are shown.

[0219] Reference Figure 13 , the object detection module 610 can detect, for example, three objects in the input image 200 and output (boy, boy's position), (bird, bird's position) and (house, house's position) as object information 1100.

[0220] Return to reference Figure 11 , the distance calculation module 612 can receive object information 1100 (i.e., (object category, object position)) from the object detection model 611, calculate the distance from each object to the synthetic area (e.g., center point 1250) by using the position information of the synthetic area in the input image 200 to be combined with the target object image, and output (object category, distance from each object to the synthetic area) as an object list 1110.

[0221] Reference Figure 13 For example, the distance calculation module 612 may calculate the distance from the boy to the composite area as 3, the distance from the bird to the composite area as 5, and the distance from the house to the composite area as 20. Therefore, the distance calculation module 612 may output (boy, 3), (bird, 5), and (house, 20) as the object list 1110.

[0222] Embedded Module

[0223] Figure 14 An example of an embedding module 620 according to an embodiment is shown.

[0224] The embedding module 620 receives the object list 1110 from the object detection module 610 and obtains one or more target object class vectors 1400 based on one or more object categories included in the object list 1110 .

[0225] Reference Figure 14 , the embedding module 620 may include a word embedding model 621 and a target object class vector extraction module 622 .

[0226] The word embedding model 621 receives one or more object categories included in the object list 1110 received from the object detection module 610 and maps the object categories to object class vectors by using one or more neural networks.

[0227] A word embedding model 621 is trained based on the vector of each word by mapping semantically similar words to closed points.

[0228] W("cat")=(0.2,-0.4,0.7,...)

[0229] W("mat")=(0.0,0.6,-0.1,...)

[0230] For example, the word embedding model 621 converts words such as cat or mat into vectors of a specific size. The matrix "W" is used to convert words into vectors. Due to the conversion through the matrix, the words become meaningful vectors. Two similar words can be converted into similar vectors, and the matrix W can be obtained through training. For example, in a set of 5,000 words, each word can be represented as [0, 0, 0, 1, 0, ..., 0, 0, 0] (5,000 columns). In this case, when each word is represented as a 32-dimensional vector through word embedding, the dimension of the matrix W is 5,000×32. 32-dimensional vectors such as [0.2, 0, 4, 0.5, ..., 0.8, 8] are obtained through training.

[0231] Figure 15 An example of word embedding according to an embodiment is shown. Figure 15 , the king and the queen, the king and the man, or the queen and the woman are in the same direction. Words with similar meanings are located in similar directions. Word embeddings effectively represent the meaning of words, and therefore can improve training performance compared to one-hot encoding. In addition, when word embeddings are pre-trained with a large amount of data, by using the trained embeddings in tasks such as document classification, relatively high performance can be achieved with less data. Types of word embeddings include, for example, LSA, Word2Vec, GloVe, and fastText.

[0232] Word2Vec, a type of word embedding, is a model created by modifying a previous neural network language model (NNLM) to enable efficient training. A language model is used to predict the next word given previously given words. Training is enabled using only unlabeled text (e.g., classification). Word2Vec creates word embeddings using the byproducts of training this language model.

[0233] Word2Vec includes two methods, such as CBOW and Skip-Gram. CBOW predicts the target word by adding the embeddings of the surrounding words, while Skip-Gram predicts the surrounding words by embedding the target word.

[0234] Figure 16 An example of the CBOW method of Word2Vec according to an embodiment is shown.

[0235] Reference Figure 16 ,First, the input layer converts all the words of the sentence to be trained into vectors by using one-hot encoding.,Given 2m word vectors as the input value of a center word.

[0236] The parameters include the parameter matrix W (W∈R VxN) between the input layer and the hidden layer, and the parameter matrix W' (W'∈R VxN) between the hidden layer and the output layer.

[0237] The goal of the model is to maximize the conditional probability of the center word given the surrounding words.

[0238] The word vector of the input layer according to the one-hot encoding is multiplied by the parameter W to become the embedded word vector.

[0239] The hidden layer averages the 2m embedding vectors. This hidden layer has fewer neural networks than the input space, so the neural network is trained using a compressed form of the input layer information. This neural network creates a hidden layer with a weight matrix of [number of words × number of neural networks]. For example, if the vocabulary contains 20,000 unique words and the hidden layer contains 300 neural networks, the hidden layer's weight matrix might be 20,000 × 300. Once this weight matrix is ​​stored, a vector with 300 elements is available to represent each word.

[0240] To calculate the score value to be sent from the hidden layer to the output layer, the score of each word is obtained by multiplying it by the parameter W'. Words that are closer in position have higher scores.

[0241] Finally, the output layer calculates each score value as a probability value by using softmax.

[0242] In order to perform training using these parameters, an objective function in which the value of H becomes 0 when the word to be predicted is accurately predicted may be defined, and training may be performed in a direction that minimizes the value of the objective function.

[0243] When the training of the word embedding neural network is completed, the weights of a single hidden layer of the neural network are used as a lookup table for word embeddings. After training with all sentences, each row of the matrix W can be used as an embedding vector for each word.

[0244] The word embedding model 621 may configure a word embedding lookup table by using embedding vectors of words obtained by training a language model for predicting the next word based on previously given words.

[0245] Figure 17 is a diagram illustrating a word embedding lookup table 1700 obtained by training the word embedding model 621 according to an embodiment.

[0246] Reference Figure 17 , the word embedding lookup table 1700 is a vector space corresponding to the word. Therefore, the word embedding lookup table 1700 obtained after the training is completed can be used to obtain the object class vector corresponding to each object category. It is shown that each symbol is mapped to a category, and for example, the distance between a dog and a cat in the vector space is smaller than the distance between a dog and a car, because the relationship between a dog and a cat is closer than the relationship between a dog and a car. Figure 17 For example, the dog class may include various types of dogs. For example, the dog class may include various types of dogs such as Chihuahuas, Yorkshire Terriers, Poodles, Dachshunds, Maltese Dogs, and Beagles. In the dog class, semantically closely related dogs may be close to each other, while semantically related dogs may be far away from each other.

[0247] When the object list includes boy, bird, and house as object categories, the word embedding model 621 can obtain a class vector corresponding to boy, a class vector corresponding to bird, and a class vector corresponding to house by using the word embedding lookup table 1700, and provide the class vectors to the target object class vector extraction module 622.

[0248] Then, the target object class vector extraction module 622 can refer to the word embedding lookup table 1700 configured by the word embedding model 621, and obtain the target object class vector by using the class vectors and distances of one or more objects received from the word embedding model 621. That is, the target object class vector extraction module 622 can determine information about the target object to be located in the synthesis area of ​​the input image by using objects such as a boy, a bird, and a house detected in the input image. For example, when the detected objects include a boy, a bird, and a house, appropriate target objects placed adjacent to the objects may include animals such as a dog or a cat, or plants such as a tree or a flower.

[0249] Figure 18 An example of a method used by the target object class vector extraction module 622 to obtain the target object class vector 1400 by referring to the word embedding lookup table 1700 according to an embodiment is shown.

[0250] Reference Figure 18Target object class vector extraction module 622 finds a class vector corresponding to boy 1810, a class vector corresponding to bird 1820, and a class vector corresponding to house 1830 in word embedding lookup table 1700. In the input image, the distance from the synthesized region to the boy object is 3, the distance from the synthesized region to the bird object is 5, and the distance from the synthesized region to the house object is 20. The distance from the synthesized region to the object can be considered to determine the target object. For example, a higher weight can be assigned to the boy, as the object closest to the synthesized region, while a lower weight can be assigned to the house, as the object farthest from the synthesized region. As described above, the center point 1840 can be found taking into account the distance from the synthesized region. Target object class vector extraction module 622 then determines the class vector within a specific range 1850, i.e., within a specific distance from the center point 1840, as the target object class vector 1400.

[0251] For example, reference Figure 18 , the class vectors within a specific distance (e.g., a specific range 1850) from the center point 1840 include D1 to D15 and C1 to C3. D1 to D15 are class vectors indicating various dogs belonging to the dog category, while C1 to C3 are class vectors indicating various cats belonging to the cat category. For example, D1 to D15 may be class vectors corresponding to dog species, such as bulldogs, Chihuahuas, Yorkshire terriers, poodles, dachshunds, Maltese dogs, and beagles, and C1 to C3 may be class vectors corresponding to cat species, such as Persian cats, American shorthair cats, and Siamese cats. As described above, the target object class vector extraction module 622 may output D1 to D15 and C1 to C3 as the target object class vector 1400.

[0252] Figure 19 An example of a method used by the target object class vector extraction module 622 to obtain the target object class vector 1400 by referring to the word embedding lookup table 1700 according to an embodiment is shown.

[0253] Reference Figure 19 , the target object class vector extraction module 622 finds a class vector corresponding to a boy, a class vector corresponding to a bird, and a class vector corresponding to a house in the word embedding lookup table 1700. Because the distance from the synthesized region to the boy object in the input image is 3, the distance from the synthesized region to the bird object is 5, and the distance from the synthesized region to the house object is 20, the target object can be determined taking into account the distance from the synthesized region to the object. For example, a higher weight can be assigned to the boy, which is the object closest to the synthesized region, so that more class vectors are included in the dog cluster adjacent to the boy, while a lower weight can be assigned to the house, which is the object farthest from the synthesized region, so that fewer class vectors are included in the cat cluster adjacent to the house.

[0254] According to the above method, for example, referring to Figure 19 The target object class vector extraction module 622 can obtain class vectors D1 to D8 and D11 to D15 within the specific range 1910 of the dog cluster, class vectors C1 to C5 within the specific range 1920 of the cat cluster, and class vectors F1 to F3 within the specific range 1930 of the flower cluster as the target object class vector 1400. Therefore, the target object class vector extraction module 622 can output D1 to D8, D11 to D15, C1 to C5, and F1 to F3 as the target object class vector 1400.

[0255] Reference above Figure 18 and 19 A method for extracting target object class vectors adjacent to objects detected in an input image by using a word embedding lookup table is described. However, Figure 18 and Figure 19 The method shown in is only an example, and various methods of extracting a target object class vector by using a word embedding lookup table may be used.

[0256] Feature extraction module

[0257] Figure 20 An example of a feature extraction module 650 according to an embodiment is shown.

[0258] Reference Figure 20 The feature extraction module 650 can receive a synthetic area image 210 to be combined with a target object image in the input image 200, extract one or more features from the synthetic area image 210 by using one or more neural networks, and output a style vector 2000 corresponding to the extracted one or more features to at least one of the image generation module 630 or the image synthesis module 640.

[0259] According to one embodiment, the feature extraction module 650 is used to extract features from the synthetic region image 210 by using one or more neural networks. The feature extraction model may include, for example, a classifier pre-trained using ImageNet for each convolution layer.

[0260] As feature extraction models, CNN and variations including attention and skip connections can be used.

[0261] Figure 21 1 shows the network architecture of CNN 1220 according to an embodiment.

[0262] CNN 1220 includes a convolutional feature extraction part 2110 and a classification part 2120. Features of the input image are extracted through the convolution layer, and classification is performed based on the extracted features through a neural network using relevant techniques available to those skilled in the art.

[0263] Each convolutional layer is used to extract features from the input data and includes a filter for extracting features and an activation function for converting the value of the filter into a nonlinear value. The filter can be a function for detecting whether the feature of the image to be extracted exists in the target data. When the feature map is extracted by the filter as described above, the value is activated by applying the activation function to the feature map. Representative activation functions include, for example, sigmoid and rectified linear unit (ReLU) functions.

[0264] Subsampling the features extracted as described above to reduce the amount of computation by reducing the size of the extracted feature map is a process called pooling.

[0265] The fully connected layer corresponds to a process of performing classification by applying the feature values ​​extracted by the convolutional layer to a neural network of the related art, and a softmax function, for example, can be used.

[0266] The feature extraction module 650 can extract one or more feature maps generated in the above CNN architecture, such as feature Figure 1-4 , and use it as the image feature of the synthesis area. The feature extraction module 650 can Figures 1 to 4 Convert to the form of vectors and output style vectors 1 to 4. The feature extraction module 650 may output, for example, one of the style vectors 1 to 4.

[0267] The feature extraction module 650 may extract various features by using convolutional layers and various filters at various image scales.

[0268] For example, lower-level features of an image may be extracted by a shallower convolution layer, and higher-level features of an image may be extracted by a deeper convolution layer. For example, in a facial image of a person, lower-level features may include local features indicating color or lines (e.g., skin color and edge direction of the chin), while higher-level features may include combined features of eyes, nose, etc. Therefore, the feature extraction module 650 may appropriately extract and use feature maps corresponding to higher-level features and feature maps corresponding to lower-level features.

[0269] In addition, the feature factors to be extracted from the image can be adjusted by using, for example, a filter used in each convolutional layer. For example, color or texture features can be extracted from the image by using a filter for extracting color information from the image or a filter for extracting texture information from the image.

[0270] For example, features of an image may include color and texture features, and color features may include image color, a color histogram representing color distribution in the image, color moments, and color coherence vectors, while texture features may include edges.

[0271] For example, when Figure 20 As shown, when a portion of a grass image is input as a composite region of the input image 200, the feature extraction module 650 can output a style vector corresponding to a color feature indicating green and a style vector corresponding to a texture feature indicating grass based on the composite region. As described above, the features of the composite region can be used by the image generation module 630 to generate an image of an object to be located in the composite region, or can be used by the image synthesis module 640 to combine the input image 200 with a target object image.

[0272] Image generation module

[0273] Figure 22 An example of the image generation module 630 according to an embodiment is shown.

[0274] Reference Figure 22 According to an embodiment, the image generation module 630 may receive the target object class vector 1400 and generate the target object image 2200 corresponding to the target object class vector 1400 by using an image generation model.

[0275] According to an example, the image generation module 630 may receive the style vector 2000 in addition to the target object class vector 1400, and generate a target object image 2200 corresponding to the target object class vector 1400 and the style vector 2000 by using the image generation model. As described above, when the style vector 2000 output from the feature extraction module 650 is also used to generate the target object image 2200, the target object image 2200 may be generated more appropriately for the synthesized region of the input image.

[0276] Representative examples of image generation models include GAN.

[0277] Figure 23 The architecture of a GAN 2300 that may be used by the image generation module 630 is shown, according to an embodiment.

[0278] Reference Figure 23 , GAN 2300 includes a generator 2320 and a discriminator 2360.

[0279] The generator 2320 generates new instances by using random noise, and the discriminator 2360 determines whether each data instance corresponds to a real training dataset by evaluating the data authenticity, that is, whether the input image is a real image or a fake image. Given the characteristics of the data instance, the label or category of the corresponding data will be predicted.

[0280] Generator 2320 is a function for receiving a random vector or latent vector "z" 2310 as input and outputting a pseudo image sample 2330. Here, "z" is simply a value randomly extracted from a uniform distribution or a normal distribution. Generator 2320 can be considered as a function for mapping a simple distribution to a complex distribution (such as a person's facial image). When the generator model includes a sufficient number of parameters, complex distributions can be approximated. The space containing the vector "z" is called the latent space. Here, the size of the latent space can be arbitrarily determined, for example, 100 dimensions. The size of the latent space is not particularly limited, but needs to be sufficient for the target information. This is because GAN 2300 maps the values ​​of the vector "z" to image attributes.

[0281] Generator 2320 is intended to generate fake data that is indistinguishable from real data.

[0282] The discriminator 2360 is trained using real training data (e.g., real-world images) and fake data generated by the generator 2320, and is used to determine whether a sample is real or fake. The discriminator 2360 is a function that receives an image as input and outputs a probability between 0 and 1 that the image is real.

[0283] By repeatedly training the discriminator 2360 in the direction of improving the discriminator 2360's discrimination ability, and by repeatedly training the generator 2320 in the direction of deceiving the discriminator 2360, the generator 2320 is ultimately designed to generate data that is difficult to determine whether it is real or forged, and the discriminator 2360 is ultimately designed to gradually improve its discrimination ability. GAN 2300 can adversarially train the generator 2320 and the discriminator 2360 until it is difficult to determine whether the image is real or forged.

[0284] Figure 24A and 24B is a diagram illustrating a method of training an image generation model according to an embodiment.

[0285] GAN-based learning is performed in two stages, and the first stage is a stage for fixing the generator 2320 and training the discriminator 2360. Unlike the discriminator training method of the related art, since the discriminator 2360 already knows which images are real and which are fake, a cost function or loss function is defined and the weights are updated by backpropagating the error.

[0286] Reference Figure 24A and 24BWhen a real image sample 2350 from the real image dataset 2340 is input, the discriminator 2360 outputs a probability value close to 1, and when a fake image sample 2330 is input, the discriminator 2360 outputs a probability value close to 0. Therefore, the loss function 2370 of the discriminator 2360 consists of the sum of two values. The sum of the difference between the output value when a real image is input and the value 1, and the difference between the output value when a fake image is input and the value 0 is the loss function 2370 of the discriminator 2360. The discriminator 2360 is trained by updating its parameters in a direction that minimizes the value of the loss function 2370.

[0287] Reference Figure 24B The second stage is to fix the discriminator 2360 and train the generator 2320. The generator 2320 is designed to deceive the discriminator 2360, and therefore the generator 232 is trained in a direction that causes the discriminator 2360 to mistake the forged image for the real image. In other words, the purpose of the generator 2320 is to deceive the discriminator 2360. In other words, when the forged image generated by the generator 2320 is input to the discriminator 2360, a value close to 1 is output. The difference between this value and the value 1 serves as the loss function of the generator 2320, and the generator 2320 is trained to minimize it.

[0288] When the above two stages are repeatedly performed, the discriminator 2360 and the generator 2320 are improved to an equal level.

[0289] Figure 25 An example of an image generation model 2500 further using the style vector 2000 according to an embodiment is shown. The image generation module 630 may use the image generation model 2500 .

[0290] Reference Figure 25 , in the image generation model 2500, the style vector 2000 is further added as the input of the generator 2320 and the discriminator 2360.

[0291] The image generation model 2500 is trained by adding the style vector 2000 as a condition y to the related art GAN.

[0292] The generator 2320 generates a fake image sample 2330 by concatenating the latent vector 2310 and the style vector 2000, and the discriminator 2360 receives the fake image sample 2330 and the style vector 2000 as input. However, the combination of the style vector 2000 is not limited to concatenation, and a simple sum or projection, for example, may also be used.

[0293] As described above, when style vector 2000 is further used for training, image generation model 2500 can generate an image by reflecting the image features corresponding to style vector 2000. For example, when style vector 2000 is associated with a corresponding feature indicating green color information, image generation model 2500 can generate an image by further using the green color information. When style vector 2000 is associated with a corresponding feature indicating grass texture edge information, image generation model 2500 can generate an image by further using the grass texture edge information.

[0294] Figure 26 20 is a diagram illustrating an operation of the image generating module 630 using the pattern vector 2000 according to an embodiment.

[0295] Reference Figure 26 For example, the image generation module 630 may receive a class vector corresponding to <bulldog> as the target object class vector 1400 , and receive a style vector including color and texture information corresponding to green and grass texture as the style vector 2000 .

[0296] Image generation module 630 can use the color and texture information of style vector 2000 to generate target object image 2600, reflecting the style vector, as a <Bulldog> image corresponding to the <Bulldog> class vector. If style vector 2000 is not used, image generation module 630 can generate any <Bulldog> image corresponding to the received <Bulldog> class vector. However, if image generation module 630 receives style vector 2000, the color and texture information of style vector 2000 may affect at least one of the foreground or background of the <Bulldog> image. Therefore, image generation module 630 may generate a <Bulldog> image reflecting a green or grassy texture.

[0297] Image synthesis module

[0298] Figure 27 is a diagram illustrating an example of the image synthesis module 640 according to an embodiment.

[0299] Reference Figure 27 , the image synthesis module 640 may include an image segmentation module 641 and an image combination module 642 .

[0300] The image segmentation module 641 uses one or more neural networks to identify the target object in the target object image 2600 received from the image generation module 630. According to an embodiment, the image segmentation module 641 may receive the target object image 2600 output from the image generation module 630, the synthetic region image 2700 extracted from the input image 200, and the style vector 2000 output from the feature extraction module 650, identify the target object in the target object image 2600 by using the received data, and provide the identified target object to the image combination module 642.

[0301] The image combination module 642 may generate and output a synthesized image 300 by combining the input image 200 with the target object recognized by the image segmentation module 641 .

[0302] The image segmentation module 641 will now be described in detail.

[0303] Semantic segmentation is the process of accurately extracting the boundaries of objects in an image and dividing the image into meaningful regions so as to divide the image into semantically or cognitively similar regions. In other words, semantic segmentation defines the boundaries of each object in the image by understanding the image at the pixel level and assigning an object category to each pixel of the image.

[0304] For example, semantic segmentation generates predictions by labeling the class (or label) value of the pixel currently being viewed. For example, to segment an image into grass and dog, the pixel regions corresponding to dogs are labeled with a value of "1" and the grass regions are labeled with a value of "0." The model then places a blue mask on the regions labeled with a value of 1 and a green mask on the regions labeled with a value of 0, thus clearly distinguishing between the dog and grass regions.

[0305] Figure 28A An example of an image segmentation module according to an embodiment is shown.

[0306] Reference Figure 28A , the image segmentation module 641 can receive the target object image 2600 output from the image generation module 630 .

[0307] Image segmentation module 641 may label a pixel region corresponding to the dog in received target object image 2600 with a value of 1 and a background region with a value of 0, and may place a gray mask, for example, on the region indicated by the value of 1 and a white mask, for example, on the region indicated by the value of 0, to check the boundary of the dog corresponding to the object in target object image 2600. By specifying the region corresponding to the dog in target object image 2600 and outputting pixel information corresponding to the dog, target object region 2810 (e.g., a dog region) in target object image 2600 may be output.

[0308] The result image 2800 may include information for distinguishing the target object region 2810 and the background region 2820 in the target object image 2600. However, the boundary region 2830 between the target object region 2810 and the background region 2820 may be unnaturally detected because pixels identified as background may be identified as objects, or pixels identified as objects may be identified as background. The boundary region 2830 detected unnaturally as described above is Figure 28A Indicated by thick lines.

[0309] Figure 28B is an example of an image segmentation module according to an embodiment.

[0310] In one embodiment, Figure 28A As shown, in order to prevent unnatural detection of object boundaries, refer to Figure 28A , the image segmentation module 641 may include, in addition to the target object image 2600 , a synthesis region image 2700 and a style vector 2000 as input.

[0311] The target object image 2600 may include three RGB channels, and the synthesized region image 2700 may also include three RGB channels. The pattern vector 2000 may include various numbers of channels. For example, when color features and edge features are included, the pattern vector 2000 may include four channels, including three RGB channels for color features and one channel for edge features.

[0312] As described above, the image segmentation module 641 receives data of ten channels and identifies the target object in the target object image by using the received data. In this case, the image segmentation module 641 can naturally detect the boundary region 2870 of the target object in the target object image by further considering the synthetic region image 2700 and the style vector 2000 indicating the characteristics of the synthetic region image 2700. The region 2870 detected more naturally using the style vector 2000 is Figure 28B Indicated by thin lines.

[0313] When an image corresponding to the target object area 2810 or 2850 is input to the image combination module 642 through the image segmentation module 641, the image combination module 642 can generate a composite image by combining the image corresponding to the target object area 2810 or 2850 received from the image segmentation module 641 on the area of ​​the input image 200 to be combined with the target object.

[0314] Figure 29 An example of a semantic segmentation model according to an embodiment is shown.

[0315] The semantic segmentation model has an encoder-decoder structure. The encoder represents the information of the input image as a compressed vector, and the decoder generates a result of the desired size.

[0316] refer to Figure 29 , the semantic segmentation model according to an embodiment includes an encoder 2910 and a decoder 2920.

[0317] The encoder 2910 performs downsampling processing to achieve depthwise convolution with less memory by reducing the size of the image. For example, convolution with a stride of 2 or more or a pooling layer is used. When this processing is performed, the feature information of the image is lost, and in most cases a fully connected network is used without providing a fully connected layer at the end.

[0318] The decoder 2920 mainly performs an upsampling process to increase the size of the downsampled image to the size of the input image.

[0319] Such encoder-decoder models include, for example, fully convolutional networks (FCNs), SegNet, and U-Net.

[0320] The pooling layers of encoder 2910 discard position information, increase views, and gather image context. However, because semantic segmentation requires accurate class map alignment, position information can be preserved. This U-Net consists of an encoder that gradually reduces spatial dimensions through pooling layers; a decoder that gradually reconstructs object details and spatial dimensions; and shortcut connections from the encoder to the decoder to allow the decoder to reconstruct object details. These connections are called skip connections.

[0321] U-Net is a model obtained by adding a skip connection 2930 to the encoder-decoder structure. When the image size is reduced (downsampled) and then increased again (upsampled), detailed pixel information is lost. However, because the above problem may be seriously considered in image segmentation requiring dense pixel-based prediction, the decoder 2920 can obtain a clearer image and perform more accurate prediction through the skip connection 2930 to provide position information directly from the encoder 2910 to the decoder 2920.

[0322] The synthesized image generated as described above by the image synthesis module 640 may be output. The synthesized image may be provided to a display and displayed on the display.

[0323] Object Control Module

[0324] Figure 30 is a view illustrating an object control module 660 according to an embodiment.

[0325] Reference Figure 30, the object control module 660 may receive user input for controlling and updating a target object included in an output composite image.

[0326] For example, when a composite image generated by the image synthesis module 640 and including a target object is output on a display, a user may view the output composite image and then desire to change the target object included in the composite image. In this case, the user may input a control command (i.e., user input) for changing or updating the target object included in the composite image into the electronic device, and the object control module 660 may receive the user input.

[0327] The user input for controlling the target object can be implemented in various forms. For example, when the display of the electronic device is implemented as a touch screen display, the user input for touching the target object in the composite image displayed on the display can be used as the user input for controlling the target object. Alternatively, instead of touching, a user gesture or voice can be used. Alternatively, a scroll bar for controlling the target object can be displayed on the display.

[0328] Figure 31 is a view illustrating an example of displaying a scroll bar 3100 to control a target object according to an embodiment.

[0329] Reference Figure 31 , the electronic device 100 can output a composite image 300 generated by combining the input image with the target object image on the display 130. In this case, the electronic device 100 can display the composite image 300 and a scroll bar 3100 for receiving a user input for controlling the target object on the display 130. The scroll bar 3100 may include a scroll button 3110 that can be moved on the scroll bar 3100. The user can move the scroll button 3110 on the scroll bar 3100 by using various input means. The distance information and position information of the scroll button 3110 moved by the user's control can be used as input for the operation of updating the target object class vector.

[0330] The object control module 660 may receive the distance information and the position information of the scroll button 3110 and provide the received distance and position information and a command for instructing to update the target object class vector to the embedding module 620 .

[0331] When receiving the distance information and the position information and a command for instructing to update the target object class vector, the embedding module 620 may update the target object class vector based on the distance information or the position information.

[0332] Figure 31The display 130 is shown generating and displaying a composite image including an updated target object image corresponding to the class vector C4 based on the user's class vector control. That is, although the composite image including the <Bulldog> image as the target object image is initially displayed, the target object image is updated to other types of dogs D2, D3, ..., and then updated to cat types, ..., C4, after all types of dogs have been displayed, through scroll control by the user who wishes to update the target object image.

[0333] Figure 32 is a diagram illustrating an operation performed by the embedding module 620 to update a target object class vector according to an embodiment.

[0334] The operations performed by the embedding module 620 for updating the target object class vector are substantially similar to those described above with reference to Figure 14 The operations performed by the embedding module 620 described above for obtaining the target object class vector. Figure 14 As mentioned above, the embedding module 620 has generated a certain number of target object class vectors. Figure 18 Embedding module 620 has already obtained fifteen class vectors D1 to D15 for the DOG class and three class vectors C1 to C3 for the CAT class. The electronic device may have already generated a target object image and a composite image corresponding to the obtained target object class vectors. Therefore, when the range controlled by the user is within a specific range 1850, the electronic device may output the generated composite image.

[0335] However, when the range according to the user's control exceeds the specific range 1850, the electronic device may update the target object class vector corresponding to the user input.

[0336] For example, refer to Figure 32 , for example, when the class vector control direction 3200 indicates the lower right direction of the specific range 1850 according to the user input, for example, class vectors C4, C5, and C6 in the CAT category can be obtained as updated target object class vectors.

[0337] The target object class vectors C4, C5, and C6 newly obtained by the embedding module 620 may pass through the image generation module 630 and the image synthesis module 640, and may generate a synthesized image including the updated target object corresponding to C4, C5, and C6.

[0338] Return Reference Figure 31, the user can control the target object class vector by using user input. For example, when the electronic device 100 initially outputs a composite image including a target object corresponding to class vector D1, and when the user moves the scroll button 3110 in units for class vector control, the electronic device 100 can output a composite image including a target object corresponding to class vector D2. For example, when the user continuously moves the scroll button 3110, the electronic device 100 can output a composite image including a target object corresponding to class vector C4. As described above, the user can update the target object included in the composite image by moving the scroll button 3110 until the user's desired target object is displayed on the composite image. For example, when the user wishes to combine another type of dog, the currently combined dogs can be sequentially changed from the most similar dog to the least similar dog by linearly moving the gauge in one direction, and further moving the gauge to change to other types of animals such as cats or lions, for example.

[0339] As described above, the object control module 660 can update the target object to be located in the composite image displayed on the display according to a simple user input, and thus can conveniently obtain the target object desired by the user.

[0340] Figure 33 3 is a block diagram showing a configuration of a processor 3300 in terms of training and processing of a neural network according to an embodiment. For example, the processor 3300 may correspond to the above-described processor 120.

[0341] Reference Figure 33 , the processor 3300 according to an embodiment may include a data learner 3310 and a data processor 3320 .

[0342] According to an embodiment, the data learner 3310 may learn criteria for detecting one or more objects in an input image to train the first neural network. The data learner 3310 may learn criteria for the type of information (e.g., feature information) of the input image used to detect the object. In addition, the data learner 3310 may learn criteria for a method for detecting an object by using feature information of the image. The data learner 3310 may obtain data (e.g., an image) to be used for learning by applying the obtained data to a data processing model (i.e., a first neural network) and learn criteria for detecting one or more objects in the image.

[0343] According to an embodiment, the data learner 3310 may learn a criterion for obtaining an object class vector by using an input object category to train the second neural network. The data learner 3310 may learn a criterion for the information type (e.g., feature information) of an image for which emotion information is to be obtained.

[0344] According to an embodiment, the data learner 3310 may learn a criterion for extracting image features from an input image to train the third neural network.

[0345] According to an embodiment, the data learner 3310 may train the fourth neural network by learning a criterion for generating an object image using the input object class vector.

[0346] According to an embodiment, the data learner 3310 may learn a criterion for recognizing an object in an input object image to train the fifth neural network.

[0347] For example, the data processing model (e.g., the first to fifth neural networks) can be configured taking into account the applicable field of the data processing model, the learning purpose, or the computing performance of the device. The data processing model can be, for example, a model based on a neural network. For example, the data processing model can use a model based on a DNN, RNN, or BRDNN, but is not limited thereto.

[0348] The data learner 3310 may train a data processing model by using a learning algorithm including, for example, error back propagation or gradient descent.

[0349] The data learner 3310 can train the data processing model through, for example, supervised learning using training data as input values. The data learner 3310 can also train the data processing model through, for example, unsupervised learning to find data processing guidelines by autonomously learning the data types required for data processing without any supervision. The data learner 3310 can also train the data processing model through, for example, reinforcement learning using feedback on whether the training result values ​​are correct.

[0350] When training the data processing model, the data learner 3310 may store the trained data processing model. In this case, the data learner 3310 may store the trained data processing model in a memory of the electronic device. Alternatively, the data learner 3310 may store the trained data processing model in a memory of a server connected to the electronic device in a wired or wireless network.

[0351] The data processor 3320 may input the image into a data processing model including the trained first neural network, and the data processing model may output information about one or more objects detected in the image as a result value. The output result value may be used to update the data processing model including the first neural network.

[0352] The data processor 3320 may input one or more object categories into a data processing model including the trained second neural network, and the data processing model may output target object class vectors adjacent to the one or more object categories as result values. The output result values ​​may be used to update the data processing model including the second neural network.

[0353] The data processor 3320 may input the image into the data processing model including the trained third neural network, and the data processing model may output feature information of the image as a result value. The output result value may be used to update the data processing model including the third neural network.

[0354] The data processor 3320 may input one or more target object class vectors into a data processing model including the trained fourth neural network, and the data processing model may output a target object image corresponding to the one or more target object class vectors as a result value. The output result value may be used to update the data processing model including the fourth neural network.

[0355] The data processor 3320 may input the target object image into the data processing model including the trained fifth neural network, and the data processing model may output the result of identifying the target object in the target object image as a result value. The output result value may be used to update the data processing model including the fifth neural network.

[0356] At least one of the data learner 3310 or the data processor 3320 may be produced in the form of at least one hardware chip and installed in an electronic device. For example, at least one of the data learner 3310 or the data processor 3320 may be produced in the form of a dedicated hardware chip for AI or as part of a processor (e.g., a CPU or an application processor) or a dedicated graphics processor (e.g., a GPU).

[0357] Information about the model configured by the data learner 3310 may be provided to the data processor 3320 in a wired or wireless manner, and data input to the data processor 3320 may be provided to the data learner 3310 as additional training data.

[0358] At least one of the data learner 3310 or the data processor 3320 may be implemented as a software module. When at least one of the data learner 3310 or the data processor 3320 is implemented as a software module (or a program module including instructions), the software module may be stored in a non-transitory computer-readable medium. In this case, at least one software module may be provided by an operating system or a specific application. Alternatively, a portion of at least one software module may be provided by the OS, while another portion may be provided by a specific application.

[0359] The data learner 3310 and the data processor 3320 may be installed in one electronic device or in separate electronic devices. For example, one of the data learner 3310 and the data processor 3320 may be included in the electronic device, and the other may be included in the server.

[0360] According to an embodiment, the data learner 3310 and the data processor 3320 may be installed in the user's electronic device, and both learning and data processing may be performed in the user's electronic device.

[0361] According to an embodiment, the data learner 3310 may be installed in a server, and the data processor 3320 including the training model may be installed in a user's electronic device.

[0362] Figure 34A An example of a data learner 3310 installed in a server 3400 and a data processor 3320 installed in a user's electronic device 3410 according to an embodiment is shown. For example, the user's electronic device 3410 may correspond to the above-described electronic device 100, or may be a different device.

[0363] Reference Figure 34A , the server 3400 can obtain an image synthesis neural network model by using the data learner 3310 to learn a method for combining two or more images according to the method described herein. The server 3400 can provide the trained neural network model to the user's electronic device 3410. The user's electronic device 3410 can implement the data processor 3320 by using the trained neural network model received from the server 3400. When the user wishes to combine images, the user's electronic device 3410 can autonomously combine the images according to the user's request by using the data processor 3320 without communicating with the server 3400, and output the synthesized image on the display of the user's electronic device 3410.

[0364] Figure 34B An example of a data learner 3310 and a data processor 3320 installed in a server 3400 according to an embodiment is shown.

[0365] like Figure 34B As shown, the data learner 3310 and the data processor 3320 are both installed in the server 3400. Therefore, the server 3400 can obtain an image synthesis neural network model by using the data learner 3310, by learning a method of combining two or more images according to the method described herein, and implement the data processor 3320 by using the obtained image synthesis neural network model.

[0366] When the user desires to combine images, the user's electronic device 3410 sends an image synthesis request to the server 3400, and the server 3400 can generate a synthesized image by combining images according to the user request using the data processor 3320, and output the synthesized image to the user's electronic device 3410 to display the synthesized image on the display of the user's electronic device 3410.

[0367] The operating method of the electronic device according to the embodiment can be implemented in the form of a program command that can be executed by various computer devices, and is recorded on a computer-readable recording medium. For example, the software (for example, program) comprising one or more instructions can be stored in a machine-readable (for example, computer-readable) storage medium (for example, internal memory) or an external memory. The computer-readable recording medium may include program commands, data files, data structures, or a combination thereof. The program commands recorded on the medium may be those specifically designed and configured for the embodiment, or those known to those of ordinary skill in the relevant art. The example of a computer-readable recording medium includes magnetic media (for example, hard disk, floppy disk, and magnetic tape), optical media (for example, CD-ROM and DVD), magneto-optical media (for example, floppy disk), and hardware specifically configured for storing and executing the equipment (for example, ROM, RAM, and flash memory, etc.) of program commands. The example of a program command includes machine code (for example, machine code generated by a compiler) and the high-level language code that can be executed by a computer using an interpreter.

[0368] According to an embodiment, a user does not need to manually find a suitable target image to be combined with an input image, and an electronic device using machine learning can generate and provide candidate images suitable for the input image.

[0369] Furthermore, according to an embodiment, the electronic device may change the candidate image into a style suitable for the input image by using features of a synthesis position selected by a user in the input image.

[0370] In addition, according to the embodiment, candidate images suitable for an input image can be updated according to user control, so the user can obtain a desired composite image more conveniently and accurately in a time-saving manner.

[0371] Although specifically shown and described with reference to the accompanying drawings, the embodiments are provided for illustrative purposes, and those skilled in the art will appreciate that various modifications and equivalent other embodiments may be made based on the present disclosure. Therefore, the true technical scope of the present disclosure is defined by the technical spirit of the appended claims.

Claims

1. An electronic device comprising: a memory storing one or more instructions; and A processor configured to execute one or more instructions to: receiving a selection of a synthesis region in an input image, detecting at least one object from an input image, obtaining, by using one or more neural networks, a target object image to be located in a synthesized area of ​​the input image based on information about at least one object detected in the input image, the information about the at least one object detected in the input image including an object category of the at least one object and distance information indicating a distance from a position of the synthesized area to a position of the at least one object detected in the input image, generating a synthetic image by combining an input image with a target object image using one or more neural networks, and A display of the electronic device is controlled to display the composite image.

2. The electronic device according to claim 1, wherein The processor is further configured to execute the one or more instructions to: detecting a plurality of objects in an input image, the at least one object being one of a plurality of objects, Obtaining object categories corresponding to the plurality of objects and position information of the plurality of objects in the input image, respectively, by using a first neural network of the one or more neural networks; and Based on the position information of the composite area and the position information of the plurality of objects, distance information from the center of the composite area to the plurality of objects is respectively obtained.

3. The electronic device according to claim 2, wherein The processor is further configured to execute the one or more instructions to obtain a class vector of a target object available in the target object image based on the object category by using a second neural network of the one or more neural networks.

4. The electronic device according to claim 3, wherein The processor is further configured to execute the one or more instructions to: obtaining class vectors for each of the plurality of objects by using a second neural network, and By using a word embedding lookup table generated based on the second neural network, a class vector of a target object adjacent to the class vectors of the multiple objects is obtained based on distance information from the center of the synthesis area to the multiple objects.

5. The electronic device according to claim 4, wherein The processor is further configured to execute the one or more instructions to obtain the class vector of the target object by assigning a higher weight to a class vector of an object located closer to a center of a synthesis area among the plurality of objects. The electronic device according to claim 3 , wherein: The processor is further configured to execute the one or more instructions to generate the target object image corresponding to the class vector of the target object by using a third neural network among the one or more neural networks.

7. The electronic device according to claim 6, wherein: The processor is further configured to execute the one or more instructions to: extracting a synthesis area image feature corresponding to the image of the synthesis area by using a fourth neural network of the one or more neural networks, and A target object image corresponding to a class vector of the target object is generated by reflecting the synthesized region image feature to the target object image using a third neural network.

8. The electronic device according to claim 7, wherein: The processor is further configured to execute the one or more instructions to combine the input image with the target object image reflecting features of the synthesis region image by using a fifth neural network among the one or more neural networks.

9. The electronic device according to claim 1, wherein The processor is further configured to execute the one or more instructions to: generating an updated target object image based on a user input for controlling the target object image included in the displayed composite image, and The display is controlled to display the updated target object image.

10. The electronic device according to claim 9, wherein The processor is further configured to execute the one or more instructions to: Output the scroll bar used to control the update of the target object image to the display, acquiring, based on a user input through a scroll bar, an updated class vector adjacent to the class vector corresponding to the target object; Based on the updated class vector, obtaining an updated target object image; generating an updated composite image by combining the input image with an updated target object image, and The display is controlled to display the updated composite image.

11. A method for operating an electronic device, the method comprising: receiving a selection of a synthesis region in an input image; detecting at least one object from an input image; obtaining, by using one or more neural networks, an image of a target object located in a synthesized region of the input image based on information about at least one object detected in the input image, the information about the at least one object detected in the input image including an object category of the at least one object and distance information indicating a distance from a position of the synthesized region to a position of the at least one object detected in the input image; generating a synthetic image by combining an input image with an image of a target object using one or more neural networks; and The composite image is displayed on a display.

12. The operating method according to claim 11, further comprising: detecting a plurality of objects in an input image, the at least one object being one of the plurality of objects; Obtaining, by using a first neural network of the one or more neural networks, object categories corresponding to the plurality of objects and position information of the plurality of objects in the input image; and Based on the position information of the composite area and the position information of the plurality of objects, distance information from the center of the composite area to the plurality of objects is respectively obtained.

13. The operating method according to claim 12, further comprising: A class vector of a target object is obtained based on the object category by using a second neural network of the one or more neural networks.

14. The operating method according to claim 13, further comprising: obtaining class vectors of the plurality of objects respectively by using a second neural network; and By using a word embedding lookup table generated based on the second neural network, a class vector of the target object adjacent to the class vectors of the multiple objects is obtained based on distance information from the center of the synthesis area to the multiple objects.

15. The operating method according to claim 14, further comprising: The class vector of the target object is obtained by assigning a higher weight to a class vector of an object located closer to the center of the synthetic region among the plurality of objects.

16. The operating method according to claim 13, further comprising: By using a third neural network of the one or more neural networks, a target object image corresponding to the class vector of the target object is generated.

17. The operating method according to claim 16, further comprising: extracting, by using a fourth neural network of the one or more neural networks, a synthetic region image feature corresponding to the image of the synthetic region; as well as A target object image corresponding to a class vector of the target object is generated by reflecting the synthesized region image feature to the target object image using a third neural network.

18. The operating method according to claim 17, further comprising: The input image is combined with a target object image reflecting features of the synthesis region image using a fifth neural network of the one or more neural networks.

19. The operating method according to claim 11, further comprising: generating an updated target object image based on a user input for controlling a target object image included in the displayed composite image; as well as The display is controlled to display the updated target object image.

20. The operating method according to claim 19, further comprising: Output the scroll bar used to control the update of the target object image to the display, acquiring, based on a user input through a scroll bar, an updated class vector adjacent to the class vector corresponding to the target object; Based on the updated class vector, obtaining an updated target object image; generating an updated composite image by combining the input image with an updated target object image, and The display is controlled to display the updated composite image.

21. A computer-readable recording medium having a computer program recorded thereon, the computer program, when executed by a processor, causing the processor to perform an operating method of an electronic device, the operating method comprising: receiving a selection of a synthesis region in an input image; detecting at least one object in an input image using one or more neural networks; obtaining, by using the one or more neural networks, an image of a target object to be located in a synthesized region of the input image based on information about one or more objects detected in the input image, the information about at least one object detected in the input image including an object class of the at least one object and distance information indicating a distance from a position of the synthesized region to a position of the at least one object detected in the input image; generating a synthetic image by combining an input image with an image of a target object using one or more neural networks; as well as The composite image is displayed on a display.

Images