Electronic device and method of operating the same

Abstract

An electronic device according to an embodiment of the present disclosure includes: a display; a memory configured to store one or more instructions; and a processor configured to execute the one or more instructions stored in the memory to recognize one or more objects from an image, determine one or more recommended applications corresponding to the one or more objects respectively by using a learning model of one or more neural networks, and display the determined one or more recommended applications on the display so that the determined one or more recommended applications run.

Description

Technical Field

[0001] The embodiments described herein relate to an electronic device and an operating method thereof, and more particularly, to a method of automatically identifying and displaying an application configuration desired by a user. Background Art

[0002] Artificial intelligence (AI) systems are computer systems that achieve human-level intelligence. Unlike existing rule-based intelligent systems, AI systems are systems in which machines perform learning and determination on their own and become intelligent. As the use of AI systems increases, the recognition rate of AI systems can be improved and user preferences can be understood more accurately. Existing rule-based intelligent systems are gradually being replaced by deep learning-based AI systems.

[0003] AI technology includes machine learning (deep learning) and component technology that uses machine learning.

[0004] Machine learning is an algorithmic technology for autonomously classifying and learning the characteristics of input data. Component technology is a technology that simulates functions such as recognition or determination of the human brain by using machine learning algorithms such as deep learning. Component technology includes technical fields such as language understanding, visual understanding, reasoning or prediction, knowledge representation, and motion control.

[0005] The various fields where AI technology is applied are as follows. Language understanding is a technology for recognizing, applying and processing human language and characters, and includes natural language processing, machine translation, dialogue systems, question and answer, speech recognition and synthesis, etc. Visual understanding is a technology for recognizing and processing objects similar to human vision, and includes object recognition, object tracking, image retrieval, person recognition, scene understanding, spatial understanding, image enhancement, etc. Reasoning or prediction is a technology for determining, logically reasoning and predicting information, and includes knowledge / probability-based reasoning, optimization prediction, preference-based planning, recommendation, etc. Knowledge representation is a technology for automatically processing human experience information into knowledge data, and includes knowledge construction (data generation and classification), knowledge management (data utilization), etc. Motion control is a technology for controlling the autonomous driving of vehicles and the movement of robots, and includes motion control (navigation, collision avoidance, driving, etc.), manipulation control (behavior control), etc. Summary of the invention

[0006] Technical issues

[0007] Various embodiments described herein provide an electronic device and an operating method thereof, the electronic device providing recommended applications corresponding to an object recognized from an image, thereby providing a user's desired application more quickly and conveniently.

[0008] Solution to the problem

[0009] An electronic device according to an embodiment of the present disclosure includes: a display; a memory configured to store one or more instructions; and a processor, the processor configured to execute the one or more instructions stored in the memory to recognize one or more objects from an image, determine one or more recommended applications corresponding to the one or more objects respectively by using a learning model of one or more neural networks, and display the determined one or more recommended applications on the display.

[0010] Disclosure of beneficial effects

[0011] The electronic device and the operation method thereof according to the embodiment of the present disclosure can effectively distribute and execute the operations for training the data recognition model and data recognition. In this way, data processing can be effectively performed to provide services matching the user's intentions, and the user's privacy can be effectively protected. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 is a diagram for describing an electronic device according to an embodiment.

[0013] Figure 2 is a block diagram showing a configuration of an electronic device according to an embodiment.

[0014] Figure 3 is a block diagram showing a configuration of an electronic device according to another embodiment.

[0015] Figure 4 is a diagram showing an example of an image according to the embodiment.

[0016] Figure 5 and Figure 6 is a diagram for describing a process of performing pre-processing on one or more objects recognized from an image according to an embodiment.

[0017] Figure 7 is a diagram for describing a process of determining one or more recommended applications respectively corresponding to one or more objects recognized from an image, performed by an electronic device according to an embodiment.

[0018] Figure 8 is a diagram of a neural network for describing a method for learning to determine a recommended application corresponding to an object according to an embodiment.

[0019] Fig. 9 and Fig.10 is a diagram for describing a process of selecting an application to be displayed on a display from among a plurality of recommended applications according to an embodiment.

[0020] Fig.11 is a diagram illustrating an example in which an electronic device receives a recommended application from an external server and installs the received recommended application according to an embodiment.

[0021] Fig.12 is a diagram illustrating an example in which an electronic device updates a recommended application to reflect text information recognized from an image according to an embodiment.

[0022] Fig.13 is a flowchart of a method of operating an electronic device according to an embodiment.

[0023] Fig.14 is a block diagram showing a configuration of a processor according to an embodiment.

[0024] Fig.15 is a block diagram showing a configuration of a data trainer according to an embodiment.

[0025] Fig.16 is a block diagram showing a configuration of a data identifier according to an embodiment.

[0026] Fig.17 is a block diagram showing a configuration of a server according to an embodiment.

[0027] Fig.18 is a diagram illustrating an example in which an electronic device and a server interact with each other to train and recognize data according to an embodiment. DETAILED DESCRIPTION

[0028] The electronic device includes: a display; a memory configured to store one or more instructions; and a processor configured to execute the one or more instructions stored in the memory to recognize one or more objects from an image, determine one or more recommended applications corresponding to the one or more objects respectively by using a learning model of one or more neural networks, and display the determined one or more recommended applications on the display.

[0029] According to an embodiment, the processor may also be configured to execute one or more instructions so as to: determine a plurality of recommended applications corresponding to one or more objects respectively based on similarities between the plurality of recommended applications and the one or more objects by using a learning model of one or more neural networks; and display the selected one or more applications on a display in response to an external input for selecting one or more applications from the plurality of recommended applications.

[0030] According to an embodiment, the processor may be further configured to execute one or more instructions to: set arrangement criteria for arranging one or more recommended applications on the display based on the image; and arrange the one or more recommended applications on the display based on the set arrangement criteria.

[0031] According to an embodiment, a learning model may be determined by learning the following method: extracting attributes of recommended applications from one or more objects in response to input of one or more objects to one or more neural networks, and determining one or more recommended applications based on the extracted attributes of the recommended applications.

[0032] According to an embodiment, the attribute of the recommended application may include at least one of the type, size, color, and shape of the application.

[0033] According to an embodiment, the processor may be further configured to execute one or more instructions to perform preprocessing to input the recognized one or more objects into the learning model, and the preprocessing may include at least one of a cropping operation and a deformation operation.

[0034] According to an embodiment, the image may include an image generated by receiving an external input for drawing an application configuration desired by the user, or a photo acquired by capturing an image including an application configuration desired by the user.

[0035] According to an embodiment, the processor may also be configured to execute one or more instructions to select one or more recommended applications corresponding to the identified one or more objects respectively from an application database (DB), and the application DB may include a list of applications installed on the electronic device and applications stored in an external server.

[0036] According to an embodiment, the processor may also be configured to execute one or more instructions so as to: when it is determined that the selected one or more recommended applications are applications that are not installed on the electronic device, receive the selected one or more recommended applications from an external server and install the received selected one or more recommended applications; and display the installed one or more recommended applications on the display so that the installed one or more recommended applications are run.

[0037] According to an embodiment, the processor may also be configured to execute one or more instructions to: identify text information from one or more objects; update one or more recommended applications corresponding to the one or more objects respectively to reflect the text information; and display the updated one or more recommended applications on the display.

[0038] According to an embodiment, a method for controlling an electronic device includes: identifying one or more objects from an image; determining one or more recommended applications corresponding to the one or more objects, respectively, by using a learning model of one or more neural networks; and displaying the determined one or more recommended applications on a display so that the determined one or more recommended applications run.

[0039] Public method

[0040] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings so that a person of ordinary skill in the art can easily perform the present disclosure. However, the present disclosure may be embodied in many different forms and is not limited to the embodiments of the present disclosure described herein. In addition, in order to clearly describe the present disclosure, parts not related to the description are omitted, and the same reference numerals are assigned to the same elements throughout the specification.

[0041] The terms used herein are general terms currently widely used in the art in consideration of the functions described in the present disclosure, but the terms may vary according to the intentions of those skilled in the art, precedents, or new technologies in the art. Therefore, the terms used herein should not be understood as simple names, but should be understood based on the meaning of the terms and the overall description of the present disclosure.

[0042] The terms used herein are used for the purpose of describing specific embodiments and are not intended to limit the scope of the present disclosure. The singular forms "a", "an", and "the" used herein are intended to also include the plural forms, unless the context clearly indicates otherwise. In addition, it should be understood that when a region is referred to as being "connected to" or "coupled to" another region, the region can be "directly connected or coupled to" another region or "electrically connected to" another region. It should be understood that when used in this article, terms such as "including", "comprising", and "having" specify the presence of the described elements, but do not exclude the presence or addition of one or more other elements.

[0043] The use of the term "the" and similar definite terms in the specification should be interpreted to cover both the singular and the plural. In addition, unless otherwise explicitly stated herein, the operations of all methods described herein can be performed in any suitable order. The present disclosure is not limited to the order of the operations described.

[0044] The appearances of the phrases "in some embodiments" or "in an embodiment" in various places in the specification are not necessarily referring to the same embodiment.

[0045] Some embodiments of the present disclosure can be represented by function block configuration and various processing operations. All or part of these function blocks can be implemented by any number of hardware and / or software components configured to perform specific functions. For example, the function blocks of the present disclosure can be implemented as one or more microprocessors, or can be implemented as circuit configurations for specific functions. In addition, for example, the function blocks of the present disclosure can be implemented with various programming or scripting languages. The function blocks can be implemented by algorithms executed on one or more processors. In addition, the present disclosure can adopt any traditional technology for electronic environment configuration, signal processing and / or data processing. The terms "mechanism", "element" and "configuration" are used in a broad sense and are not limited to mechanical or physical configurations.

[0046] In addition, the connecting lines or connecting members shown in the drawings are intended to represent exemplary functional connections and / or physical or logical connections between elements. It should be noted that many alternative or additional functional connections, physical connections or logical connections may exist in a practical device.

[0047] Hereinafter, the present disclosure will be described in detail with reference to the accompanying drawings.

[0048] Figure 1 is a diagram showing an example in which an electronic device displays a plurality of applications on a display according to an embodiment.

[0049] refer to Figure 1 , the electronic device 100 may be a home appliance including a display 230, but the embodiment is not limited thereto. For example, the electronic device 100 may be implemented as various devices such as a mobile phone, a tablet personal computer (PC), a digital camera, a video camera, a laptop computer, a desktop computer, an e-book terminal, a digital broadcast terminal, a personal digital assistant (PDA), a portable multimedia player (PMP), a navigation, an MP3 player, and a wearable device. In addition, the electronic device 100 may be a fixed or mobile electronic device, and may be a digital broadcast receiver capable of receiving digital broadcasts.

[0050] The term “user” used in the embodiments of the present disclosure refers to a person who controls a function or operation of the electronic device 100 , and may include an observer, an administrator, or an installation engineer.

[0051] The electronic device 100 according to the embodiment may display one or more applications on the display 230 so that the one or more applications are running. Figure 1 As shown, the electronic device 100 can display a calendar application, a weather application, and a notepad application on the display 230, and allow the user to use the application displayed on the display 230 as needed. In this case, the application can be used as a meaning including a theme indicating a style, a widget, and a user interface of the application, but the embodiment is not limited thereto. The user can select a desired application from a plurality of applications that can be installed and used in the electronic device 100, and the electronic device 100 can display the application selected by the user on the display 230. In addition, the user can change at least one of the position and size of one or more applications displayed on the display 230 based on the user's preference.

[0052] As the types of electronic devices 100 that allow users to use applications by displaying one or more applications on the display 230 increase, there is a need for a method that can easily identify the configuration of a user-desired application and provide the user-desired application in a user-desired form based on the identification result.

[0053] Figure 2is a block diagram showing a configuration of an electronic device according to an embodiment.

[0054] Figure 2 The electronic device 100a shown may be Figure 1 The embodiment of the electronic device 100 is shown. Figure 2 , the electronic device 100a according to the embodiment may include a memory 210, a processor 220, and a display 230. However, the electronic device 100a may be implemented by more elements than those shown, and the embodiment is not limited to the above example.

[0055] Hereinafter, elements of the electronic device 100 a will be sequentially described.

[0056] The memory 210 according to an embodiment may store a program for processing and control of the processor 220 , and may store data input to or output from the electronic device 100 a .

[0057] The memory 210 may include at least one selected from the following storage media: flash memory, hard disk, micro multimedia card type memory, card type memory (e.g., SD or XD memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, and optical disk.

[0058] According to an embodiment, the memory 210 may include a module including one or more instructions for the following operations: recognizing one or more objects from an image; determining one or more recommended applications corresponding to the one or more objects, respectively, by using a learning model of one or more neural networks; and displaying the determined one or more recommended applications on a display so that the determined one or more recommended applications are run.

[0059] The processor 220 may control the overall operation of the electronic device 100a. For example, the processor 220 may control the display 230 by executing one or more instructions stored in the memory 210, and may execute reference Figures 1 to 18 The functions of the electronic device 100a are described.

[0060] The processor 220 according to an embodiment may recognize one or more objects from an image.

[0061] The image may be an image showing a configuration of an application desired by the user. The image may include an image generated by receiving an external input for drawing a configuration of an application desired by the user, or a photo acquired by capturing an image including a configuration of an application desired by the user, but the embodiment is not limited thereto.

[0062] One or more objects recognized from the image may correspond to applications, respectively, and may indicate properties of the application desired by the user. For example, the user may desire a memo application that does not display lines on a yellow desktop or a schedule management application that can check to-do tasks in units of 24 hours, but the embodiment is not limited thereto.

[0063] The processor 220 according to an embodiment may analyze an image to recognize one or more objects based on a boundary line for dividing a specific area included in the image, a degree of color change in the image, a shape indicating a specific area included in the image, etc., but the embodiment is not limited thereto.

[0064] The processor 220 according to an embodiment may determine one or more recommended applications respectively corresponding to one or more objects by using a learning model of one or more neural networks.

[0065] A neural network may be a set of algorithms for learning a method of recognizing an object from a specific image input to the neural network based on artificial intelligence (AI). For example, a neural network may learn a method of recognizing an object from an image based on supervised learning using a specific image as an input value and unsupervised learning discovering a pattern for recognizing an object from an image by self-learning without separately learning the type of data required to recognize the object from the image. In addition, for example, a neural network may learn a method of recognizing an object from an image by using reinforcement learning using feedback on whether the result of recognizing an object according to the learning is correct.

[0066] For example, the processor 220 may utilize a learning model using one or more neural networks to determine one or more recommended applications corresponding to one or more objects, respectively, based on the similarity between one or more recommended applications and one or more objects. The processor 220 may extract attributes of the recommended applications from one or more objects, and determine one or more recommended applications based on the extracted attributes of the recommended applications. The attributes of the recommended applications may include at least one of the type, size, color, and shape of the application. For example, when the first object identified from the image includes a blue background and an icon indicating today's weather, the processor 220 may extract a weather application, a blue background, and an icon indicating the weather as attributes of the recommended application. The processor 220 may determine the recommended application based on the attributes of the recommended application (including at least one of the weather application, the blue background, and the icon indicating the weather).

[0067] The processor 220 according to an embodiment may select one or more recommended applications corresponding to the first object from an application database (DB). The application DB may include a list of applications installed on the electronic device 100a and applications stored in an external server. The application stored in the external server may refer to an application not installed on the electronic device 100a, and the electronic device 100a may receive the application from the external server and install the received application. For example, the electronic device 100a may receive an application not installed on the electronic device 100a from an external server through an application store, and install the received application, but the embodiment is not limited thereto.

[0068] The processor 220 according to an embodiment may display one or more recommended applications on the display 230 so that the one or more recommended applications are executed.

[0069] The processor 220 may set an arrangement standard for arranging one or more recommended applications on the display 230 based on the image. For example, the processor 220 may determine at least one of the size, position, and transparency of each object from the one or more objects recognized from the image. In addition, the processor 220 may arrange the recommended application corresponding to each object on the display 230 based on at least one of the size, position, and transparency of each object.

[0070] For example, when a first object and a second object are identified from an image, the first object may be located at the upper end of the image and have a size corresponding to 2 / 3 of the image, and the second object may be located at the lower end of the image and have a size corresponding to 1 / 3 of the image. In this case, the processor 220 may set the size and position of the recommended application as an arrangement standard for arranging the recommended applications corresponding to the first object and the second object, respectively. Therefore, the processor 220 may arrange the recommended application corresponding to the first object at the upper end of the display 230 in a size corresponding to 2 / 3 of the size of the display 230, and may arrange the recommended application corresponding to the second object at the lower end of the display 230 in a size corresponding to 1 / 3 of the size of the display 230.

[0071] In addition, the processor 220 according to the embodiment can identify text information from one or more objects identified according to the image. For example, the image may include a calendar for October 2017 drawn by a user, and text information indicating an event associated with the user may be written on the calendar. In this case, the processor 220 may identify the calendar for October 2017 as an object, and identify text information indicating an event from the identified object. In addition, the processor 220 may update the recommended application (e.g., A calendar application) corresponding to the identified object (e.g., the calendar for October 2017) to reflect the identified text information. For example, when "birthday" is written in an area corresponding to October 17 in the October 2017 calendar included in the image, the processor 220 may update the A calendar application by adding a birthday event to October 17 of the A calendar application. In addition, the processor 220 may display the updated A calendar application with the birthday event added on the display 230.

[0072] Therefore, the processor 220 according to an embodiment may more easily determine the configuration of an application desired by a user, and provide a recommended application corresponding to the application desired by the user in a form desired by the user.

[0073] The display 230 according to the embodiment may display one or more recommended applications corresponding to one or more objects respectively identified from the image. In addition, the display 230 may display one or more recommended applications based on the attributes of the recommended applications extracted from the one or more objects. Therefore, the number of recommended applications displayed on the display 230 may be changed according to the number of objects identified from the image. The size and position of the recommended applications displayed on the display 230 may be changed according to the size and position of the objects identified from the image.

[0074] When the display 230 is implemented as a touch screen, the display 230 can be used as an input device as well as an output device. For example, the display 230 may include at least one of a liquid crystal display (LCD), a thin film transistor-liquid crystal display (TFT-LCD), an organic light emitting diode (OLED), a flexible display, a three-dimensional (3D) display, and an electrophoretic display. According to an embodiment, the electronic device 100a may include two or more displays 230.

[0075] Figure 3 is a block diagram showing a configuration of an electronic device according to another embodiment.

[0076] like Figure 3As shown, the electronic device 100 b may include a memory 210 , a processor 220 , and a display 230 , and may further include a tuner 140 , a communicator 150 , a detector 160 , an inputter / outputter 170 , a video processor 180 , an audio processor 115 , and a user inputter 190 .

[0077] Because the memory 210, the processor 220 and the display 230 are similar to those in the above reference Figure 2 The memory 210, processor 220 and display 230 described above are the same, so in Figure 3 Its redundant description will be omitted.

[0078] The tuner 140 can tune only the frequency of a channel to be received by the electronic device 100b among many propagation components by performing amplification, mixing, resonance, etc. on a broadcast signal received in a wired or wireless manner. The broadcast signal may include audio, video, and additional information (e.g., an electronic program guide (EPG)).

[0079] The broadcast signal received by the tuner 140 may be decoded (eg, subjected to audio decoding, video decoding, or additional information decoding) and divided into audio, video, and / or additional information. The divided audio, video, and / or additional information may be stored in the memory 210 under the control of the processor 220.

[0080] The tuner 140 of the electronic device 100b may be single or multiple. The tuner 140 may be implemented as an all-in-one with the electronic device 100b, or implemented as a separate device including a tuner electrically connected to the electronic device 100b (e.g., a set-top box (not shown) or a tuner (not shown) connected to the input / output device 170).

[0081] The communicator 150 may connect the electronic device 100b to an external device (eg, an audio device) under the control of the processor 220. The processor 220 may transmit and receive content to and from an external device connected through the communicator 150, download an application from the external device, or perform web browsing.

[0082] According to the performance and configuration of the electronic device 100b, the communicator 150 may include one of a wireless local area network (WLAN) 151, Bluetooth 152, and a wired Ethernet 153. In addition, the communicator 150 may include a combination of the WLAN 151, Bluetooth 152, and the wired Ethernet 153. Under the control of the processor 220, the communicator 150 may receive a control signal of the controller 101. The control signal may be implemented as a Bluetooth type, a radio frequency (RF) signal type, or a Wi-Fi type.

[0083] In addition to Bluetooth, the communicator 150 may also include other short-range communications such as near field communication (NFC) (not shown) and Bluetooth low energy (BLE) (not shown).

[0084] The communicator 150 according to an embodiment may receive a learning model using one or more neural networks from an external server. In addition, the communicator 150 may receive an image for recognizing one or more objects from an external device.

[0085] The detector 160 may detect a user voice, a user image, or a user interaction, and may include a microphone 161 , a camera 162 , and an optical receiver 163 .

[0086] The microphone 161 may receive a voice spoken by a user, convert the received voice into an electric signal, and output the electric signal to the processor 220.

[0087] The camera 162 may receive an image (e.g., continuous frames) corresponding to a user motion including a gesture within the camera recognition range. For example, the camera 162 may capture an image including a configuration of an application desired by the user. For example, the camera 162 may capture a photo of a picture depicting a configuration of an application desired by the user and store the captured photo, and the processor 220 may provide one or more recommended applications corresponding to one or more objects recognized from the photo captured by the camera 162, respectively.

[0088] When the electronic device 100b is controlled by a control device, the optical receiver 163 may receive an optical signal (including a control signal) received from the control device. The optical receiver 163 may receive an optical signal corresponding to a user input (e.g., touch, press, touch gesture, voice, or motion) from the control device. Under the control of the processor 220, the control signal may be extracted from the received optical signal.

[0089] When the electronic device 100 b is controlled by the control device, the optical receiver 163 according to an embodiment may receive an external input for drawing an image indicating an application configuration desired by a user.

[0090] Under the control of the processor 220, the input / output device 170 may receive video (e.g., moving images, etc.), audio (e.g., voice, music, etc.), and additional information (e.g., electronic program guide (EPG), etc.) from the outside of the electronic device 100b. The input / output device 170 may include one of a high-definition multimedia interface (HDMI) port 171, a component jack 172, a PC port 173, and a universal serial bus (USB) port 174. The input / output device 170 may include a combination of the HDMI port 171, the component jack 172, the PC port 173, and the USB port 174.

[0091] The memory 210 according to an embodiment may store a program for processing and control of the processor 220 and may store data input to or output from the electronic device 100 b .

[0092] The programs stored in the memory 210 may be classified into a plurality of modules according to their functions. For example, the programs may be classified into a learning module 211 and the like.

[0093] The learning module 211 may include a learning module determined by learning the following method: a method of extracting attributes of recommended applications from one or more objects in response to input of one or more objects to one or more neural networks and a method of determining one or more recommended applications based on the extracted attributes of the recommended applications. The learning model may be received from an external server, and the received learning model may be stored in the learning module 211.

[0094] The processor 220 can control the overall operation of the electronic device 100b and the signal flow between the internal elements of the electronic device 100b, and perform data processing functions. When receiving user input or satisfying stored preset conditions, the processor 220 can execute the operating system (OS) and various applications stored in the memory 210.

[0095] According to an embodiment, the processor 220 can execute one or more instructions stored in the memory 210 to recognize one or more objects from an image, determine one or more recommended applications corresponding to the one or more objects respectively by using a learning model of one or more neural networks, and display the determined one or more recommended applications on a display so that the determined one or more recommended applications are run.

[0096] The video processor 180 may process image data to be displayed by the display 230 , and may perform various image processing operations, such as decoding, rendering, scaling, noise filtering, frame rate conversion, and resolution conversion, on the image data.

[0097] The display 230 may display a video included in a broadcast signal received through the tuner 140 on a screen under the control of the processor 220. In addition, the display 230 may display content (e.g., video) input through the communicator 150 or the inputter / outputter 170. The display 230 may output an image stored in the memory 210 under the control of the processor 220.

[0098] The audio processor 115 may process the audio data and may perform various processes such as decoding, amplification, or noise filtering on the audio data.

[0099] The audio outputter 125 may output audio included in a broadcast signal received through the tuner 140, audio input through the communicator 150 or the inputter / outputter 170, and audio stored in the memory 210 under the control of the processor 220. The audio outputter 125 may include at least one of a speaker 126, a headphone output terminal 127, or a Sony / Philips Digital Interface (S / PDIF) output terminal 128.

[0100] The user input device 190 refers to a device that allows a user to input data for controlling the electronic device 100b. For example, the user input device 190 may include a keyboard, a dome switch, a touch pad, a roller, and a roller switch, but the embodiment is not limited thereto. In addition, the user input device 190 may be an element of a control device or an element of the electronic device 100b.

[0101] The user input device 190 according to an embodiment may receive an input for drawing a configuration of an application desired by the user. For example, when the user input device 190 is implemented as a touch screen or a touch pad, the user input device 190 may receive a drawing input using a user's finger or an electronic pen. In addition, when a plurality of recommended applications corresponding to an object recognized from an image is determined, the user input device 190 may receive an input for selecting a recommended application to be displayed on the display 230.

[0102] at the same time, Figure 2 and Figure 3 The block diagram of the electronic device 100a and 100b shown is a block diagram of an embodiment. Each element of the block diagram can be integrated, added or omitted according to the specification of the electronic device 100a and 100b actually implemented. For example, when necessary, two or more elements can be integrated into one element, or one element can be subdivided into two or more elements. In addition, the function performed by each block is provided for describing an embodiment, and specific operation or its equipment does not limit the scope of the present disclosure.

[0103] Figure 4 are diagrams showing various examples of images according to the embodiment.

[0104] The electronic device 100 according to the embodiment can recognize one or more objects from the image. The image is an image indicating the configuration of the application desired by the user, and may include a picture drawn by the user or a captured photo, but the embodiment is not limited thereto. For example, the electronic device 100 may receive an external input for drawing a picture indicating the configuration of the application desired by the user through a display 230 implemented as a touch screen or a touch pad, and may acquire an image by generating a picture corresponding to the received external input. In addition, the electronic device 100 may acquire a photo acquired by capturing a picture indicating the configuration of the application desired by the user using a camera 162 as an image. In addition, the electronic device 100 may acquire a photo acquired by capturing an application displayed on a display of an external device using a camera 162, or a screen acquired by capturing an application displayed on a display of an external device using a camera 162 as an image. According to an embodiment, the electronic device 100 may receive an image from an external device. For example, the electronic device 100 may receive an image generated by another device of the user by using wireless communication. Therefore, the electronic device 100 may acquire various types of images and recognize one or more objects from the acquired images.

[0105] refer to Figure 4 , the image may include a photo 401 obtained by capturing a picture drawn by a user using the camera 162, an image 402 indicating an application displayed on a display of an external device, and a picture 403 drawn by the user through a display implemented as a touch screen. For example, Figure 4 The photo 401 shown in FIG. 4 is obtained by capturing a picture drawn by a user, and the photo 401 may include a configuration of an application desired by the user.

[0106] In addition, the user may want to use the same application that is displayed on another device. Figure 4 In the image 402 shown in , the image used to determine the recommended application may include an image obtained by capturing a screen displaying an application desired by the user, or an image obtained by capturing a photo of a screen displaying the desired application, but the embodiment is not limited thereto.

[0107] In addition, when the display 230 of the electronic device 100 is implemented as a touch screen, the electronic device 100 can receive an external input for drawing a configuration of a desired application through the display 230 implemented as a touch screen, and can generate a picture corresponding to the received external input. Figure 4 In the illustrated image 403, the user may want to display a to-do list display application and a music playback application on the display of the electronic device 100. Therefore, the user can draw a picture (e.g., Figure 4The image 403 shown in the figure) and the picture drawn by the user can be stored as an image.

[0108] Figure 5 and Figure 6 is a diagram for describing a process of performing pre-processing on one or more objects recognized from an image according to an embodiment.

[0109] The electronic device 100 according to the embodiment may recognize one or more objects from an image, and perform preprocessing to input the recognized one or more objects into one or more neural networks.

[0110] For example, refer to Figure 5 , the image 500 may be an image including a configuration of an application desired by the user. The image 500 may be a picture drawn by the user. The image 500 may include a first object 501 indicating a calendar application at the upper end of the image 500, a second object 502 indicating a memo application at the lower right end of the image 500, and a third object 503 indicating a weather application at the lower left end of the image 500.

[0111] The electronic device 100 according to the embodiment can determine the recommended applications corresponding to the first to third objects 501, 502 and 503 respectively identified from the image by using a learning model of one or more neural networks. In this case, the electronic device 100 can perform preprocessing to input the first to third objects 501, 502 and 503 into the learning model. In addition, preprocessing can be performed so that the learning model learns a method of extracting the attributes of the recommended applications from one or more objects and determining one or more recommended applications based on the extracted attributes of the recommended applications. For example, the learning model can be determined by learning the following method: extracting the attributes of the recommended applications from one or more objects in response to the input of one or more objects to one or more neural networks, and determining one or more recommended applications based on the extracted attributes of the recommended applications. In this case, the one or more objects input to the one or more neural networks may be objects on which preprocessing has been performed.

[0112] For example, Figure 5 As shown, the electronic device 100 may perform a cropping operation on the first to third objects 501, 502, and 503 included in the image. The electronic device 100 may determine one or more recommended applications corresponding to the first to third objects 511, 512, and 513, respectively, in response to inputting the first to third objects 511, 512, and 513, on which the cropping operation has been performed, into the learning model.

[0113] According to an embodiment, the method of cropping the first to third objects 501, 502, and 503 may be changed. For example, the electronic device 100 may crop the first to third objects 501, 502, and 503 by using at least one of an operation of analyzing a color change in the image 500, an operation of detecting a boundary line indicating a specific area in the image 500, and an operation of extracting a shape of a specific area included in the image 500.

[0114] In addition, reference Figure 6 When the image 520 is a photo captured by a user, the shape of one or more objects recognized from the photo may be changed according to the capture angle, capture environment, etc. For example, Figure 6 As shown, when the image 520 is a photo captured in a tilted state, the plurality of objects 531, 532, and 533 identified from the image 520 may have deformed shapes. In this case, the electronic device 100 may correct the deformed shapes of the objects 531, 532, and 533 by performing a deformation operation on the objects 531, 532, and 533 having the deformed shapes. The deformation operation may refer to an operation of transforming the shape of an image according to a specific rule, and various deformation algorithms may be used according to embodiments.

[0115] The electronic device 100 according to the embodiment can obtain a plurality of shape-corrected objects 541, 542, and 543 by performing a deformation operation on the objects 531, 532, and 533 having deformed shapes. In addition, the electronic device 100 can determine one or more recommended applications corresponding to the shape-corrected objects 541, 542, and 543, respectively, by using a learning model of one or more neural networks.

[0116] Figure 7 is a diagram for describing a process of determining one or more recommended applications respectively corresponding to one or more objects recognized from an image, performed by an electronic device according to an embodiment.

[0117] The electronic device 100 according to the embodiment may determine one or more recommended applications corresponding to one or more objects recognized from the image, respectively, by using a learning model of one or more neural networks. Figure 7 , the electronic device 100 can identify a first object 601 indicating a calendar application, a second object 602 indicating a memo application, and a third object 603 indicating a weather application from an image. The electronic device 100 can determine one or more recommended applications corresponding to the first to third objects 601, 602, and 603, respectively, by using a learning model 610 of one or more neural networks. As described above, preprocessing including at least one of a cropping operation and a deformation operation can be performed on the first to third objects 601, 602, and 603 input to the learning model 610.

[0118] The electronic device 100 according to the embodiment may determine one or more recommended applications corresponding to the first to third objects 601, 602, and 603, respectively, according to similarities between the one or more recommended applications and the first to third objects 601, 602, and 603. The electronic device 100 may extract attributes of the recommended applications from the first to third objects 601, 602, and 603, and may determine one or more recommended applications based on the extracted attributes of the recommended applications.

[0119] For example, Figure 7 As shown, the electronic device 100 can extract at least one of the type of application (e.g., calendar application), the shape of the application (e.g., a grid-type calendar), and the color of the application (e.g., white background and black letters) as the attribute of the recommended application corresponding to the first object 601. In addition, the electronic device 100 can extract the type of application (e.g., memo application) and the color of the application (yellow background) as the attributes of the recommended application corresponding to the second object 602, and can extract the type of application (e.g., weather application), the type of application (a picture indicating today's weather), and the color of the application (e.g., blue background) as the attributes of the recommended application corresponding to the third object 603.

[0120] The electronic device 100 according to the embodiment may determine one or more recommended applications based on the extracted attributes of the recommended applications. For example, the electronic device 100 may select one or more recommended applications from the application DB based on the extracted attributes of the recommended applications. As described above, the application DB may include a list of applications installed on the electronic device 100 and applications not installed on the electronic device 100. The electronic device 100 may receive applications not installed on the electronic device 100 from an external server through an application store, and install the received applications.

[0121] The electronic device 100 according to the embodiment can calculate the scores of multiple applications selected from the application DB according to the degree of matching with the extracted attributes of the recommended application. For example, as the degree of matching with the extracted attributes of the recommended application is higher, the calculated score can be further increased. The electronic device 100 can give priority to recommending applications with higher calculated scores. Therefore, the electronic device 100 can determine a preset number of applications as multiple recommended applications in descending order of scores, and can display the recommended applications on the display 230 to allow the user to select one of the recommended applications.

[0122] For example, the electronic device 100 may determine the A calendar application 621 and the B calendar application 622 as the recommended applications corresponding to the first object 601 based on the attributes of the recommended applications extracted from the first object 601 (e.g., calendar application, white background and black letters, grid-shaped calendar, etc.). In addition, the electronic device 100 may determine the C memo application 631 and the D memo application 632 as the recommended applications corresponding to the second object 602 based on the attributes of the recommended applications extracted from the second object 602 (e.g., memo application, yellow background, etc.), and may determine the E weather application 641 and the F weather application 642 as the recommended applications corresponding to the third object 603 based on the attributes of the recommended applications extracted from the third object 603 (e.g., weather application, picture indicating today's weather, blue background, etc.).

[0123] The electronic device 100 according to the embodiment can display the recommended applications corresponding to the first to third objects 601, 602 and 603 respectively on the display 230, and allow the user to select a desired application from the recommended applications corresponding to the first to third objects respectively. Therefore, the electronic device 100 can more accurately and easily recommend applications that meet the conditions desired by the user.

[0124] Figure 8 is a diagram illustrating an example of a neural network that learns a method of determining a recommended application corresponding to an object according to an embodiment.

[0125] refer to Figure 8 , one or more neural networks 700 may learn a method of determining a recommended application corresponding to a learning object by using a plurality of learning objects 701 to 705 as input values. The learning objects 701 to 705 may be extracted from an image indicating a configuration of an application desired by a user, and may be in the form of a picture drawn by the user or a captured photo. For example, the objects 701 to 705 may be objects extracted from a picture drawn by the user on paper, such as Figure 8 , and may be objects generated by using specific picture editing software, such as the second object 702 and the fifth object 705. In addition, the objects 701 to 705 may be objects generated by capturing an application displayed on a display of another device, such as the third object 703, and may be objects generated by receiving a drawing input from a display of the electronic device 100 implemented as a touch screen, such as the fourth object 704, but the embodiment is not limited thereto.

[0126] The one or more neural networks 700 may learn a method of extracting attributes of recommended applications from the learning objects 701 to 705. For example, the attributes of the recommended applications may include at least one of the type, size, color, and shape of the application, and the extracted attributes of the recommended applications may be indicated by keywords, but the embodiment is not limited thereto. In addition, the one or more neural networks 700 may learn a method of determining one or more recommended applications based on the extracted attributes of the recommended applications, and may generate a learning model 710 based on the learning results.

[0127] According to an embodiment, an operation of learning a method of determining one or more recommended applications corresponding to an object by using one or more neural networks 700 may be performed in advance. In addition, when a learning object input to one or more neural networks 700 is added, the learning model 710 may be updated.

[0128] In addition, according to an embodiment, the operation of learning a method for determining one or more recommended applications corresponding to an object by using one or more neural networks 700 may be performed by the electronic device 100 and may be performed by an external server. For example, the operation of learning a method for identifying an object corresponding to an object by using one or more neural networks 700 may require a relatively complex amount of calculation. Therefore, the external server may perform the learning operation, and the electronic device 100 may receive the learning model 710 from the external server, thereby reducing the amount of calculation to be performed in the electronic device 100. The electronic device 100 may receive the learning model 710 from the external server in advance, store the received learning model 710 in a memory, and determine one or more recommended applications corresponding to the object by using the stored learning model 710.

[0129] Fig. 9 and Fig.10 is a diagram for describing a process of selecting an application to be displayed on a display from among a plurality of recommended applications according to an embodiment.

[0130] The electronic device 100 according to an embodiment may determine a plurality of recommended applications respectively corresponding to one or more objects extracted from an image, and allow a user to select one recommended application from among the recommended applications.

[0131] For example, refer to Figure 8, the electronic device 100 may determine A calendar application 801 and B calendar application 802 as recommended applications corresponding to the first object 601 based on the attributes of the recommended applications extracted from the first object 601 according to image recognition (e.g., calendar application, white background and black letters, grid-shaped calendar, etc.). In addition, the electronic device 100 may determine C memo application 811 and D memo application 812 as recommended applications corresponding to the second object 602 based on the attributes of the recommended applications extracted from the second object 602 (e.g., memo application, yellow background, etc.), and may determine E weather application 821 and F weather application 822 as recommended applications corresponding to the third object 603 based on the attributes of the recommended applications extracted from the third object 603 (e.g., weather application, picture indicating today's weather, blue background, etc.).

[0132] The electronic device 100 according to the embodiment includes an A calendar application 801, a B calendar application 802, a C memo application 811, a D memo application 812, an E weather application 821, and an F weather application 822 on the display 230, and may receive an input of selecting some of the displayed recommended applications from the user. Fig. 9 As shown, when an input of selecting B calendar application 802 , C memo application 811 , and E weather application 821 is received from the user, the electronic device 100 may display the selected applications on the display 230 .

[0133] When displaying a recommended application selected from a plurality of recommended applications on the display 230, the electronic device 100 according to an embodiment may set an arrangement standard for arranging the recommended applications based on an image, and may arrange the selected recommended application on the display 230 based on the set arrangement standard. Fig.10 , the electronic device 100 may recognize at least one of the positions and sizes of the first object 831 indicating the calendar application, the second object 832 indicating the weather application, and the third object 833 indicating the memo application based on the image 830. In addition, the electronic device 100 may set an arrangement standard for arranging the B calendar application 841 corresponding to the first object 831, the C memo application 842 corresponding to the third object 833, and the F weather application 843 corresponding to the second object 832 on the display 230. For example, the electronic device 100 may arrange the B calendar application 841 at the upper end with a size corresponding to 2 / 3 of the size of the display 230, may arrange the C memo application 842 at the lower left end with a size corresponding to 1 / 3 of the size of the display 230, and may arrange the F weather application 843 at the lower right end with a size corresponding to 1 / 3 of the size of the display 230. In addition, the electronic device 100 may arrange the B calendar application 841, the C memo application 842, and the F weather application 843 on the display 230 according to the set arrangement standard.

[0134] Fig.11 is a diagram illustrating an example in which an electronic device receives a recommended application from an external server and installs the received recommended application according to an embodiment.

[0135] When an application selected from among a plurality of recommended applications is not installed on the electronic device 100 , the electronic device 100 according to an embodiment may receive the selected application from an external server and install the received application.

[0136] For example, refer to Fig.11 , the electronic device 100 may determine the C memo application 811 and the D memo application 812 as recommended applications, and may receive an input of selecting the C memo application 811 from the user. When the C memo application 811 is an application that is not installed on the electronic device 100, the electronic device 100 may display a message 910 to select whether to download the C memo application 811 from the application store. When a user input selecting to download the C memo application 811 is received, the electronic device 100 may access the application store and receive and install the C memo application 811. In addition, the electronic device 100 may set an arrangement standard for arranging the C memo application 811 based on an image, and may arrange the C memo application 811 in the area 901 included in the screen 900 of the display 230 based on the set arrangement standard. Therefore, even when the recommended application is not installed on the electronic device 100, the electronic device 100 may more conveniently provide a recommended application that meets the conditions desired by the user.

[0137] Fig.12 is a diagram illustrating an example in which an electronic device updates a recommended application to reflect text information recognized from an image according to an embodiment.

[0138] The electronic device 100 according to an embodiment may recognize text information from an object and update a recommended application corresponding to the object to add the text information.

[0139] For example, refer to Fig.12 , object 1001 may be a picture of a calendar for November 2017 drawn on it, and text information indicating a specific event (e.g., birthday, dinner, etc.) may be written on the calendar for November 2017. The electronic device 100 may recognize the text information from the object 1001, and update the recommended application corresponding to the object 1001 to reflect the recognized text information. According to an embodiment, when a plurality of recommended applications corresponding to the object 1001 are determined, the electronic device 100 may receive an external input for selecting a recommended application to be displayed on the display 230 from the recommended applications, and update the selected recommended application to reflect the recognized text information. For example, Fig.12As shown, the electronic device 100 can identify "Birthday" written on November 16 and "Dinner" written on November 29 as text information from the object 1001. In addition, the electronic device 100 can determine the K calendar application 1010 as a recommended application corresponding to the object 1001, and can update the K calendar application 1010 by adding the recognized text information ("Birthday" and "Dinner") to the November 2017 calendar of the K calendar application 1010. Therefore, the electronic device 100 can display the K calendar application 1010 indicating the November 2017 calendar on the display 230. In the displayed K calendar application 1010, the electronic device 100 can display "Birthday" in the area corresponding to November 16, and can display "Dinner" in the area corresponding to November 29.

[0140] According to an embodiment, in the object 1001, a green highlight may be displayed in an area corresponding to November 16, and a red highlight may be displayed in an area corresponding to November 29. In this case, the electronic device 100 may display a green highlight and a red highlight in areas corresponding to November 16 and November 29, respectively, in the K calendar application 1010 displayed on the display 230. When the K calendar application 1010 does not provide the green highlight and the red highlight, the electronic device 100 may display the highlight with different colors or may display the highlight in a different form, but the embodiment is not limited thereto.

[0141] Fig.13 is a flowchart of a method of operating an electronic device according to an embodiment.

[0142] In operation S1100, the electronic device 100 recognizes one or more objects from an image.

[0143] The one or more objects recognized from the image may correspond to applications, respectively, and may indicate attributes of the application desired by the user. The electronic device 100 according to the embodiment may analyze the image to recognize the one or more objects based on a boundary line for dividing a specific area included in the image, a degree of color change in the image, a shape indicating a specific area included in the image, etc., but the embodiment is not limited thereto.

[0144] In operation S1110, the electronic device 100 determines one or more recommended applications respectively corresponding to one or more objects by using a learning model of one or more neural networks.

[0145] For example, the electronic device 100 may utilize a learning model using one or more neural networks to determine one or more recommended applications corresponding to one or more objects, respectively, based on the similarity between the one or more recommended applications and the one or more objects. In addition, the electronic device 100 may extract attributes of the recommended applications from the one or more objects, and determine one or more recommended applications based on the extracted attributes of the recommended applications. The attributes of the recommended applications may include at least one of the type, size, color, and shape of the application.

[0146] In addition, the electronic device 100 according to the embodiment may select one or more recommended applications corresponding to the object from the application DB. The application DB may include a list of applications installed on the electronic device 100 and applications stored in an external server. The application stored in the external server may refer to an application that is not installed on the electronic device 100a, and the electronic device 100a may receive the application from the external server and install the received application. For example, the electronic device 100a may receive an application that is not installed on the electronic device 100a from an external server through an application store and install the received application, but the embodiment is not limited thereto.

[0147] In operation S1120, the electronic device 100 according to an embodiment may display one or more recommended applications on the display 230 so that the one or more recommended applications are executed.

[0148] The electronic device 100 according to the embodiment may set an arrangement standard for arranging one or more recommended applications on the display 230 based on the image. For example, the electronic device 100 may determine at least one of the size, position, and transparency of each object from one or more objects recognized from the image. In addition, the electronic device 100 may arrange the recommended application corresponding to each object on the display 230 based on at least one of the size, position, and transparency of each object.

[0149] Fig.14 is a block diagram showing a configuration of a processor according to an embodiment.

[0150] refer to Fig.14 , the processor 220 according to an embodiment may include a data trainer 1210 and a data identifier 1220 .

[0151] The data trainer 1210 may learn a standard for determining a recommended application corresponding to an object recognized from an image. The data trainer 1210 may learn a standard for determining which information about an object is used to determine a recommended application corresponding to an object. In addition, the data trainer 1210 may learn a standard for a method of determining a recommended application by using information about an object. In addition, the data trainer 1210 may learn a standard for extracting attributes of a recommended application from an object recognized from an image, and may learn a standard for determining one or more recommended applications corresponding to an object based on the extracted attributes of the recommended application.

[0152] The data trainer 1210 may acquire an object to be used for learning, and may apply the acquired object to a data recognition model to be described below, so as to learn a criterion for determining a recommended application corresponding to the object.

[0153] The data identifier 1220 may determine one or more recommended applications corresponding to the object and output a determination result. The data identifier 1220 may determine one or more recommended applications corresponding to the object using a learned data recognition model. The data identifier 1220 may acquire an object according to a preset standard acquired through learning, and may use the data recognition model by using the acquired object as an input value. In addition, the data identifier 1220 may determine one or more recommended applications corresponding to the object using the data recognition model. In addition, the result value output from the data recognition model by using the object as an input value may be used to update the data recognition model.

[0154] At least one of the data trainer 1210 and the data identifier 1220 may be manufactured in the form of at least one hardware chip and may be installed on an electronic device. For example, at least one of the data trainer 1210 and the data identifier 1220 may be manufactured in the form of a dedicated hardware chip for AI, or may be manufactured as part of an existing general-purpose processor (e.g., a central processing unit (CPU) or an application processor) or a graphics-specific processor (e.g., a graphics processing unit (GPU)), and may be installed on the above-mentioned various electronic devices.

[0155] In this case, the data trainer 1210 and the data identifier 1220 may be installed on a single device or may be installed on separate devices, respectively. For example, one of the data trainer 1210 and the data identifier 1220 may be included in a single electronic device 110, and the other may be included in a server. In addition, the model information constructed by the data trainer 1210 may be provided to the data identifier 1220 through wired or wireless communication, and the data input to the data identifier 1220 may be provided to the data trainer 1210 as additional training data.

[0156] At least one of the data trainer 1210 and the data identifier 1220 may be implemented as at least one software module. When at least one of the data trainer 1210 or the data identifier 1220 is implemented as a software module (or a program module including instructions), the software module may be stored in a non-transitory computer-readable medium. In addition, in this case, at least one software module may be provided by an operating system (OS) or may be provided by a specific application. Alternatively, a portion of at least one software module may be provided by the OS, and the remaining portion thereof may be provided by a specific application.

[0157] Fig.15 is a block diagram showing a configuration of a data trainer according to an embodiment.

[0158] refer to Fig.15 According to an embodiment, the data trainer 1210 may include a data acquirer 1211, a preprocessor 1212, a training data selector 1213, a model learner 1214 and a model evaluator 1215.

[0159] The data acquirer 1211 may acquire data required for learning to determine one or more recommended applications corresponding to an object. The data acquirer 1211 may acquire data from an external server (eg, a social network server, a cloud server, or a content providing server).

[0160] The data acquirer 1211 may acquire objects required for learning to determine one or more recommended applications corresponding to the objects. For example, the data acquirer 1211 may acquire objects from at least one external device connected to the electronic device 100 via a network, and may acquire images by extracting one or more objects from a picture drawn by a user or a captured photo.

[0161] The preprocessor 1212 may preprocess the acquired data for use in learning to determine one or more recommended applications corresponding to the object. The preprocessor 1212 may process the acquired data in a preset format so that the model learner 1214 described below may use the acquired data to learn to determine one or more recommended applications corresponding to the object. For example, the preprocessor 1212 may process the acquired object in a preset format by performing preprocessing including at least one of a cropping operation and a deformation operation on the acquired object.

[0162] The training data selector 1213 may select data required for learning from a plurality of preprocessed data. The selected data may be provided to the model learner 1214. The training data selector 1213 may select data required for learning from a plurality of preprocessed data according to a preset criterion for determining one or more recommended applications corresponding to an object. In addition, the training data selector 1213 may select data according to a preset criterion through learning performed by the model learner 1214 to be described below.

[0163] The model learner 1214 may learn criteria about which training data will be used to determine one or more recommended applications corresponding to an object. For example, the model learner 1214 may learn criteria for extracting attributes of the recommended application from the object, and may learn at least one of the type, color, size, and shape of the recommended application.

[0164] In addition, the model learner 1214 can use the training data to train a data recognition model for determining one or more recommended applications corresponding to the object. In this case, the data recognition model can be a pre-built model. For example, the data recognition model can be a model pre-built by receiving basic training data (e.g., sample images, etc.).

[0165] The data recognition model may be constructed by considering the application field of the recognition model, the purpose of learning, the computer performance of the device, etc. The data recognition model may be, for example, a model based on a neural network. For example, a model such as a deep neural network (DNN), a recurrent neural network (RNN), or a bidirectional recurrent deep neural network (BRDNN) may be used as a data recognition model, but the embodiment is not limited thereto.

[0166] According to various embodiments, when there are multiple pre-built recognition models, the model learner 1214 may determine a data recognition model with a large correlation between the input training data and the basic training data as a data recognition model to be learned. In this case, the basic training data may be pre-classified for each data type, and a recognition model may be pre-built for each data type. For example, the basic training data may be pre-classified based on various criteria (e.g., the region where the training data is generated, the time when the training data is generated, the size of the training data, the type of the training data, the creator of the training data, and the type of the object in the training data).

[0167] In addition, the model learner 1214 can learn the data recognition model by using a learning algorithm including, for example, error back propagation or gradient descent.

[0168] In addition, the model learner 1214 can learn the data recognition model by, for example, supervised learning using the training data as an input value. In addition, for example, the model learner 1214 can self-learn the type of data required for determining the user condition without separate supervision, thereby training the data recognition model by unsupervised learning that finds a standard for determining the user condition. In addition, the model learner 1214 can learn the data recognition model by, for example, reinforcement learning using feedback on whether the result of determining the user condition according to the training is correct.

[0169] In addition, when the data recognition model is learned, the model learner 1214 can store the learned data recognition model. In this case, the model learner 1214 can store the learned data recognition model in the memory of the device including the data identifier 1220. Alternatively, the model learner 1214 can store the learned data recognition model in the memory of the device including the data identifier 1220 to be described below. Alternatively, the model learner 1214 can store the learned data recognition model in the memory of a server connected to the electronic device 100 via a wired or wireless network.

[0170] In this case, the memory storing the learned data recognition model may also store, for example, commands or data associated with at least one other element of the device. In addition, the memory may store software and / or programs. The program may include, for example, a kernel, middleware, an application programming interface (API), and / or an application program (or "application").

[0171] When the evaluation data is input to the data recognition model and the recognition result output from the evaluation data does not meet a specific standard, the model evaluator 1215 may cause the model learner 1214 to learn again. In this case, the evaluation data may be data preset for evaluating the data recognition model.

[0172] For example, when the number or ratio of the evaluation data with inaccurate analysis results among the analysis results of the learned data recognition model for the evaluation data exceeds a preset threshold, the model evaluator 1215 may evaluate that the specific standard is not met. For example, in the case where the specific standard is defined as a ratio of 2%, when the learned data recognition model outputs incorrect recognition results for more than 20 evaluation data among a total of 1000 evaluation data, the model evaluator 1215 may evaluate that the learned data recognition model is inappropriate.

[0173] When there are multiple learned data recognition models, the model evaluator 1215 can evaluate whether each learned data recognition model meets a specific criterion and determine the model that meets the specific criterion as the final data recognition model. In this case, when there are multiple models that meet the specific criterion, the model evaluator 1215 can determine any one model or a specific number of models preset in descending order of evaluation scores as the final data recognition model.

[0174] Meanwhile, at least one of the data acquirer 1211, the preprocessor 1212, the training data selector 1213, the model learner 1214, and the model evaluator 1215 in the data trainer 1210 may be manufactured in the form of a hardware chip and installed on an electronic device. For example, at least one of the data acquirer 1211, the preprocessor 1212, the training data selector 1213, the model learner 1214, and the model evaluator 1215 may be manufactured in the form of a dedicated hardware chip for AI, or may be manufactured as part of an existing general-purpose processor (e.g., a CPU or an application processor) or a graphics-specific processor (e.g., a GPU) and may be installed on the above-mentioned various electronic devices.

[0175] In addition, the data acquirer 1211, the preprocessor 1212, the training data selector 1213, the model learner 1214, and the model evaluator 1215 may be installed on a single device or may be installed on separate devices, respectively. For example, some of the data acquirer 1211, the preprocessor 1212, the training data selector 1213, the model learner 1214, and the model evaluator 1215 may be included in the electronic device 100, while others may be included in the server.

[0176] In addition, at least one of the data acquirer 1211, the preprocessor 1212, the training data selector 1213, the model learner 1214, and the model evaluator 1215 may be implemented as a software module. When at least one of the data acquirer 1211, the preprocessor 1212, the training data selector 1213, the model learner 1214, and the model evaluator 1215 is implemented as a software module (or a program module including instructions), the software module may be stored in a non-transitory computer-readable medium. In addition, in this case, at least one software module may be provided by the OS or may be provided by a specific application. Alternatively, a portion of at least one software module may be provided by the OS, and the remainder thereof may be provided by a specific application.

[0177] Fig.16 is a block diagram showing a configuration of a data identifier according to an embodiment.

[0178] refer to Fig.16According to an embodiment, the data identifier 1220 may include a data acquirer 1221 , a preprocessor 1222 , a recognition data selector 1223 , a recognition result provider 1224 , and a model updater 1225 .

[0179] The data acquirer 1221 may acquire data required for determining one or more recommended applications corresponding to the object, and the preprocessor 1222 may preprocess the acquired data so that the acquired data is used to determine one or more recommended applications corresponding to the object. The preprocessor 1222 may process the acquired data in a preset format so that the recognition result provider 1224, which will be described below, may use the acquired data to determine one or more recommended applications corresponding to the object.

[0180] The recognition data selector 1223 may select data required for determining one or more recommended applications corresponding to the object from the plurality of pre-processed data. The selected data may be provided to the recognition result provider 1224. The recognition data selector 1223 may select all or part of the pre-processed data according to a preset standard for determining one or more recommended applications corresponding to the object.

[0181] The recognition result provider 1224 may apply the selected data to the data recognition model to determine one or more recommended applications corresponding to the object. The recognition result provider 1224 may provide a recognition result according to the recognition purpose of the data. The recognition result provider 1224 may apply the selected data to the data recognition model by using the data selected by the recognition data selector 1223 as an input value. In addition, the analysis result may be determined by the recognition model.

[0182] The recognition result provider 1224 may provide recognition information indicating the determined one or more recommended applications. For example, the recognition result provider 1224 may provide type, size, and location information about the determined one or more recommended applications.

[0183] The model updater 1225 may update the data recognition model based on the evaluation of the recognition result provided by the recognition result provider 1224. For example, the model updater 1225 may provide the recognition result provided by the recognition result provider 1224 to the model learner 1014 so that the model learner 1014 updates the data recognition model.

[0184] At least one of the data acquirer 1221, the preprocessor 1222, the recognition data selector 1223, the recognition result provider 1224, and the model updater 1225 in the data identifier 1220 may be manufactured in the form of at least one hardware chip and installed on the electronic device 100. For example, at least one of the data acquirer 1221, the preprocessor 1222, the recognition data selector 1223, the recognition result provider 1224, and the model updater 1225 may be manufactured in the form of a dedicated hardware chip for AI, or may be manufactured as part of an existing general-purpose processor (e.g., a CPU or an application processor) or a graphics-dedicated processor (e.g., a GPU) and may be installed on the above-mentioned various electronic devices.

[0185] In addition, the data acquirer 1221, the preprocessor 1222, the recognition data selector 1223, the recognition result provider 1224, and the model updater 1225 may be installed on a single device or may be installed on separate devices, respectively. For example, some of the data acquirer 1221, the preprocessor 1222, the recognition data selector 1223, the recognition result provider 1224, and the model updater 1225 may be included in the electronic device 100, while others may be included in the server.

[0186] In addition, at least one of the data acquirer 1221, the preprocessor 1222, the recognition data selector 1223, the recognition result provider 1224, and the model updater 1225 may be implemented as a software module. When at least one of the data acquirer 1221, the preprocessor 1222, the recognition data selector 1223, the recognition result provider 1224, and the model updater 1225 is implemented as a software module (or a program module including instructions), the software module may be stored in a non-transitory computer-readable medium. In addition, in this case, at least one software module may be provided by the OS or may be provided by a specific application. Alternatively, a portion of at least one software module may be provided by the OS, and the remainder thereof may be provided by a specific application.

[0187] Fig.17 is a block diagram showing a configuration of a server according to an embodiment.

[0188] refer to Fig.17 , the server 2000 according to the embodiment may include a DB 2100 , a communicator 2200 , and a processor 2300 .

[0189] The DB 2100 may store one or more objects recognized from an image indicating a configuration of an application desired by a user.

[0190] The communicator 2200 may include one or more elements for communicating with the electronic device 100 .

[0191] The processor 2300 may control the overall operation of the server 2000. For example, the processor 2300 may control the overall operation of the DB 2100 and the communicator 2200 by executing a program stored in the DB 2100 of the server 2000. The processor 2300 may execute the program stored in the DB 2100 to perform some operations of the electronic device 100, which have been described with reference to FIG. Figures 1 to 15 Described.

[0192] The processor 2300 may execute a function of determining one or more recommended applications corresponding to an object, which is performed by Figures 1 to 16 Functions performed by the electronic device 100 in.

[0193] Fig.18 is a diagram illustrating an example in which an electronic device and a server interact with each other to train and recognize data according to an embodiment.

[0194] refer to Fig.18 , the server 2000 may learn the criteria for determining one or more recommended applications corresponding to the object. The server 2000 may learn the criteria for determining one or more recommended applications corresponding to the object by acquiring data to be used for learning and applying the acquired data to the data recognition model.

[0195] In this case, the model learner 2104 of the server 2000 may execute Fig.16 The functions of the data trainer 1210 shown. The model learner 2104 of the server 2000 can learn the criteria about which training data will be used to determine the one or more recommended applications corresponding to the object. In addition, the model learner 2104 of the server 2000 can learn the criteria about the method of determining the one or more recommended applications corresponding to the object. The model learner 2104 can learn the criteria for determining the one or more recommended applications corresponding to the object by acquiring data to be used for learning and applying the acquired data to the data recognition model.

[0196] In addition, the recognition result provider 1224 of the electronic device 100 may apply the data selected by the recognition data selector 1023 to the data recognition model generated by the server 2000 to determine one or more recommended applications corresponding to the object. For example, the recognition result provider 1224 may send the data selected by the recognition data selector 1223 to the server 2000, and the server 2000 may request to determine one or more recommended applications corresponding to the object by applying the data selected by the recognition data selector 1223 to the recognition model.

[0197] Alternatively, the recognition result provider 1224 of the electronic device 100 may receive the recognition model generated by the server 2000 from the server 2000, and may use the received recognition model to determine one or more recommended applications corresponding to the object. In this case, the recognition result provider 1224 of the electronic device 100 may apply the data selected by the recognition data selector 1223 to the data recognition model received from the server 2000 to determine one or more recommended applications corresponding to the object.

[0198] In addition, the electronic device 100 and the server 2000 can effectively distribute and perform operations for training and data recognition of the data recognition model. In this way, data processing can be effectively performed to provide services that match the user's intentions, and the user's privacy can be effectively protected.

[0199] Some embodiments may be implemented in the form of a recording medium including computer executable instructions (e.g., computer executable program modules). Computer readable media may be any available media accessible to a computer, and may include any volatile and non-volatile media and any removable and non-removable media. In addition, computer readable recording media may include any computer storage media and communication media. Computer storage media may include any volatile and non-volatile media and any removable and non-removable media implemented by any method or technology to store information (e.g., computer readable instructions, data structures, program modules, or other data). Communication media may include computer readable instructions, data structures, program modules, other data of modulated data signals (e.g., carrier waves), or other transmission mechanisms, and may include any information transmission media.

[0200] Furthermore, the term "machine" (-or / -er) used herein may be a hardware component such as a processor or a circuit, and / or a software component executed by a hardware component such as a processor.

[0201] The above description of the present disclosure is for illustration, and those skilled in the art will appreciate that other specific forms can be easily modified without changing the technical spirit or basic features of the present disclosure. Therefore, it should be understood that the above embodiments are illustrative in all respects, rather than restrictive. For example, each element described as a single type can be implemented in a distributed manner. Similarly, the elements described as distributed can be implemented in a combined form.

[0202] The scope of the present disclosure is determined by the appended claims rather than the above detailed description, and the meaning and scope of the claims will be interpreted so that all changes or modifications derived from equivalent concepts fall within the scope of the present disclosure.

Claims

1. An electronic device, include: monitor; a memory configured to store one or more instructions; as well as A processor configured to execute the one or more instructions stored in the memory to: identifying one or more objects from an image, the image showing a layout of at least one application desired by a user, each object corresponding to one of the at least one application and indicating properties of the application, the properties including a type of the application; determining one or more recommended applications respectively corresponding to the one or more objects by using a learning model of one or more neural networks, the determination being based on similarities between attributes of the one or more recommended applications and attributes indicated by the one or more objects, wherein the learning model extracts types of recommended applications from the one or more objects in response to input of the one or more objects to the one or more neural networks, and determining the one or more recommended applications based on the extracted types of recommended applications; as well as Based on the layout of the at least one application desired by the user in the image, the determined one or more recommended applications are displayed on the display.

2. The electronic device according to claim 1, in, The processor is further configured to execute the one or more instructions to: In response to an external input of selecting one or more applications from the one or more recommended applications, the selected one or more applications are displayed on the display so that the selected one or more applications are executed.

3. The electronic device according to claim 1, in, The processor is further configured to execute the one or more instructions to: setting an arrangement criterion based on the layout for arranging the one or more recommended applications on the display based on the image; as well as The one or more recommended applications are arranged on the display based on the set arrangement criteria.

4. The electronic device according to claim 1, in, The attributes further include at least one of size, color, and shape of the application.

5. The electronic device according to claim 1, in, The processor is further configured to execute the one or more instructions to perform pre-processing for inputting the identified one or more objects into the learning model, and The preprocessing includes at least one of a cropping operation and a deformation operation.

6. The electronic device according to claim 1, in, The image includes an image generated by receiving an external input for drawing a configuration of an application desired by a user, or a photo acquired by capturing an image including a configuration of an application desired by a user.

7. The electronic device according to claim 1, in, The processor is further configured to execute the one or more instructions to: selecting the one or more recommended applications respectively corresponding to the identified one or more objects from the application database DB, and The application database DB includes a list of applications installed on the electronic device and applications stored in an external server.

8. The electronic device according to claim 7, in, The processor is further configured to execute the one or more instructions to: When it is determined that the selected one or more recommended applications are applications that are not installed on the electronic device, receiving the selected one or more recommended applications from the external server, and installing the received selected one or more recommended applications; as well as The one or more recommended applications that have been installed are displayed on the display so that the one or more recommended applications that have been installed are run.

9. The electronic device according to claim 1, in, The processor is further configured to execute the one or more instructions to: identifying textual information from the one or more objects; updating the one or more recommended applications respectively corresponding to the one or more objects to reflect the text information; as well as The updated one or more recommended applications are displayed on the display.

10. A method for operating an electronic device, the method include: identifying one or more objects from an image, the image showing a layout of at least one application desired by a user, each object corresponding to one of the at least one application and indicating properties of the application, the properties including a type of the application; determining one or more recommended applications respectively corresponding to the one or more objects by using a learning model of one or more neural networks, the determination being based on similarities between attributes of the one or more recommended applications and attributes indicated by the one or more objects, wherein the learning model extracts types of recommended applications from the one or more objects in response to input of the one or more objects to the one or more neural networks, and determining the one or more recommended applications based on the extracted types of recommended applications; as well as Based on the layout of the at least one application desired by the user in the image, the determined one or more recommended applications are displayed on a display.

11. The method according to claim 10, in, The displaying includes: in response to an external input selecting one or more applications from the one or more recommended applications, displaying the selected one or more applications on the display.

12. The method according to claim 10, in, The display includes: setting an arrangement criterion based on the layout for arranging the one or more recommended applications on a display of the electronic device based on the image; and The one or more recommended applications are arranged on a display of the electronic device based on the set arrangement criteria.

13. A computer-readable recording medium having recorded thereon a program for implementing the method according to claim 10.

Images