Electronic device and control method therefor

Abstract

This electronic device comprises: a memory; a communication interface; and at least one processor. The at least one processor: acquires screen information provided by an application when a user input for selecting an application is received; identifies the type of the application on the basis of the screen information provided by the application; identifies, on the basis of the type of the application, key information for controlling the application; generates an operation user interface (UI) on the basis of the key information; and transmits the operation UI to a terminal device via the communication interface.

Description

Electronic device and method of controlling the same

[0001] The present disclosure relates to an electronic device and a control method thereof, and more particularly, to an electronic device that generates and provides a virtual operation UI and a control method thereof.

[0002] Electronic devices can be connected to dedicated remote control devices. Since remote control devices are typically manufactured by the same manufacturer, they must have pre-defined keys.

[0003] If the electronic device is implemented as a TV, the remote control device may be a device for controlling channels and volume, etc.

[0004] If the electronic device is implemented as a game console, the remote control device may be a game pad or a control pad for key operations.

[0005] The problem is that the key structure of the remote control device corresponding to each specific TV or specific game console is fixed, so the user is forced to use the specific remote control device corresponding to the electronic device.

[0006] A single electronic device can provide multiple services. For example, a smart TV can provide content like movies and dramas, as well as games. However, the inconvenience of using different remote controls for each service provided by the device has persisted.

[0007] The present disclosure is designed to improve the above-described problems, and an object of the present disclosure is to provide an electronic device and a control method thereof that generates a virtual operation UI and provides it to a user terminal device.

[0008] According to one embodiment, an electronic device includes a memory, a communication interface, and at least one processor, wherein when a user input for selecting an application is received, the at least one processor obtains screen information provided by the application, identifies a type of the application based on the screen information provided by the application, identifies key information for controlling the application based on the type of the application, generates an operation UI (User Interface) based on the key information, and transmits the operation UI to a terminal device through the communication interface.

[0009] The at least one processor may obtain an image frame included in the screen information, identify at least one object included in the image frame, determine the type of the application based on the at least one object, and identify the key information including a key structure corresponding to the type of the application.

[0010] The at least one processor can, when a selectable object among the at least one object is identified, identify the type of the application as a first type and obtain the key information including a first key structure corresponding to the first type.

[0011] The above selectable object is a preset object that represents an item that can be selected based on a user key input, the first type is a content viewing service type, and the first key structure may be a structure that includes a movement key, a first function key for selection, and a second function key for cancellation.

[0012] The at least one processor may, when a movable object among the at least one object is identified, identify the type of the application as a second type and obtain the key information including a second key structure corresponding to the second type.

[0013] The above movable object is a preset object whose displayed position can be changed based on a user key input, the second type is a game service type, and the second key structure may be a structure including a movement key and a plurality of function keys to which functions are not mapped.

[0014] The at least one processor may obtain operation guide information corresponding to the application or log information corresponding to the application, and identify the key information including the key mapping value of each of the plurality of function keys based on at least one of the operation guide information or the log information.

[0015] The at least one processor can obtain operation guide information corresponding to the application, and identify the key information including at least one of a key shape, a key size, a key position, a key recognition method, or a key mapping value based on the operation guide information.

[0016] The at least one processor can obtain log information corresponding to the application, obtain key usage frequency based on the log information, and identify the key information including at least one of a key shape, a key size, a key position, a key recognition method, or a key mapping value based on the key usage frequency.

[0017] The at least one processor may obtain tilt information of the terminal device through the communication interface, and change the operation UI when the terminal device is identified as having rotated in a preset direction based on the tilt information.

[0018] According to one embodiment, a method for controlling an electronic device includes, when a user input for selecting an application is received, a step of obtaining screen information provided by the application, a step of identifying a type of the application based on the screen information provided by the application, a step of identifying key information for controlling the application based on the type of the application, a step of generating an operation UI (User Interface) based on the key information, and a step of transmitting the operation UI to a terminal device.

[0019] The step of identifying the type of the application may include obtaining an image frame included in the screen information, identifying at least one object included in the image frame, and determining the type of the application based on the at least one object, and the step of identifying the key information may identify the key information including a key structure corresponding to the type of the application.

[0020] The step of identifying the type of the application may identify the type of the application as a first type when a selectable object among the at least one object is identified, and the step of identifying the key information may obtain the key information including a first key structure corresponding to the first type.

[0021] The above selectable object is a preset object that represents an item that can be selected based on a user key input, the first type is a content viewing service type, and the first key structure may be a structure that includes a movement key, a first function key for selection, and a second function key for cancellation.

[0022] The step of identifying the type of the application may include, when a movable object among the at least one object is identified, identifying the type of the application as a second type, and the step of identifying the key information may obtain the key information including a second key structure corresponding to the second type.

[0023] The above movable object is a preset object whose displayed position can be changed based on a user key input, the second type is a game service type, and the second key structure may be a structure including a movement key and a plurality of function keys to which functions are not mapped.

[0024] The step of identifying the key information may include obtaining operation guide information corresponding to the application or log information corresponding to the application, and identifying the key information including the key mapping value of each of the plurality of function keys based on at least one of the operation guide information or the log information.

[0025] The step of identifying the above key information may include obtaining operation guide information corresponding to the application, and identifying the key information including at least one of a key shape, a key size, a key position, a key recognition method, or a key mapping value based on the operation guide information.

[0026] The step of identifying the above key information may include obtaining log information corresponding to the application, obtaining key usage frequency based on the log information, and identifying the key information including at least one of a key shape, a key size, a key location, a key recognition method, or a key mapping value based on the key usage frequency.

[0027] The above control method may include a step of obtaining inclination information of the terminal device, and a step of changing the operation UI when the terminal device is identified as having rotated in a preset direction based on the inclination information.

[0028] FIG. 1 is a drawing for explaining a system including an electronic device and a terminal device according to one embodiment.

[0029] FIG. 2 is a drawing for explaining a system including an electronic device and a plurality of terminal devices, according to one embodiment.

[0030] FIG. 3 is a block diagram illustrating an electronic device according to one embodiment.

[0031] FIG. 4 is a block diagram illustrating a specific configuration of the electronic device of FIG. 3, according to one embodiment.

[0032] Figure 5 is a drawing for explanation according to one embodiment.

[0033] FIG. 6 is a drawing for explaining an operation for connecting to a terminal device according to one embodiment.

[0034] FIG. 7 is a drawing for explaining an operation for connecting to a terminal device according to one embodiment.

[0035] FIG. 8 is a drawing for explaining an operation UI corresponding to an application according to one embodiment.

[0036] FIG. 9 is a drawing for explaining an operation UI according to an operation pad, according to one embodiment.

[0037] FIG. 10 is a drawing for explaining an operation of providing a manipulation UI according to one embodiment.

[0038] FIG. 11 is a diagram for explaining an operation of identifying key information according to one embodiment.

[0039] FIG. 12 is a diagram for explaining an operation of identifying a key structure among key information according to one embodiment.

[0040] FIG. 13 is a diagram for explaining an operation of identifying key information according to one embodiment.

[0041] FIG. 14 is a drawing for explaining an application type according to one embodiment.

[0042] FIG. 15 is a diagram for explaining key usage frequency based on operation guide information according to one embodiment.

[0043] FIG. 16 is a diagram for explaining log information according to one embodiment.

[0044] FIG. 17 is a drawing for explaining touch settings of an operation UI according to one embodiment.

[0045] FIG. 18 is a drawing for explaining an operation of changing touch sensitivity according to one embodiment.

[0046] FIG. 19 is a drawing for explaining an operation of changing an operation UI according to the inclination of a terminal device, according to one embodiment.

[0047] FIG. 20 is a drawing for explaining an operation of changing an operation UI using log information according to one embodiment.

[0048] FIG. 21 is a drawing for explaining a guide screen for changing the operation UI according to one embodiment.

[0049] FIG. 22 is a drawing for explaining an operation of changing an operation UI using log information according to one embodiment.

[0050] FIG. 23 is a drawing for explaining an operation of displaying a guide screen related to an operation UI according to one embodiment.

[0051] FIG. 24 is a drawing for explaining an operation of displaying a guide screen related to an operation UI according to one embodiment.

[0052] FIG. 25 is a drawing for explaining an operation of displaying a guide screen related to an operation UI according to one embodiment.

[0053] FIG. 26 is a drawing for explaining an operation of displaying a guide screen related to an operation UI according to one embodiment.

[0054] FIG. 27 is a drawing for explaining an operation for changing an operation UI according to one embodiment.

[0055] FIG. 28 is a drawing for explaining an operation for changing an operation UI according to one embodiment.

[0056] FIG. 29 is a diagram for explaining an operation of receiving screen information through a server according to one embodiment.

[0057] FIG. 30 is a drawing for explaining a method for controlling an electronic device according to one embodiment.

[0058] Hereinafter, the present disclosure will be described in detail with reference to the attached drawings.

[0059] The terms used in the embodiments of this disclosure have been selected from widely used, current terms, taking into account the functions of this disclosure. However, these terms may vary depending on the intentions of those skilled in the art, precedents, the emergence of new technologies, etc. Furthermore, in certain cases, terms may be arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the description of the relevant disclosure. Therefore, the terms used in this disclosure should not be defined simply as names of terms, but rather based on the meanings of the terms and the overall content of this disclosure.

[0060] In this specification, expressions such as “has,” “can have,” “includes,” or “may include” indicate the presence of a feature (e.g., a number, function, operation, or component such as a part), and do not exclude the presence of additional features.

[0061] The expression "at least one of A and / or B" should be understood to mean either "A" or "B" or "A and B".

[0062] As used herein, the expressions “first,” “second,” “first,” or “second,” etc., may describe various components, regardless of order and / or importance, and are only used to distinguish one component from another, but do not limit the components.

[0063] When it is said that a component (e.g., a first component) is “(operatively or communicatively) coupled with / to” or “connected to” another component (e.g., a second component), it should be understood that the component may be directly coupled to the other component, or may be connected through another component (e.g., a third component).

[0064] Singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, terms such as "comprise" or "consist of" are intended to indicate the presence of a feature, number, step, operation, component, part, or combination thereof described in the specification, but should be understood not to preclude the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0065] In the present disclosure, a "module" or "part" performs at least one function or operation and may be implemented in hardware or software, or a combination of hardware and software. Furthermore, multiple "modules" or multiple "parts" may be integrated into at least one module and implemented as at least one processor, excluding any "modules" or "parts" that need to be implemented as specific hardware.

[0066] In this specification, the term user may refer to a person using an electronic device or a device using an electronic device (e.g., an artificial intelligence electronic device).

[0067] An embodiment of the present disclosure will be described in more detail with reference to the attached drawings below.

[0068] FIG. 1 is a drawing for explaining a system including an electronic device (100) and a terminal device (200), according to one embodiment.

[0069] FIG. 1 is a drawing for explaining a system (1000) including an electronic device (100) and a terminal device (200).

[0070] An electronic device (100) may include a display. The electronic device (100) may display content. The content may include a screen related to an application running on the electronic device (100). The content may include an image frame. The content may refer to a still image or a moving image. For example, the content may refer to a game, broadcast content, educational content, etc. The content is not limited to any particular embodiment.

[0071] The terminal device (200) may include a display. The terminal device (200) may display a manipulation UI (10) related to content displayed on the electronic device (100) on the display. The manipulation UI (10) may include at least one UI for manipulating content displayed on the electronic device (100). The manipulation UI (10) may vary depending on the content displayed on the electronic device (100). The manipulation UI (10) may vary depending on the application running on the electronic device (100). The terminal device (200) may be described as an external device.

[0072] The operation UI (10) may be described as a UI related to control, a UI for control, an operation UI, a UI related to operation, a UI for operation, etc.

[0073] FIG. 2 is a drawing for explaining a system (2000) including an electronic device (100) and a plurality of terminal devices (200), according to one embodiment.

[0074] Referring to FIG. 2, the electronic device (100) can be simultaneously connected to a terminal device (200) and a control pad (300). When at least one user key input is received from the terminal device (200) or the control pad (300), the electronic device (100) can provide a screen corresponding to the received user key input.

[0075] Assume that content A is displayed on an electronic device (100). A dedicated controller may exist for manipulating content A. It is assumed that the dedicated controller is a control pad (300). The control pad (300) may refer to a physically existing hardware device.

[0076] The operation pad (300) may include at least one button (or element). UI information corresponding to the operation pad (300) may include information related to the button of the operation pad (300). The UI information may include at least one of the shape, position, area, size, color, or transparency of the button.

[0077] Based on the UI information corresponding to the operation pad (300), the electronic device (100) or terminal device (200) can display the operation UI (10).

[0078] According to various embodiments, the arrangement structure of the buttons included in the operation pad (300) may be displayed on the electronic device (100) or terminal device (200).

[0079] According to various embodiments, the arrangement structure of the buttons included in the operation pad (300) may be partially changed and displayed on the electronic device (100) or the terminal device (200). This is because the sizes of the electronic device (100) and the terminal device (200) are different. If the UI information corresponding to the operation pad (300) is displayed as is without change, there is a problem that the user may experience inconvenience due to the arrangement or size of the operation UI (10).

[0080] Accordingly, the UI information of the operation pad (300) can be partially modified so that the electronic device (100) or terminal device (200) can display the operation UI (10).

[0081] The control pad (300) may be described as a remote control device, joystick, game pad, control pen, etc.

[0082] FIG. 3 is a block diagram illustrating an electronic device (100) according to one embodiment.

[0083] The electronic device (100) may include at least one of a memory (110), at least one processor (120), or a communication interface (130).

[0084] The memory (110) can store applications, screen information provided by the applications, etc. The memory (110) can include at least one instruction executed by at least one processor (120).

[0085] At least one processor (120) can receive a user input for selecting an application. At least one processor (120) can execute the selected application based on the user input.

[0086] At least one processor (120) can obtain screen information provided by the application.

[0087] Screen information can represent various information related to the screen used to display services provided by the application. Screen information can be information related to the execution of the application. At least one processor (120) can output screen information through a display (140).

[0088] At least one processor (120) can identify the type of an application based on screen information provided by the application. The type of the application may represent a classification criterion determined by the user. The type may be described as a category, group, etc.

[0089] At least one processor (120) can identify key information for controlling an application based on the type of the application.

[0090] Key information may be information used to generate an operation UI. Key information may include various information related to keys included in the operation UI. Key information may include at least one of a key structure, key shape, key size, key location, key recognition method, or key mapping value.

[0091] At least one processor (120) can generate an operation UI (User Interface) based on key information. At least one processor (120) can transmit the operation UI to a terminal device via a communication interface (130).

[0092] At least one processor (120) can obtain an image frame included in the screen information. At least one processor (120) can obtain at least one image frame provided by the application.

[0093] For example, at least one processor (120) can determine the type of application based on one image frame.

[0094] For example, at least one processor (120) may determine the type of application based on a plurality of image frames. At least one processor (120) may receive image frames during a threshold period of time. At least one processor (120) may analyze each of the plurality of image frames received during the threshold period of time and synthesize the analysis results to identify the type of application.

[0095] At least one processor (120) can identify at least one object included in an image frame. At least one processor (120) can determine the type of application based on the at least one object.

[0096] According to one embodiment, at least one processor (120) can identify all details included in the key information based on the type of application.

[0097] According to one embodiment, at least one processor (120) may first identify only a key structure among a plurality of elements included in the key information based on the type of the application. At least one processor (120) may identify key information including a key structure corresponding to the type of the application.

[0098] At least one processor (120) can identify the type of the application as a first type when at least one selectable object is identified among the objects.

[0099] At least one processor (120) can obtain key information including a first key structure corresponding to a first type.

[0100] A selectable object may be a preset object that represents an item that can be selected based on user key input.

[0101] A selectable object can represent an object that can be selected by the user from among multiple objects present in an image frame. The user's settings for the selectable object may be predetermined. For example, the selectable object can represent a menu object, an item object, etc.

[0102] A selectable object can represent an object of a preset shape with a threshold width or larger. For example, a selectable object can be defined as a square object with a width of at least 10 pixels and a height of at least 10 pixels. The detailed dimensions, such as the width, height, and squareness, can be customized according to the user's preferences.

[0103] The first type may be a content viewing service. A content viewing service may refer to a type of video content viewing, such as movies or dramas. A content viewing service may refer to a service for viewing (or watching) videos.

[0104] The first key structure may be a structure including a movement key, a first function key for selection, and a second function key for cancel.

[0105] At least one processor (120) can identify the type of the application as a second type when at least one movable object among the objects is identified.

[0106] At least one processor (120) can obtain key information including a second key structure corresponding to the second type.

[0107] A movable object may be a preset object whose displayed position can change based on user key input.

[0108] The second type may be a game service type.

[0109] The second key structure may be a structure including a movement key and multiple function keys to which functions are not mapped.

[0110] A movable object may represent at least one object with a movable attribute among a plurality of objects identified in an image frame. A movable object may be an object that moves in the game. A movable object may be an object whose user settings are predetermined. For example, a movable object may be an animal object, a human object, a ball object, a vehicle object, etc.

[0111] Function keys included in the second key structure may not have functions mapped to them. This lack of function mapping may indicate that the functions of each function key are undetermined. For function keys for which functions are not defined, at least one processor (120) may use additional information to determine the function (or key mapping value) corresponding to the function key. The additional information may include at least one of operation guide information and application-specific log information.

[0112] At least one processor (120) can obtain operation guide information corresponding to an application or log information corresponding to the application. At least one processor (120) can identify key information including key mapping values ​​of each of a plurality of function keys based on at least one of the operation guide information or the log information.

[0113] A detailed description of the key structure is given in Fig. 12.

[0114] The above description describes the actions that determine the key structure. In addition to the key structure, additional information may be required to generate the operation UI. This additional information may include at least one of the following: operation guide information or application-specific log information.

[0115] At least one processor (120) can obtain operation guide information corresponding to an application. At least one processor (120) can identify key information including at least one of a key shape, a key size, a key position, a key recognition method, or a key mapping value based on the operation guide information.

[0116] Specific descriptions related to the operating guide information are described in Fig. 15.

[0117] At least one processor (120) can obtain log information corresponding to the application. At least one processor (120) can obtain key usage frequency based on the log information. At least one processor (120) can identify key information including at least one of a key shape, a key size, a key location, a key recognition method, or a key mapping value based on the key usage frequency.

[0118] Log information can be divided into first log information and second log information depending on the time of creation.

[0119] The first log information may be information already stored in the memory (110). The first log information may represent information actually entered by the user using the operation pad (300) corresponding to the application at a time prior to generating the operation UI.

[0120] The second log information may be information collected based on the usage history of the operation UI. The second log information may be information entered by the user after the operation UI is created. The second log information may be information regarding user key inputs entered according to the operation UI displayed on the terminal device (200).

[0121] In the description below, each example is described as one log piece of information. However, in an example where the first log piece and the second log piece are used simultaneously, an ordinal number may be described.

[0122] At least one processor (120) can obtain tilt information of the terminal device via a communication interface. If the at least one processor (120) determines that the terminal device has rotated in a preset direction based on the tilt information, the at least one processor (120) can change the operation UI. A detailed description of this is provided in FIG. 19.

[0123] According to various embodiments, the electronic device (100) may generate an operation UI without separately identifying the application type. At least one processor (120) may generate the operation UI based on at least one of operation guide information or log information. In the description below, it is described that the key structure is first identified based on the application type, and then the key shape, etc. are determined using additional information (operation guide information or log information). However, this may be only one embodiment. At least one processor (120) may directly generate key information without identifying the application type.

[0124] FIG. 4 is a block diagram for explaining a specific configuration of the electronic device (100) of FIG. 3, according to one embodiment.

[0125] Referring to FIG. 4, the electronic device (100) may include at least one of a memory (110), at least one processor (120), a communication interface (130), a display (140), an operation interface (150), an input / output interface (160), a speaker (170), a microphone (180), and a camera (190).

[0126] The memory (110) may be implemented as an internal memory such as a ROM (e.g., an electrically erasable programmable read-only memory (EEPROM)) or RAM included in at least one processor (120), or may be implemented as a separate memory from at least one processor (120). The memory (110) may be implemented as a memory embedded in the electronic device (100) or as a memory detachable from the electronic device (100) depending on the purpose of data storage. For example, data for driving the electronic device (100) may be stored in a memory embedded in the electronic device (100), and data for expanding functions of the electronic device (100) may be stored in a memory detachable from the electronic device (100).

[0127] In the case of memory embedded in the electronic device (100), it may be implemented as at least one of volatile memory (e.g., dynamic RAM (DRAM), static RAM (SRAM), or synchronous dynamic RAM (SDRAM)), non-volatile memory (e.g., one time programmable ROM (OTPROM), programmable ROM (PROM), erasable and programmable ROM (EPROM), electrically erasable and programmable ROM (EEPROM), mask ROM, flash ROM, flash memory (e.g., NAND flash or NOR flash), etc.), hard drive, or solid state drive (SSD), and in the case of memory that can be attached or detached to the electronic device (100), it may be implemented in the form of a memory card (e.g., compact flash (CF), secure digital (SD), micro secure digital (Micro-SD), mini secure digital (Mini-SD), extreme digital (xD), multi-media card (MMC), etc.), external memory that can be connected to a USB port (e.g., USB memory), etc.

[0128] The memory (110) can store at least one instruction. Based on the instruction stored in the memory (110), at least one processor (120) can perform various operations.

[0129] At least one processor (120) may be implemented as a digital signal processor (DSP), a microprocessor, or a time controller (TCON) that processes digital signals. However, the present invention is not limited thereto, and may include one or more of a central processing unit (CPU), a micro controller unit (MCU), a micro processing unit (MPU), a controller, an application processor (AP), a graphics-processing unit (GPU), a communication processor (CP), or an advanced reduced instruction set computer (RISC) machines (ARM) processor, or may be defined by the relevant terminology. At least one processor (120) may be implemented as a system on chip (SoC) having a built-in processing algorithm, a large scale integration (LSI), or may be implemented in the form of a field programmable gate array (FPGA). At least one processor (120) may perform various functions by executing computer executable instructions stored in a memory.

[0130] The communication interface (130) is a configuration that performs communication with various types of external devices according to various types of communication methods. The communication interface (130) may include a wireless communication module or a wired communication module. Each communication module may be implemented in the form of at least one hardware chip.

[0131] A wireless communication module may be a module that communicates wirelessly with an external device. For example, the wireless communication module may include at least one of a Wi-Fi module, a Bluetooth module, an infrared communication module, or other communication modules.

[0132] Wi-Fi and Bluetooth modules can communicate via Wi-Fi and Bluetooth, respectively. When using a Wi-Fi or Bluetooth module, various connection information, such as the service set identifier (SSID) and session key, is first transmitted and received. This information is then used to establish a communication connection before various other information can be transmitted and received.

[0133] Infrared communication modules perform communication based on infrared communication (IrDA, infrared Data Association) technology, which transmits data wirelessly over short distances using infrared light, which is between visible light and millimeter waves.

[0134] In addition to the above-described communication method, other communication modules may include at least one communication chip that performs communication according to various wireless communication standards such as zigbee, 3G (3rd Generation), 3GPP (3rd Generation Partnership Project), LTE (Long Term Evolution), LTE-A (LTE Advanced), 4G (4th Generation), 5G (5th Generation), etc.

[0135] A wired communication module may be a module that communicates with an external device via a wire. For example, the wired communication module may include at least one of a Local Area Network (LAN) module, an Ethernet module, a paired cable, a coaxial cable, a fiber optic cable, or an Ultra Wide-Band (UWB) module.

[0136] According to various embodiments, the communication interface (130) may utilize the same communication module (e.g., a Wi-Fi module) to communicate with an external device such as a remote control device and an external server.

[0137] According to various embodiments, the communication interface (130) may utilize different communication modules to communicate with external devices, such as remote control devices, and external servers. For example, the communication interface (130) may utilize at least one of an Ethernet module or a Wi-Fi module to communicate with an external server, and may also utilize a Bluetooth module to communicate with an external device, such as a remote control device. However, this is merely an embodiment, and the communication interface (130) may utilize at least one of various communication modules when communicating with multiple external devices or external servers.

[0138] The display (140) may be implemented as a variety of displays such as a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display panel (PDP), etc. The display (140) may also include a driving circuit, a backlight unit, etc., which may be implemented as a form such as an a-si TFT (amorphous silicon thin film transistor), an LTPS (low temperature poly silicon) TFT, an OTFT (organic TFT), etc. The display (140) may be implemented as a touch screen combined with a touch sensor, a flexible display, a three-dimensional display (3D display, three-dimensional dispaly), etc. According to an embodiment of the present disclosure, the display (140) may include not only a display panel that outputs an image, but also a bezel that houses the display panel. In particular, according to an embodiment of the present disclosure, the bezel may include a touch sensor for detecting user interaction.

[0139] The operating interface (150) may be implemented as a device such as a button, a touch pad, a mouse, and a keyboard, or as a touch screen capable of performing the above-described display function and operating input function. The button may be a mechanical button, a touch pad, a wheel, or any other type of button formed in any area of ​​the front, side, or back of the main body of the electronic device (100).

[0140] The input / output interface (160) may be any one of HDMI (High Definition Multimedia Interface), MHL (Mobile High-Definition Link), USB (Universal Serial Bus), DP (Display Port), Thunderbolt, VGA (Video Graphics Array) port, RGB port, D-SUB (D-subminiature), and DVI (Digital Visual Interface). The input / output interface (160) may input / output at least one of audio and video signals. Depending on the implementation example, the input / output interface (160) may include a port that inputs / outputs only audio signals and a port that inputs / outputs only video signals as separate ports, or may be implemented as a single port that inputs / outputs both audio signals and video signals. The electronic device (100) may transmit at least one of the audio and video signals to an external device (e.g., an external display device or an external speaker) through the input / output interface (160). An output port included in the input / output interface (160) can be connected to an external device, and the electronic device (100) can transmit at least one of an audio and video signal to the external device through the output port.

[0141] The input / output interface (160) can be connected to a communication interface. The input / output interface (160) can transmit information received from an external device to the communication interface or transmit information received through the communication interface to the external device.

[0142] The speaker (170) may be a component that outputs various audio data as well as various notification sounds or voice messages.

[0143] The microphone (180) is a component that receives a user's voice or other sounds and converts them into audio data. The microphone (180) can receive the user's voice in an activated state. For example, the microphone (180) can be formed integrally on the upper side, the front side, the side side, etc. of the electronic device (100). The microphone (180) can include various components such as a microphone that collects the user's voice in analog form, an amplifier circuit that amplifies the collected user's voice, an A / D conversion circuit that samples the amplified user's voice and converts it into a digital signal, and a filter circuit that removes noise components from the converted digital signal.

[0144] The camera (190) is a device configured to capture a subject and generate a captured image, and the captured image includes both moving images and still images. The camera (190) can acquire images for at least one external device and can be implemented with a camera, lens, infrared sensor, or the like.

[0145] The camera (190) may include a lens and an image sensor. The type of lens may include a general-purpose lens, a wide-angle lens, a zoom lens, etc., and may be determined according to the type, characteristics, usage environment, etc. of the electronic device (100). The image sensor may include a complementary metal oxide semiconductor (CMOS) and a charge-coupled device (CCD).

[0146] Figure 5 is a drawing for explanation according to one embodiment.

[0147] Referring to FIG. 5, the electronic device (100) may include at least one of an image frame analysis module (121), a key information generation module (122), an operation UI generation module (123), or a data storage module (111). The electronic device (100) may additionally include a communication interface (130).

[0148] The image frame analysis module (121) can receive an image frame provided by the application (S1).

[0149] The image frame analysis module (121) can analyze an image frame to identify (or obtain) the type of application.

[0150] The image frame analysis module (121) may include at least one of an object analysis module or an application analysis module.

[0151] The object analysis module can extract objects contained in an image frame and analyze their characteristics (or properties). The object analysis module can generate analysis results indicating the characteristics of objects contained in the image frame.

[0152] The application analysis module can identify the type of application based on the analysis results generated by the object analysis module.

[0153] For example, if a selectable object is identified in the analysis results, the application analysis module can identify the application as a first type (content viewing service type).

[0154] For example, if a movable object is identified in the analysis results, the application analysis module can identify the application as a second type (game service type).

[0155] The image frame analysis module (121) can transmit the type of application generated by the application analysis module to the key information generation module (122) (S2).

[0156] The key information generation module (122) can receive the type of application from the image frame analysis module (121). The key information generation module (122) can generate key information based on the type of application.

[0157] The key information generation module (122) can generate information for representing at least one key constituting the operation UI. The key information generation module (122) can include at least one of a key structure generation module, a key shape generation module, a key size generation module, a key location generation module, a key recognition method generation module, or a key mapping value generation module.

[0158] The key structure generation module can identify the key structure based on the type of application transmitted from the image frame analysis module (121).

[0159] The key information generation module (122) may require additional information to determine information other than the key structure (key shape, key size, key location, key recognition method, or key mapping value, etc.).

[0160] The key information generation module (122) can transmit a signal requesting additional information to the data storage module (111) (S3).

[0161] The data storage module (111) can receive a signal requesting additional information from the key information generation module (122).

[0162] The data storage module (111) may be a module that stores various information related to the operation UI. The data storage module (111) may include at least one of a first data set including at least one operation guide information, a second data set including at least one log information, or a third data set including at least one previously generated operation UI.

[0163] When additional information is received, the data storage module (111) can transmit response information to the key information generation module (122) (S4). The response information can include data stored in at least one of the first data set, the second data set, and the third data set.

[0164] The key information generation module (122) can receive response information from the data storage module (111). The key information generation module (122) can additionally determine information other than the key structure (key shape, key size, key location, key recognition method, or key mapping value, etc.) based on the response information from the data storage module (111).

[0165] According to various embodiments, even if the key structure has already been determined, the key information generation module (122) can change the key structure based on the response information.

[0166] In various embodiments, if the key structure is not determined, the key information generation module (122) can newly identify the key structure based on the response information.

[0167] When key information is generated, the key information generation module (122) can transmit the key information to the operation UI generation module (123) (S5).

[0168] The manipulation UI generation module (123) can generate a manipulation UI for controlling an application based on key information. The manipulation UI can be described as a virtual manipulation UI or a virtual manipulation screen.

[0169] The manipulation UI generation module (123) can transmit the generated manipulation UI to the communication interface (130) (S6). The communication interface (130) can transmit the manipulation UI to the terminal device (200) (S7).

[0170] The terminal device (200) may include at least one of a communication interface (230) or a display (240).

[0171] The terminal device (200) can receive the operation UI from the electronic device (100) via the communication interface (230). The communication interface (230) can transmit the operation UI to the display (240) (S8). The terminal device (200) can output the operation UI via the display.

[0172] According to various embodiments, the manipulation UI generation module (123) may transmit the manipulation UI to the data storage module (111). The data storage module (111) may store the generated manipulation UI in a third data set. The data storage module (111) may provide the already stored manipulation UI upon request (or call).

[0173] FIG. 6 is a drawing for explaining an operation for connecting to a terminal device (200) according to one embodiment.

[0174] Referring to FIG. 6, the electronic device (100) can perform a communication connection with the terminal device (200). The electronic device (100) can provide a guide screen (610) for the communication connection with the terminal device (200).

[0175] The guide screen (610) may include a QR code containing information for connection with the electronic device (100) and text indicating a guide related to the connection (Please scan the QR code).

[0176] When the QR code included in the guide screen (610) is scanned, the terminal device (200) can display a guide screen (620) corresponding to the QR code. The guide screen (620) may be a screen indicating a connection with the electronic device (100). The guide screen (620) may include information indicating that a virtual pad (or virtual operation UI) is provided or a UI confirming a connection with the electronic device (100).

[0177] When a user input is received through the guide screen (620) displayed on the terminal device (200), the terminal device (200) can transmit a signal requesting a connection to the electronic device (100). When the user input is received, the electronic device (100) and the terminal device (200) can be mutually connected for communication.

[0178] According to various embodiments, the electronic device (100) and the terminal device (200) can be connected to each other by simply scanning a QR code without user input.

[0179] FIG. 7 is a drawing for explaining an operation for connecting to a terminal device (200) according to one embodiment.

[0180] Referring to FIG. 7, the electronic device (100) may display a connection guide screen with an external device in relation to the displayed content. When the content to be displayed on the electronic device (100) is specified (or selected), a device for manipulating the specified content may be required. If a dedicated controller (hereinafter referred to as a control pad (300)) exists, the control pad (300) must be connected to the electronic device (100). If the control pad (300) exists but is not connected to the electronic device (100), the electronic device (100) may display a control UI using the terminal device (200). The user may manipulate the content displayed on the electronic device (100) using the terminal device (200) instead of the control pad (300).

[0181] The electronic device (100) may display a connection guide screen (or guide screen) for connection with the control pad (300) or terminal device (200). The connection guide screen may include at least one of a UI (710) indicating an attempt to connect with a dedicated controller or external device, or a UI (720) indicating a list of connectable devices.

[0182] The UI (710) may include text. Depending on the implementation example, the UI (710) may include an image, icon, etc. indicating an attempt to connect with a controller or external device, etc.

[0183] The UI (720) may include a list of external devices that can be connected to the current electronic device (100) and a button for receiving a command to initiate a connection with the external device.

[0184] The list of external devices may include both a terminal device (200) and a control pad (300). If the list of external devices connectable to the electronic device (100) includes both a terminal device (200) and a control pad (300), the electronic device (100) may display the control pad (300) with priority. For example, the method of displaying with priority may mean displaying the device with priority at the top of the list of external devices or highlighting the device with priority in a preset color.

[0185] FIG. 8 is a drawing for explaining an operation UI corresponding to an application according to one embodiment.

[0186] Referring to FIG. 8, the terminal device (200) can display a control UI (10) corresponding to an application. Different control UIs may exist for each application. If the application providers who created the applications are different, the control UIs on the control pad may differ. Even if the applications are provided by the same provider, the control UIs on the control pad may differ if the applications provided are different.

[0187] An electronic device (100) can acquire an operation UI corresponding to a specific application. Since the operation UI corresponding to each application is different, the operation UI displayed may be different depending on the application.

[0188] According to the embodiment (810) of FIG. 8, the terminal device (200) can display an operation UI (10-1) corresponding to application A.

[0189] According to the embodiment (820) of FIG. 8, the terminal device (200) can display an operation UI (10-2) corresponding to the B application.

[0190] According to the embodiment (830) of FIG. 8, the terminal device (200) can display an operation UI (10-3) corresponding to the C application.

[0191] At least one of the shape, position, area, size, color, and transparency of the elements included in the operation UI (10) corresponding to each application may be different.

[0192] FIG. 9 is a drawing for explaining an operation UI according to an operation pad, according to one embodiment.

[0193] Referring to Fig. 9, the terminal device (200) can display a control UI (10) corresponding to the control pad. Fig. 8 illustrates that the control UI varies depending on the application. In other implementation examples, different control UIs may exist depending on the control pad. If the control pad varies, the control UI may also vary.

[0194] The electronic device (100) can obtain first manipulation UI information corresponding to a specific manipulation pad. The electronic device (100) can obtain second manipulation UI information by changing the first manipulation UI information based on size information of the terminal device (200). The electronic device (100) can display a second UI based on the second manipulation UI information. Since the first manipulation UI information corresponding to each application is different, the manipulation UI (10) displayed may be different depending on the application.

[0195] According to the embodiment (910) of FIG. 9, the terminal device (200) can display an operation UI (10-4) corresponding to the operation pad (301).

[0196] According to the embodiment (920) of FIG. 9, the terminal device (200) can display an operation UI (10-5) corresponding to the operation pad (302).

[0197] According to the embodiment (930) of FIG. 9, the terminal device (200) can display an operation UI (10-6) corresponding to the operation pad (303).

[0198] According to the embodiment (940) of FIG. 9, the terminal device (200) can display an operation UI (10-7) corresponding to the operation pad (304).

[0199] FIG. 10 is a drawing for explaining an operation of providing a manipulation UI according to one embodiment.

[0200] Referring to FIG. 10, the electronic device (100) can execute an application (S1010). The electronic device (100) can receive user input for selecting a specific application.

[0201] The electronic device (100) can generate an operation UI corresponding to the application (S1020). The operation UI may be a UI that controls an application running on the electronic device (100). The operation UI may represent a virtual operation pad screen provided to an external device (or terminal device) connected to the electronic device (100).

[0202] The electronic device (100) can provide a manipulation UI (S1030). The electronic device (100) can provide the manipulation UI to a terminal device (200) connected to the electronic device (100). The terminal device (200) can display the received manipulation UI. The terminal device (200) can receive a user key input through the manipulation UI. When a user key input is received through the manipulation UI, the electronic device (100) can change the screen provided by the application through the user key input.

[0203] FIG. 11 is a diagram for explaining an operation of identifying key information according to one embodiment.

[0204] Steps S1110 and S1130 of Fig. 11 may correspond to steps S1010 and S1030 of Fig. 10. Duplicate explanation is omitted.

[0205] The electronic device (100) can acquire screen information provided by an application (S1121). The screen information may include various information related to the screen displayed when the application is executed. For example, the screen information may include identification information of the application providing the screen, image frames constituting the screen, timing control information for the image frames, etc.

[0206] The electronic device (100) can identify the type of application based on screen information (S1122).

[0207] An application type can represent a preset classification criterion. For example, application types may include content viewing service types, game service types, education service types, chat service types, etc. These preset classification criteria can be changed based on user settings.

[0208] For example, the electronic device (100) can identify the type of application using the application's identification information.

[0209] For example, the electronic device (100) can analyze an image frame to identify the type of application. An artificial intelligence model can be used to analyze the image frame. The artificial intelligence model can be a model that analyzes the image and outputs a probability value for the type of application. The Automatic Contents Recognition (ACR) function can be used to analyze the image.

[0210] The electronic device (100) can identify key information corresponding to the type of application (S1123).

[0211] Key information may be information used to generate an operation UI. Key information may include various information related to keys included in the operation UI. Key information may include at least one of a key structure, key shape, key size, key location, key recognition method, or key mapping value.

[0212] A key structure can indicate the form in which a key is composed. A key structure can indicate whether a key performing a specific function is included, or the number of keys.

[0213] For example, a first key structure may represent a structure including a navigation key, a first function key (select), and a second function key (cancel). The first key structure may include a navigation key, a first function key for selection, and a second function key for cancel. The navigation key may be composed of at least one sub-key (detail key). For example, the navigation key may include at least one of an up key, a down key, a left key, a right key, and a joystick key.

[0214] For example, the second key structure may include a movement key and a function key (undetermined). The exact function of the function key may be determined based on operation guide information or log information, which will be described later. A description of this is provided in Fig. 13.

[0215] Key shape can indicate the appearance of a key.

[0216] Key size can represent the physical dimensions of the UI that represents the key.

[0217] Key positions can indicate the coordinates at which the key is displayed.

[0218] The key recognition method may indicate the touch method by which the key is recognized. For example, the key recognition method may indicate a touch method, a swipe method, etc. The swipe method may be described as a drag method.

[0219] A key mapping value can indicate the command to be executed corresponding to a key. For example, a key mapping value can indicate which command is executed when a specific key is pressed.

[0220] FIG. 12 is a diagram for explaining an operation of identifying a key structure among key information according to one embodiment.

[0221] Referring to FIG. 12, after acquiring screen information, the electronic device (100) can acquire an image frame included in the screen information (S1205).

[0222] The electronic device (100) can identify at least one object included in an image frame (S1210).

[0223] The electronic device (100) can determine whether a selectable object is identified (S1215).

[0224] A selectable object can represent an object that can be selected by the user from among multiple objects present in an image frame. The user's settings for the selectable object may be predetermined. For example, the selectable object can represent a menu object, an item object, etc.

[0225] A selectable object can represent an object of a preset shape with a threshold width or larger. For example, a selectable object can be defined as a square object with a width of at least 10 pixels and a height of at least 10 pixels. The detailed dimensions, such as the width, height, and squareness, can be customized according to the user's preferences.

[0226] Step S1215 may be replaced by an operation in which a threshold number of selectable objects are identified. One selectable object may also exist in the game service type. If a threshold number of selectable objects is identified, the electronic device (100) may identify the application as a second application type (content viewing service type).

[0227] When a selectable object is identified in the image frame (S1215-Y), the electronic device (100) can identify the application providing screen information as a first application type (S1220). The first application type may be a content viewing service type.

[0228] Once the first application type is identified, the electronic device (100) can identify a first key structure (S1225). The first key structure may represent a structure including a navigation key and a function key (selection key, cancel key). The function keys included in the first key structure may include keys that perform a selection function and a cancel function. The navigation key and function keys may represent multiple concepts.

[0229] If a selectable object is not identified in the image frame (S1215-N), the electronic device (100) can determine whether a movable object is identified in the image frame (S1230).

[0230] A movable object may represent at least one object with a movable attribute among a plurality of objects identified in an image frame. A movable object may be an object that moves in the game. A movable object may be an object whose user settings are predetermined. For example, a movable object may be an animal object, a human object, a ball object, a vehicle object, etc.

[0231] When a movable object is identified in the image frame (S1230-Y), the electronic device (100) can identify a second application type that provides screen information (S1235). The second application type can indicate a game service type.

[0232] When the application is identified as a second application type, the electronic device (100) can identify a second key structure (S1240). The second key structure may represent a structure including a navigation key and a function key (an undefined function key). Any function key included in the second key structure may be a key whose function is not defined. The navigation key and function key may represent multiple concepts.

[0233] If a movable object is not identified in the image frame (S1230-N), the electronic device (100) may identify the application as a third application type (S1245). The third application type may be a default type. The default type may be a type predefined according to a user's settings. The default type may be a type determined by default when the type of the application is not clearly identified.

[0234] Once the basic type is identified, the electronic device (100) can identify a third key structure (S1250). The third key structure may represent a structure including a movement key and a function key (selection key). The function key included in the third key structure may be a key that performs a selection.

[0235] Once the key structure is identified, the electronic device (100) can identify (or acquire) key information including the key structure (S1255). The electronic device (100) can acquire information (key shape, key size, key location, key recognition method, or key mapping value) for indicating a key other than the key structure. A description related thereto is provided in FIG. 13.

[0236] According to various embodiments, when the first key structure or the third key structure is identified, the electronic device (100) can generate an operation UI based on the first key structure or the third key structure. This is because the mapping operation (selection, cancellation, etc.) corresponding to the function key is clearly defined. However, when the second key structure is identified, additional operations may be required to generate the operation UI because the mapping operation corresponding to the function key is not defined.

[0237] FIG. 13 is a diagram for explaining an operation of identifying key information according to one embodiment.

[0238] Step S1355 of Fig. 13 may correspond to step S1255 of Fig. 12. Duplicate explanation is omitted.

[0239] The electronic device (100) can identify key information including a key structure. The electronic device (100) may require additional information representing the key in addition to the key structure.

[0240] The electronic device (100) can identify whether operation guide information corresponding to the application has been acquired (S1360). The operation guide information may be information indicating the operation UI used to execute the application. The operation guide information may include key information used to control the application. An example related to operation guide information is described in FIG. 15.

[0241] For example, operation guide information can be provided directly in the application.

[0242] For example, operation guide information may be stored in advance in the electronic device (100). The electronic device (100) may obtain operation guide information corresponding to an application from among a plurality of pre-stored operation guide information.

[0243] For example, operation guide information may be received through an external server. The electronic device (100) may request operation guide information corresponding to an application from the external server and receive the operation guide information from the external server.

[0244] When it is identified that the operation guide information has been acquired (S1360-Y), the electronic device (100) can identify key information including at least one of a key shape, a key size, a key position, and a key recognition method or a key mapping value based on the operation guide information (S1365).

[0245] If the operation guide information corresponding to the application is not obtained (S1360-N), the electronic device (100) can identify whether the log information corresponding to the application is obtained (S1370).

[0246] When log information corresponding to the application is acquired (S1370-Y), the electronic device (100) can acquire key usage frequencies for multiple keys based on the log information (S1375). The key usage frequency may be information indicating which keys the user actually used while the application was running. A related description is provided in FIG. 16.

[0247] The electronic device (100) can identify key information including at least one of a key shape, a key size, a key location, a key recognition method, and a key mapping value based on the key usage frequency (S1380).

[0248] For example, the electronic device (100) can identify a function key with a key usage frequency greater than a threshold ratio as a first shape.

[0249] For example, the electronic device (100) can identify a function key whose key usage frequency is greater than a threshold ratio as a first size. The electronic device (100) can obtain key information that the larger the key size, the greater the key usage frequency.

[0250] For example, the electronic device (100) may identify a function key with a key usage frequency exceeding a threshold ratio as the first position. The electronic device (100) may place the key with the highest usage frequency in a preset area (the area facing downward) of the operation UI that is easy for the user to touch. The electronic device (100) may determine the key arrangement order based on the order of usage frequency.

[0251] If log information corresponding to the application is not acquired (S1370-N), the electronic device (100) can identify basic key information (S1385). Basic key information may be key information determined regardless of operation guide information, log information, etc. Basic key information may include at least one of key mapping values ​​among key shapes, key sizes, key positions, and key recognition methods frequently used by users.

[0252] FIG. 14 is a drawing for explaining an application type according to one embodiment.

[0253] Referring to FIG. 14, the electronic device (100) can determine the application type by analyzing the image frame provided by the application. The electronic device (100) can display the image frame that constitutes the screen related to the application. The electronic device (100) can analyze the displayed image frame to determine whether the currently displayed image frame (screen) is provided by a certain type of application. The electronic device (100) can determine the application type based on at least one of the UI structure or the UI pattern in the image frame.

[0254] The structure of the UI can indicate the layout structure and layout location of the UIs included in the image frame.

[0255] UI patterns can represent repeated positions of UIs included in an image frame, repeated shapes of UIs, etc.

[0256] There may be various ways to analyze an image frame.

[0257] For example, the electronic device (100) can determine the application type by analyzing a single image frame provided by an application. The electronic device (100) can determine the type of a specific application by analyzing a single screen (image frame) related to the specific application. The screen to be analyzed may be the initial screen or a screen at a preset point in time. The preset point in time may be changed according to a user command or user settings.

[0258] For example, the electronic device (100) can determine the application type by analyzing multiple image frames provided to a single application. The electronic device (100) can receive multiple image frames provided by a specific application over time. The electronic device (100) can determine the type of a specific application by analyzing multiple screens (image frames) provided at different times.

[0259] For example, the electronic device (100) can determine the application type by analyzing at least one image frame provided by multiple applications. The electronic device (100) can analyze at least one screen (image frame) displayed on the display (140), etc. The screens displayed on the display (140) may be screens provided by multiple applications. The electronic device (100) can analyze the image frames provided by each of the multiple applications and determine the type of application that provided the last displayed image frame. There may be cases where a user executes various applications according to a certain routine. The electronic device (100) can determine the type of application that provides the final screen by comprehensively considering the screens provided by the multiple applications.

[0260] For example, assume that a first application provides a first image frame, and that a second application provides a second image frame after the first image frame has been provided. The electronic device (100) can analyze both the first image frame and the second image frame to determine the type of the second application.

[0261] An embodiment (1410) of FIG. 14 illustrates that an application of the content viewing service type is provided on an electronic device (100). The electronic device (100) can display the screen of the embodiment (1410) by executing the application of the content viewing service type. The electronic device (100) can identify that the application is of the content viewing service type by using the screen information of the embodiment (1410). The electronic device (100) can determine that the application is of the content viewing service type by identifying a selectable object (item object) in an image frame provided by the application.

[0262] The screens provided by applications of the content viewing service type may vary depending on the application.

[0263] For example, a screen provided by an application of the content viewing service type may include a UI corresponding to the content (e.g., a thumbnail screen).

[0264] For example, a screen provided by an application of the content viewing service type may include a UI corresponding to the content (e.g., a thumbnail) and a UI indicating a specific function (e.g., an external device connection function, an external application connection function).

[0265] A content viewing service type may represent an application type that provides services for users to view multimedia content, such as videos. Content viewing service types may be described as media viewing service types, content streaming service types, online content service types, multimedia service types, digital video streaming service types, or video viewing service types.

[0266] An embodiment (1420) of FIG. 14 illustrates that an application of the game service type is provided on an electronic device (100). The electronic device (100) can display the screen of the embodiment (1420) by executing the application of the game service type. The electronic device (100) can identify that the application is of the game service type by using the screen information of the embodiment (1420). The electronic device (100) can determine that the application is of the game service type by identifying a movable object (such as a person or a vehicle) in an image frame provided by the application.

[0267] For example, the electronic device (100) can determine the type of application as a game service type when a threshold number of objects moving in real time in an image frame are identified.

[0268] For example, if a preset number of objects (cars, airplanes, guns, people, virtual characters) is identified more than a threshold number (or less than a threshold number), the electronic device (100) may determine the type of application as a game service type. Since preset objects may also be identified in the content viewing service type, the electronic device (100) may additionally check the structure of the UI or the pattern of the UI in addition to the preset objects. If the arrangement positions of the preset objects are not in a consistent pattern, the electronic device (100) may determine the type of application as a game service type.

[0269] For example, the electronic device (100) can analyze multiple image frames received through a specific application and determine the application type as a game service type when the position of a preset object changes.

[0270] A game service type can represent an application that provides game services to users. Game service types can be described as online game service types, cloud game service types, digital game service types, game streaming service types, mobile game service types, virtual reality service types, or augmented reality service types.

[0271] FIG. 15 is a diagram for explaining key usage frequency based on operation guide information according to one embodiment.

[0272] The embodiment (1510) of FIG. 15 may display operation guide information. The operation guide information may include key information of the operation pad for controlling a specific application. The operation guide information may include various information related to the operation pad. The operation guide information may include at least one of the key structure, key shape, key size, key location, key recognition method, or key mapping value of the operation pad.

[0273] FIG. 16 is a diagram for explaining log information according to one embodiment.

[0274] Embodiment 16 (1610) of FIG. 16 may display log information. The log information may include various information related to the execution or use of an application. The log information may include at least one of the execution time, execution command, execution content, type of input key, number of key inputs, and key input time.

[0275] The electronic device (100) can identify the operation pad corresponding to the application and obtain log information of the operation pad. For example, the log information of the operation pad may only include the key type and the number of key inputs.

[0276] The electronic device (100) can obtain key types and key input counts using log information. The electronic device (100) can obtain usage frequency (or usage frequency information) based on the types and input counts of keys input by the user.

[0277] Referring to Example 1610, the key most frequently used by a user during a preset time may be a direction key (movement key). The second most frequently used key may be A. The third most frequently used key may be B.

[0278] The electronic device (100) can generate an operation UI using the key usage frequency. The electronic device (100) can generate an operation UI that includes only the directional keys, A key, and B key whose usage frequency is greater than a threshold value. The electronic device (100) can generate an operation UI such that among the function keys (A key, B key) excluding the directional keys, the A key, which has a higher usage frequency, is placed lower than the B key.

[0279] FIG. 17 is a drawing for explaining touch settings of an operation UI according to one embodiment.

[0280] The embodiment (1710) of FIG. 17 may represent the operation UI before being transformed in a preset manner.

[0281] The embodiment (1720) of Fig. 17 may represent an operation UI modified in a preset manner. The electronic device (100) may change the touch recognition method of a preset key or a key existing in a preset area.

[0282] For example, the electronic device (100) can change the key recognition method of preset keys (LR, LB, L3, R3, RT, RB) from touch to swipe.

[0283] For example, the electronic device (100) can change the key recognition method of keys (LR, LB, L3, R3, RT, RB) existing in a preset area from a touch method to a swipe method. Keys (1721) and (1723) can be indicated as being recognized in a swipe form.

[0284] The electronic device (100) can acquire keys whose frequency of use exceeds a threshold ratio. The electronic device (100) can change the touch recognition range of keys whose frequency of use exceeds the threshold ratio. To enable frequently used keys to be recognized more easily, the electronic device (100) can change the touch recognition range of a first size to a second size larger than the first size. Range (1721) represents the touch recognition range changed to the second size.

[0285] FIG. 18 is a drawing for explaining an operation of changing touch sensitivity according to one embodiment.

[0286] Referring to embodiment 1810 of FIG. 18, the electronic device (100) can change the key recognition method using the user's log information. The electronic device (100) can obtain log information including the user's key type and key input location for a threshold period of time. The key input location can indicate the touch range input by the user to press a specific key.

[0287] The electronic device (100) can analyze log information to provide a key recognition method suitable for the user.

[0288] For example, the electronic device (100) may set the touch sensitivity (included in the key recognition method) for a movement key to a first range. The first range may represent a range (or distance) that a user must touch (or push) in order for the movement key to be recognized. The electronic device (100) may analyze log information to determine that the user's touch range is smaller than a threshold range (or threshold distance). The electronic device (100) may change the touch sensitivity to a second range that is smaller than the first range. The electronic device (100) may generate an operation UI based on the changed touch sensitivity.

[0289] Example (1820) may illustrate an operation UI with altered touch sensitivity. The electronic device (100) may be configured to have a higher touch sensitivity than before. Higher touch sensitivity may mean that a key is recognized even when moved within a smaller range (smaller distance).

[0290] An opposite case to the embodiment of Fig. 18 may be applied. The electronic device (100) may change the key recognition method to lower the touch sensitivity compared to the existing one.

[0291] FIG. 19 is a drawing for explaining an operation of changing the operation UI according to the inclination of the terminal device (200) according to one embodiment.

[0292] The embodiment (1910) of Fig. 19 may be intended to represent each axis based on the terminal device (200). It is assumed that the terminal device (200) is placed in a space including the x-axis, y-axis, and z-axis. The front of the terminal device (200) may be in the direction indicated by the arrow.

[0293] Rotation around the x-axis can be the roll direction. Rotation around the y-axis can be the pitch direction. Rotation around the z-axis can be the yaw direction.

[0294] The x-axis, y-axis, and z-axis can be written as the first axis, second axis, third axis, etc.

[0295] The roll direction, pitch direction, and yaw direction may be described as the first direction, the second direction, the third direction, etc. The direction may be described as the rotation direction.

[0296] The embodiment (1920) of FIG. 19 may represent a situation in which the terminal device (200) is rotated in the pitch direction. In a situation in which a user holds the terminal device (200), the terminal device (200) may be erected in a direction facing the x-axis. However, in a situation in which the user places the terminal device (200) on the floor, the terminal device (200) may be laid down in a direction facing the z-axis. It is assumed that the terminal device (200) is rotated in the pitch direction (or in the opposite direction of the pitch direction).

[0297] The embodiment (1930) of FIG. 19 may indicate that the operation UI is changed according to the inclination information of the terminal device (200). The terminal device (200) may obtain the inclination information of the terminal device (200) through a gyro sensor. The key position may be changed according to the degree of rotation in the pitch direction.

[0298] As the terminal device (200) moves in the pitch direction (or in the opposite direction of the pitch direction), the spacing between preset keys (e.g., movement keys, stick keys) can change.

[0299] In a situation where the terminal device (200) faces the x-axis, the distance between the first key (L stick key) and the second key (R stick key) may be the first distance (d1).

[0300] In a situation where the terminal device (200) faces the z-axis, the distance between the first key (L stick key) and the second key (R stick key) may be a second distance (d1) smaller than the first distance (d1).

[0301] When the terminal device (200) is rotated based on the pitch direction, the spacing of preset keys included in the operation UI may change.

[0302] The subject of the change action in the operation UI can be diverse.

[0303] For example, an operation of changing the operation UI according to the inclination information of the terminal device (200) can be performed independently by the terminal device (200).

[0304] For example, an operation of changing the operation UI according to the inclination information of the terminal device (200) may be performed in the electronic device (100). The terminal device (200) may transmit the inclination information to the electronic device (100). The electronic device (100) may receive the inclination information from the terminal device (200). The electronic device (100) may change the operation UI based on the received inclination information.

[0305] FIG. 20 is a drawing for explaining an operation of changing an operation UI using log information according to one embodiment.

[0306] An embodiment (2010) of FIG. 20 illustrates displaying an image frame provided by an application on an electronic device (100). A terminal device (200) may be connected to the electronic device (100). The terminal device (200) may display an operation UI provided by the electronic device (100). A user of the terminal device (200) may input a key through the operation UI. The user key input entered on the terminal device (200) may be transmitted to the electronic device (100). The electronic device (100) may display an image frame corresponding to the user key input.

[0307] The electronic device (100) can collect user key inputs for a threshold period of time. The electronic device (100) can obtain log information collected during the threshold period of time regarding information on user key inputs.

[0308] Log information may include key type, key input location, etc.

[0309] Embodiment 20 (2020) of FIG. 20 may display log information. The log information may indicate how many times a specific key was entered and from which location it was entered.

[0310] The embodiment (2030) of Fig. 20 can display a changed operation UI using log information. Based on the log information, the most touched key may be the X key. The electronic device (100) can change the key recognition method (key recognition range) or key size using the number of key inputs and the key input location.

[0311] For example, the more keys are entered, the more the operating UI can be changed to make the keys easier to recognize.

[0312] For example, the recognition position of an existing key may be changed based on the key input position.

[0313] FIG. 21 is a drawing for explaining a guide screen for changing the operation UI according to one embodiment.

[0314] The embodiment (2110) of Fig. 21 may represent a key (2111) for setting the operation UI. When a specific key (2111) is input by the user, detailed settings of the operation UI may be determined (or changed).

[0315] An embodiment (2120) of FIG. 21 illustrates an operation for setting an operation UI. When a key (2111) is input, the electronic device (100) can provide a guide screen for setting an operation UI.

[0316] The guide screen may include image information and text information that guide the user in operating the operation UI. Guide audio information may be output through the electronic device (100) along with the guide screen.

[0317] The electronic device (100) may change the color of a key that the user must touch to be different from other keys in order to emphasize the guide operation. The electronic device (100) may display a preset indicator within a threshold distance from the key that the user must touch to emphasize the guide operation.

[0318] FIG. 22 is a drawing for explaining an operation of changing an operation UI using log information according to one embodiment.

[0319] Embodiment 22 (2210) of FIG. 22 may illustrate changing the position of a key using log information.

[0320] As in the embodiment (2020) of FIG. 20, the electronic device (100) can collect log information. The electronic device (100) can analyze the user's usage pattern related to the user's operation UI through the log information.

[0321] The usage pattern may indicate a specific pattern related to the input of keys included in the operation UI. The electronic device (100) may identify a pattern in which a user inputs a key closer to the center point of the operation UI than the existing key location through analysis of the operation log.

[0322] The electronic device (100) can generate an operation UI in which the key position is positioned closer to the center point of the operation UI based on the analysis results (usage patterns) of the operation log. The electronic device (100) can provide the newly generated operation UI.

[0323] According to various embodiments, the change operation may be performed directly by the terminal device (200) instead of the electronic device (100).

[0324] FIG. 23 is a drawing for explaining an operation of displaying a guide screen related to an operation UI according to one embodiment.

[0325] Referring to FIG. 23, an electronic device (100) can be connected to a terminal device (200) (S2305). The electronic device (100) can generate an operation UI to be provided to the terminal device (200) (S2310). The electronic device (100) can display a first guide screen for selecting a posture for holding the terminal device (200) (S2315). The electronic device (100) can display a guide screen for selecting one posture among multiple postures. A description related to the first guide screen is described in the embodiment (2510) of FIG. 25.

[0326] The electronic device (100) can receive a user input for selecting a first posture (S2320). The electronic device (100) can identify that the first posture has been selected based on the user input entered through the first guide screen.

[0327] When the first posture is selected, the electronic device (100) can transmit the first touch expected location corresponding to the first posture to the terminal device (200) (S2325). The electronic device (100) may store the first touch expected location corresponding to the first posture in advance.

[0328] The electronic device (100) may display a second guide screen requesting execution of a guide displayed on the electronic device (100) (S2330). A description related to the second guide screen is described in the embodiment (2520) of FIG. 25.

[0329] The electronic device (100) can display a third guide screen to guide the first posture (S2335). A description related to the third guide screen is described in the embodiment (2610) of FIG. 26.

[0330] A user may follow specific guide actions while viewing a third guide screen displayed on the electronic device (100). The third guide screen may include information indicating a guide action to hold the terminal device (200) in a specific posture. The third guide screen may include information guiding the user to hold the terminal device (200) until a haptic interaction (e.g., vibration) occurs.

[0331] After the third guide screen is displayed, the terminal device (200) can obtain the user's first touch input location through the third guide screen (S2340). The first touch input location may be described as the first key input location. The first touch input location does not indicate a single specific location, but may include multiple locations touched by the user.

[0332] The terminal device (200) can identify whether the difference between the first touch expected position and the first touch input position is within a threshold range (S2345). If the difference between the first touch expected position and the first touch input position is within the threshold range (S2345), the terminal device (200) can provide a haptic interaction (e.g., vibration) (S2350). The terminal device (200) can transmit a notification indicating that the first posture has been completed to the electronic device (100) (S2355).

[0333] FIG. 24 is a drawing for explaining an operation of displaying a guide screen related to an operation UI according to one embodiment.

[0334] Step S2455 of Fig. 24 may correspond to step S2355 of Fig. 23. Duplicate explanation is omitted.

[0335] The electronic device (100) can receive a notification from the terminal device (200) indicating that the first posture has been completed. The electronic device (100) can obtain the second touch expected location of the first button included in the operation UI (S2460). The electronic device (100) can transmit the second touch expected location corresponding to the first button to the terminal device (200) (S2461).

[0336] The electronic device (100) may display a fourth guide screen to guide the first button (S2465). The fourth guide screen may include information guiding the user to touch the first button. A description of the fourth guide screen is provided in screens (2620, 2630) of FIG. 26.

[0337] After the fourth guide screen is displayed, the terminal device (200) can obtain the user's second touch input location through the fourth guide screen (S2470). The terminal device (200) can identify whether the difference between the second touch expected location and the second touch input location is within a threshold range (S2475).

[0338] If the difference between the second touch expected position and the second touch input position is within a threshold range (S2475-Y), the terminal device (200) may provide a haptic interaction (e.g., vibration) (S2480). The terminal device (200) may transmit at least one of a notification indicating that the first button has been touched or the second touch input position (S2485).

[0339] The electronic device (100) may receive at least one of a notification indicating that a first button has been touched from the terminal device (200) or a second touch input location. The electronic device (100) may determine whether to change the location of the first button based on the difference between the second touch expected location and the second touch input location. If the difference is within a threshold range, the electronic device (100) may not change the operation UI.

[0340] If the difference is greater than the threshold range, the electronic device (100) can update the operation UI by changing the position of the first button based on the difference between the second touch expected position and the second touch input position (S2490). The operation of updating the operation UI can be replaced with an operation of changing the key position. For example, the key position can be changed from the second touch expected position to the second touch input position.

[0341] An update action may include changing the key position. Since a new manipulation UI is created, the update action of the manipulation UI may be described as a creation action of the manipulation UI.

[0342] The electronic device (100) can transmit the operation UI to the terminal device (200) (S2495). The terminal device (200) can receive the operation UI from the electronic device (100) and display the operation UI.

[0343] FIG. 25 is a drawing for explaining an operation of displaying a guide screen related to an operation UI according to one embodiment.

[0344] The embodiment (2510) of FIG. 25 may represent a first guide screen displayed on an electronic device (100). The first guide screen may be a screen for guiding a user's preferred posture. The electronic device (100) may display the first guide screen to guide the user to select one of multiple postures.

[0345] An embodiment (2520) of FIG. 25 may represent a second guide screen displayed on an electronic device (100). The second guide screen may include information requesting a specific action to be taken following a guide displayed on the electronic device (100).

[0346] FIG. 26 is a drawing for explaining an operation of displaying a guide screen related to an operation UI according to one embodiment.

[0347] An embodiment (2610) of FIG. 26 may represent a third guide screen displayed on an electronic device (100). The third guide screen may include guide information indicating to assume a specific posture. The guide information may include at least one of text information or image information indicating the guide posture. The electronic device (100) may output audio information requesting to assume a specific posture along with the third guide screen. The user may follow the third guide screen. The user assuming a specific posture may mean that the user touches a specific location (or a specific area). The terminal device (200) may output a haptic interaction (e.g., vibration) when a suitable location corresponding to the guide action is touched.

[0348] According to various embodiments, the first guide screen for the posture selection operation according to embodiment (2510) of FIG. 25 may not be displayed. The electronic device (100) may display a third guide screen for a pre-designated posture. The user may be requested to assume a specific posture according to the third guide screen displayed on the electronic device (100) without selecting a separate posture.

[0349] Screen (2620) of FIG. 26 may be a fourth guide screen that guides the user to click a specific button. The fourth guide screen may include indicators (e.g., outline highlighting, specific area notification icons, etc.) to highlight the location of a specific button. The electronic device (100) may display an area corresponding to a specific button in a preset color to highlight the location of the specific button.

[0350] FIG. 27 is a drawing for explaining an operation for changing an operation UI according to one embodiment.

[0351] Steps S2710, S2721, S2722, and S2723 of FIG. 27 may correspond to steps S1110, S1121, S1122, and S1123 of FIG. 11. Duplicate explanations are omitted.

[0352] Referring to FIG. 27, after identifying key information, the electronic device (100) can generate a first operation UI corresponding to the key information (S2724). The electronic device (100) can display a guide screen for changing the key information corresponding to the first operation UI (S2725). A description of the guide screen is provided in FIG. 28.

[0353] The electronic device (100) can receive user input through a guide screen (S2726).

[0354] For example, user input may be received through the electronic device (100). User input may be received through a remote control device corresponding to the electronic device (100).

[0355] For example, user input may be received through a terminal device (200). The terminal device (200) may receive user key input and transmit it to the electronic device (100).

[0356] The electronic device (100) can generate a second operation UI by changing the first operation UI based on user input (S2727).

[0357] The change operation may include an operation that changes at least one of the key structure, key shape, key size, key position, key recognition method, or key mapping value.

[0358] The electronic device (100) can transmit a second manipulation UI to the terminal device (200). The terminal device (200) can receive the second manipulation UI. The terminal device (200) can display the second manipulation UI.

[0359] FIG. 28 is a drawing for explaining an operation for changing an operation UI according to one embodiment.

[0360] The screen (2800) of Fig. 28 may be a guide screen for changing the operation UI. The guide screen (2800) may include information about a previously determined operation UI (first operation UI).

[0361] The guide screen (2800) may include at least one of a UI (2810) indicating a selected application, a UI (2820) indicating information for notifying that an operation UI is displayed and / or information for guiding a setting change, a UI (2830) guiding a method for changing a location, a UI (2840) indicating an image corresponding to a previously determined first operation UI, and a UI (2850) indicating key information including a key location, key mapping value, etc. for the previously determined first operation UI.

[0362] FIG. 29 is a diagram for explaining an operation of receiving screen information through a server according to one embodiment.

[0363] Referring to FIG. 29, the electronic device (100) can execute an application (S2905). The electronic device (100) can request an application screen from the server (400) (S2910). The server (400) can generate first screen information in response to the request received from the electronic device (100) (S2915). The server (400) can transmit the first screen information to the electronic device (100) (S2920).

[0364] The electronic device (100) can receive first screen information from the server (400). The electronic device (100) can display the first screen information (S2925).

[0365] The electronic device (100) can identify the application type based on the first screen information (S2930). The electronic device (100) can identify key information corresponding to the application (S2935). The electronic device (100) can generate an operation UI based on the key information (S2940). The electronic device (100) can transmit the operation UI to the terminal device (200) (S2945).

[0366] The terminal device (200) can receive a manipulation UI from the electronic device (100). The terminal device (200) can display the manipulation UI (S2946). The terminal device (200) can obtain a user key input through the manipulation UI (S2950). The terminal device (200) can transmit the user key input to the electronic device (100) (S2955).

[0367] The electronic device (100) can receive a user key input from the terminal device (200). The electronic device (100) can obtain the user key input (S2960). The electronic device (100) can transmit the user key input to the server (400) (S2965).

[0368] The server (400) can receive a user key input from the electronic device (100). The server (400) can generate second screen information based on the user key input (S2970). The server (400) can transmit the second screen information to the electronic device (100) (S2975).

[0369] The electronic device (100) can receive second screen information from the server (400). The electronic device (100) can display the second screen information (S2980).

[0370] FIG. 30 is a drawing for explaining a control method of an electronic device (100) according to one embodiment.

[0371] Referring to FIG. 30, a method for controlling an electronic device includes a step of receiving a user input for selecting an application (S3005), a step of obtaining screen information provided by the application (S3010), a step of identifying the type of the application based on the screen information provided by the application (S3015), a step of identifying key information for controlling the application based on the type of the application (S3020), a step of generating an operation UI (User Interface) based on the key information (S3025), and a step of transmitting the operation UI to a terminal device (S3030).

[0372] The step of identifying the type of application (S3015) may include obtaining an image frame included in screen information, identifying at least one object included in the image frame, and determining the type of application based on the at least one object, and the step of identifying key information (S3020) may identify key information including a key structure corresponding to the type of application.

[0373] The step of identifying the type of the application (S3015) identifies the type of the application as a first type when at least one selectable object is identified, and the step of identifying key information (S3020) can obtain key information including a first key structure corresponding to the first type.

[0374] The selectable object is a preset object that represents an item that can be selected based on a user key input, the first type is a content viewing service type, and the first key structure may be a structure that includes a movement key, a first function key for selection, and a second function key for cancellation.

[0375] The step of identifying the type of the application (S3015) may identify the type of the application as a second type if a movable object among at least one object is identified, and the step of identifying key information (S3020) may obtain key information including a second key structure corresponding to the second type.

[0376] A movable object is a preset object whose displayed position can be changed based on user key input, the second type is a game service type, and the second key structure can be a structure including a movement key and a plurality of function keys to which functions are not mapped.

[0377] The step of identifying key information (S3020) may include obtaining operation guide information corresponding to an application or log information corresponding to an application, and identifying key information including key mapping values ​​of each of a plurality of function keys based on at least one of the operation guide information or the log information.

[0378] The step of identifying key information (S3020) may obtain operation guide information corresponding to the application, and identify key information including at least one of a key shape, a key size, a key position, a key recognition method, or a key mapping value based on the operation guide information.

[0379] The step of identifying key information (S3020) may include obtaining log information corresponding to an application, obtaining key usage frequency based on the log information, and identifying key information including at least one of a key shape, a key size, a key location, a key recognition method, or a key mapping value based on the key usage frequency.

[0380] The control method may include a step of obtaining inclination information of a terminal device, and a step of changing an operation UI when the terminal device is identified as having rotated in a preset direction based on the inclination information.

[0381] The methods according to the various embodiments of the present disclosure described above can be implemented in the form of an application that can be installed on an existing electronic device.

[0382] The methods according to the various embodiments of the present disclosure described above can be implemented only with a software upgrade or a hardware upgrade for an existing electronic device.

[0383] The various embodiments of the present disclosure described above may also be performed through an embedded server provided in an electronic device, or an external server of at least one of the electronic device and the display device.

[0384] According to an example embodiment of the present disclosure, the various embodiments described above may be implemented as software including instructions stored in a machine-readable storage medium that can be read by a machine (e.g., a computer). The device may include an electronic device according to the disclosed embodiments, which is a device that can call instructions stored in the storage medium and operate according to the called instructions. When the instructions are executed by a processor, the processor may directly or under the control of the processor use other components to perform a function corresponding to the instructions. The instructions may include code generated or executed by a compiler or interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' means that the storage medium does not contain signals and is tangible, but does not distinguish between data being stored semi-permanently or temporarily in the storage medium.

[0385] According to one embodiment of the present disclosure, the method according to the various embodiments described above may be provided as included in a computer program product. The computer program product may be traded as a commodity between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or online through an application store. In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0386] Each of the components (e.g., modules or programs) according to the various embodiments described above may be composed of a single or multiple entities, and some of the sub-components described above may be omitted, or other sub-components may be further included in various embodiments. Alternatively or additionally, some components (e.g., modules or programs) may be integrated into a single entity, which may perform the same or similar functions as those performed by each of the respective components prior to integration. Operations performed by modules, programs or other components according to various embodiments may be executed sequentially, in parallel, iteratively or heuristically, or at least some operations may be executed in a different order, omitted, or other operations may be added.

[0387] Although the preferred embodiments of the present disclosure have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above, and various modifications may be made by a person having ordinary skill in the art to which the present disclosure pertains without departing from the gist of the present disclosure as claimed in the claims, and such modifications should not be understood individually from the technical idea of ​​the present disclosure.

Claims

1. In electronic devices, memory; Communication interface; and comprising at least one processor; At least one processor of the above, When a user input for selecting an application is received, the screen information provided by the application is obtained, Identify the type of the application based on the screen information provided by the application, Identify key information for controlling the application based on the type of the application, Generate an operation UI (User Interface) based on the above key information, An electronic device that transmits the operation UI to a terminal device through the communication interface.

2. In paragraph 1, At least one processor of the above, Obtain the image frame included in the above screen information, Identifying at least one object contained in the image frame; determining the type of the application based on at least one object; An electronic device identifying said key information, said key information including a key structure corresponding to said type of said application.

3. In paragraph 2, At least one processor of the above, When a selectable object among at least one of the above objects is identified, the type of the above application is identified as a first type, An electronic device that obtains key information including a first key structure corresponding to the first type.

4. In paragraph 3, The above selectable objects are: A preset object that represents selectable items based on user key input. The above first type is, It is a content viewing service type, The above first key structure is, An electronic device having a structure including a movement key, a first function key for selection, and a second function key for cancel.

5. In paragraph 2, At least one processor of the above, If a movable object is identified among at least one of the above objects, the type of the application is identified as a second type, An electronic device that obtains key information including a second key structure corresponding to the second type.

6. In paragraph 5, The above movable objects are, A preset object whose displayed position can change based on user key input. The second type above is, It is a game service type, The above second key structure is, An electronic device having a structure including a plurality of function keys, the function of which is not mapped to a movement key.

7. In paragraph 6, At least one processor of the above, Obtaining operation guide information corresponding to the above application or log information corresponding to the above application, An electronic device that identifies the key information including the key mapping value of each of the plurality of function keys based on at least one of the operation guide information or the log information.

8. In paragraph 2, At least one processor of the above, Obtain operation guide information corresponding to the above application, An electronic device that identifies key information including at least one of a key shape, a key size, a key position, a key recognition method, or a key mapping value based on the above operation guide information.

9. In paragraph 2, At least one processor of the above, Obtain log information corresponding to the above application, Obtain the key usage frequency based on the above log information, An electronic device that identifies the key information including at least one of a key shape, a key size, a key position, a key recognition method, or a key mapping value based on the key usage frequency.

10. In paragraph 1, At least one processor of the above, Through the above communication interface, the inclination information of the terminal device is obtained, An electronic device that changes the operation UI when it is determined that the terminal device has rotated in a preset direction based on the above inclination information.

11. In a method for controlling an electronic device, When a user input for selecting an application is received, a step of obtaining screen information provided by the application; A step of identifying the type of the application based on screen information provided by the application; A step of identifying key information for controlling the application based on the type of the application; A step of generating an operation UI (User Interface) based on the above key information; and A control method, comprising: a step of transmitting the above-mentioned operation UI to a terminal device.

12. In paragraph 11, The step of identifying the type of the above application is: Obtain the image frame included in the above screen information, Identifying at least one object contained in the image frame; determining the type of the application based on at least one object; The step of identifying the above key information is: A control method for identifying the key information including a key structure corresponding to the type of the application.

13. In paragraph 12, The step of identifying the type of the above application is: When a selectable object among at least one of the above objects is identified, the type of the above application is identified as a first type, The step of identifying the above key information is: A control method for obtaining key information including a first key structure corresponding to the first type.

14. In paragraph 13, The above selectable objects are: A preset object that represents selectable items based on user key input. The above first type is, It is a content viewing service type, The above first key structure is, A control method, comprising a structure including a movement key, a first function key for selection, and a second function key for cancellation.

15. In paragraph 12, The step of identifying the type of the above application is: If a movable object is identified among at least one of the above objects, the type of the application is identified as a second type, The step of identifying the above key information is: A control method for obtaining key information including a second key structure corresponding to the second type.

Images