Electronic device and control method thereof

Abstract

This electronic device comprises: a memory for storing instructions; a communication interface for connecting to a terminal device; a display; and at least one processor including processing circuitry, wherein the instructions, when executed individually or collectively by the at least one processor, control the display to display a screen including a plurality of objects and a pointer user interface (UI), receive, through the communication interface, touch data from the terminal device operating in a first mode or motion data from the terminal device operating in a second mode, perform a function corresponding to the touch data when the touch data is received, and control the display so that the pointer UI is moved on the basis of the motion data when the motion data is received, and the terminal device operates in the second mode on the basis of a pre-configured event.

Description

Electronic device and method of controlling the same

[0001] The present disclosure relates to an electronic device and a control method thereof. More specifically, the present disclosure relates to an electronic device that is remotely controlled according to a command received from an external device and a control method thereof.

[0002] A dedicated remote control device can be used to remotely control an electronic device that includes a display. Users can control the electronic device via the remote control device. However, if the remote control device does not include a display, there is a problem with touch recognition.

[0003] If a terminal device (e.g., a smartphone) that recognizes a user's touch is present, user input can be received using touch technology. The terminal device can transmit a touch signal corresponding to the user input to the electronic device. The electronic device can then perform a function corresponding to the user input based on the touch signal. However, using only touch technology presents a problem: it is difficult to quickly move the pointer displayed on the electronic device.

[0004] One or more embodiments can improve the above-described problem, and one or more embodiments of the present disclosure can provide an electronic device and a control method thereof that receives motion data from a terminal device according to a preset event and moves a pointer UI based on the received motion data.

[0005] According to one or more embodiments, an electronic device includes at least one processor including a memory storing instructions, a communication interface connecting to a terminal device, a display, and processing circuitry, wherein the instructions, when individually or collectively executed by the at least one processor, control the display to display a screen including a plurality of objects and a pointer UI (User Interface), receive touch data from the terminal device operating in a first mode or receive motion data from the terminal device operating in a second mode through the communication interface, and when the touch data is received, control the display to move the pointer UI based on the motion data, and the terminal device operates in the second mode based on a preset event.

[0006] The above instructions, when individually or collectively executed by the at least one processor, receive a notification from the terminal device through the communication interface, the notification indicating that an application for controlling the electronic device has been executed, and when the notification is received, control the display to display the pointer UI.

[0007] The function corresponding to the above touch data may include at least one of a function for moving the pointer UI, a function for selecting an object corresponding to a location where the pointer UI is displayed among the plurality of objects, a function for displaying a home screen, a function for displaying a previous screen, or a function for scrolling the current screen.

[0008] The above touch data may include at least one of a drag signal, a tap signal, a long tap signal, a double tap signal, or an edge scroll signal.

[0009] The above preset event may include at least one of an event in which a preset area of ​​the display of the terminal device is touched, an event in which a floating UI displayed on the display of the terminal device is touched, an event in which one area of ​​the display of the terminal device is touched for a threshold period of time or longer, an event in which a preset physical button of the terminal device is touched, and an event in which a preset UI displayed on the display of the terminal device is touched.

[0010] The instructions, when individually or collectively executed by the at least one processor, may perform a function corresponding to the touch data when the touch data including the user input received through the first area of ​​the display of the terminal device is received, and may control the display to move the pointer UI based on the motion data when the motion data is received based on the user input received through the second area of ​​the display of the terminal device.

[0011] When the above instructions are individually or collectively executed by the at least one processor, when the motion data obtained while a user input for selecting a floating UI displayed on the display of the terminal device is received, the display can be controlled to move the pointer UI based on the motion data.

[0012] The above motion data may include gyroscope data obtained through a gyro sensor included in the terminal device or acceleration data obtained through an acceleration sensor included in the terminal device.

[0013] The instructions, when individually or collectively executed by the at least one processor, adjust the movement speed of the pointer UI based on movement pattern information of the pointer UI stored in the memory, and the movement pattern information of the pointer UI may include information indicating a pattern in which the pointer UI is moved to select an object.

[0014] The above instructions, when individually or collectively executed by the at least one processor, can control the display to display identification information corresponding to the terminal device together with the pointer UI.

[0015] According to one or more embodiments, a method of controlling an electronic device connected to a terminal device includes the steps of displaying a screen and a pointer UI (User Interface) including a plurality of objects, receiving touch data from the terminal device operating in a first mode or receiving motion data from the terminal device operating in a second mode, and when the touch data is received, controlling the pointer UI to move based on the motion data when the motion data is received from the terminal device, wherein the terminal device operates in the second mode based on a preset event.

[0016] The above control method may include a step of receiving a notification from the terminal device, the notification indicating that an application for controlling the electronic device has been executed, and a step of displaying the pointer UI when the notification is received.

[0017] The function corresponding to the above touch data may include at least one of a function for moving the pointer UI, a function for selecting an object corresponding to a location where the pointer UI is displayed among the plurality of objects, a function for displaying a home screen, a function for displaying a previous screen, or a function for scrolling the current screen.

[0018] The above touch data may include at least one of a drag signal, a tap signal, a long tap signal, a double tap signal, or an edge scroll signal.

[0019] The above preset event may include at least one of an event in which a preset area of ​​the display of the terminal device is touched, an event in which a floating UI displayed on the display of the terminal device is touched, an event in which one area of ​​the display of the terminal device is touched for a threshold period of time or longer, an event in which a preset physical button of the terminal device is touched, and an event in which a preset UI displayed on the display of the terminal device is touched.

[0020] The step of performing a function corresponding to the touch data may include performing a function corresponding to the touch data when the touch data including a user input received through the first area of ​​the display of the terminal device is received, and the step of controlling the pointer UI to move based on the motion data may include controlling the pointer UI to move based on the motion data when the motion data is received based on a user input received through the second area of ​​the display of the terminal device.

[0021] The step of controlling the pointer UI to move based on the motion data may include controlling the pointer UI to move based on the motion data when the motion data obtained while a user input for selecting a floating UI displayed on the display of the terminal device is received.

[0022] The above motion data may include gyroscope data obtained through a gyro sensor included in the terminal device or acceleration data obtained through an acceleration sensor included in the terminal device.

[0023] The control method includes a step of adjusting a movement speed of the pointer UI based on movement pattern information of the pointer UI stored in the electronic device, and the movement pattern information of the pointer UI may include information indicating a pattern in which the pointer UI is moved to select an object.

[0024] The above control method may include a step of displaying identification information corresponding to the terminal device together with the pointer UI.

[0025] The embodiments will be more clearly understood through the detailed description set forth below with reference to the accompanying drawings.

[0026] FIG. 1 is a drawing for explaining an operation of controlling an electronic device through a terminal device according to one or more embodiments.

[0027] FIG. 2 is a block diagram illustrating an electronic device according to one or more embodiments.

[0028] FIG. 3 is a block diagram illustrating a specific configuration of the electronic device of FIG. 2 according to one or more embodiments.

[0029] FIG. 4 is a diagram illustrating an operation of displaying a pointer UI on an electronic device according to one or more embodiments.

[0030] FIG. 5 is a diagram illustrating an operation of displaying a pointer UI on an electronic device according to one or more embodiments.

[0031] FIG. 6 is a diagram illustrating an operation of displaying a pointer UI on an electronic device according to one or more embodiments.

[0032] FIG. 7 is a diagram for explaining an operation of controlling an electronic device through a terminal device according to one or more embodiments.

[0033] FIG. 8 is a diagram illustrating an operation of displaying a pointer UI on an electronic device according to one or more embodiments.

[0034] FIG. 9 is a diagram illustrating an operation of displaying a pointer UI after a preset event is identified, according to one or more embodiments.

[0035] FIG. 10 is a diagram for explaining an operation of connecting an electronic device and a terminal device according to one or more embodiments.

[0036] FIG. 11 is a diagram for explaining an operation of connecting a pre-registered electronic device and a terminal device according to one or more embodiments.

[0037] FIG. 12 is a drawing for explaining a touch input table according to one or more embodiments.

[0038] FIG. 13 is a diagram illustrating an operation of an electronic device performing a function corresponding to touch data according to one or more embodiments.

[0039] FIG. 14 is a diagram for explaining an operation of a terminal device identifying a function corresponding to touch data according to one or more embodiments.

[0040] FIG. 15 is a diagram illustrating an operation of moving a pointer UI based on touch data according to one or more embodiments.

[0041] FIG. 16 is a diagram illustrating an operation of moving a pointer UI based on touch data according to one or more embodiments.

[0042] FIG. 17 is a drawing for explaining an operation of moving a pointer UI by touching a preset area according to one or more embodiments.

[0043] FIG. 18 is a drawing for explaining an operation of moving a pointer UI by touching a preset area according to one or more embodiments.

[0044] FIG. 19 is a diagram for explaining an operation of processing touch data and an operation of processing motion data according to one or more embodiments.

[0045] FIG. 20 is a diagram illustrating a plotting UI according to one or more embodiments.

[0046] FIG. 21 is a diagram for explaining an operation of controlling a pointer UI displayed on an electronic device through a floating UI according to one or more embodiments.

[0047] FIG. 22 is a diagram illustrating displaying a pointer UI based on a preset time-recognized touch according to one or more embodiments.

[0048] FIG. 23 is a diagram illustrating displaying a pointer UI based on a preset time-recognized touch according to one or more embodiments.

[0049] FIG. 24 is a drawing for explaining an operation of moving a pointer UI using a physical button of a terminal device according to one or more embodiments.

[0050] FIG. 25 is a drawing for explaining an operation of moving a pointer UI using a physical button of a terminal device according to one or more embodiments.

[0051] FIG. 26 is a diagram illustrating an operation of moving a pointer UI by identifying a preset event according to one or more embodiments.

[0052] FIG. 27 is a diagram illustrating an operation of moving a pointer UI by identifying a preset event according to one or more embodiments.

[0053] FIG. 28 is a drawing for explaining a mode of a terminal device according to one or more embodiments.

[0054] FIG. 29 is a drawing for explaining a movable state of a plotting UI according to one or more embodiments.

[0055] FIG. 30 is a drawing for explaining an operation of displaying a pointer UI corresponding to an area of ​​a terminal device touched by a user according to one or more embodiments.

[0056] FIG. 31 is a diagram illustrating a user pattern for moving a pointer UI according to one or more embodiments.

[0057] FIG. 32 is a diagram illustrating a user pattern for moving a pointer UI according to one or more embodiments.

[0058] FIG. 33 is a diagram for explaining an operation of changing the movement speed of a pointer UI based on the number of objects, according to one or more embodiments.

[0059] FIG. 34 is a diagram for explaining an operation of controlling the movement speed of a pointer UI based on the distance between an electronic device and a terminal device according to one or more embodiments.

[0060] FIG. 35 is a drawing for explaining a user's grab method according to one or more embodiments.

[0061] FIG. 36 is a drawing for explaining an operation of changing the position of a floating UI according to one or more embodiments.

[0062] FIG. 37 is a drawing for explaining an embodiment in which a terminal device and a remote control device are simultaneously recognized by an electronic device according to one or more embodiments.

[0063] FIG. 38 is a drawing for explaining an embodiment in which multiple terminal devices are simultaneously recognized by an electronic device according to one or more embodiments.

[0064] FIG. 39 is a drawing for explaining an embodiment in which multiple terminal devices are simultaneously recognized by an electronic device according to one or more embodiments.

[0065] FIG. 40 is a drawing for explaining an operation of displaying a pointer UI differently depending on a terminal device according to one or more embodiments.

[0066] Figure 41 is a drawing for explaining an operation in which an electronic device receives touch data and motion data from a terminal device.

[0067] FIG. 42 is a drawing for explaining a method of controlling an electronic device according to one or more embodiments.

[0068] Hereinafter, the present disclosure will be described in detail with reference to the attached drawings.

[0069] The terms used in the embodiments of this disclosure have been selected from widely used, current terms, taking into account the functions of this disclosure. However, these terms may vary depending on the intentions of those skilled in the art, precedents, the emergence of new technologies, etc. Furthermore, in certain cases, terms may be arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the description of the relevant disclosure. Therefore, the terms used in this disclosure should not be defined simply as names of terms, but rather based on the meanings of the terms and the overall content of this disclosure.

[0070] In this specification, expressions such as “has,” “can have,” “includes,” or “may include” indicate the presence of a feature (e.g., a number, function, operation, or component such as a part), and do not exclude the presence of additional features.

[0071] The expression "at least one of A and / or B" should be understood to mean either "A" or "B" or "A and B".

[0072] As used herein, the expressions “first,” “second,” “first,” or “second,” etc., may describe various components, regardless of order and / or importance, and are only used to distinguish one component from another, but do not limit the components.

[0073] When it is said that a component (e.g., a first component) is “(operatively or communicatively) coupled with / to” or “connected to” another component (e.g., a second component), it should be understood that the component may be directly coupled to the other component, or may be coupled via another component (e.g., a third component).

[0074] Singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, terms such as "comprise" or "consist of" are intended to indicate the presence of a feature, number, step, operation, component, part, or combination thereof described in the specification, but should be understood not to preclude the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0075] In the present disclosure, a "module" or "part" performs at least one function or operation and may be implemented in hardware or software, or a combination of hardware and software. Furthermore, multiple "modules" or multiple "parts" may be integrated into at least one module and implemented as at least one processor, excluding any "modules" or "parts" that need to be implemented as specific hardware.

[0076] In this specification, the term user may refer to a person using an electronic device or a device using an electronic device (e.g., an artificial intelligence electronic device).

[0077] One or more embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings.

[0078] FIG. 1 is a drawing for explaining an operation of controlling an electronic device (100) through a terminal device (200) according to one or more embodiments.

[0079] Referring to FIG. 1, an electronic device (100) and a terminal device (200) can be connected to each other.

[0080] The electronic device (100) may include various devices including a display. The electronic device (100) may be an electronic whiteboard, a TV, a desktop PC, a laptop, a smartphone, a tablet PC, a server, etc. The above-described examples are merely examples for describing the electronic device and are not necessarily limited to the above-described devices.

[0081] The electronic device (100) may be a device including a display. The electronic device (100) may be a device that outputs content including images. The electronic device (100) may be described as a display device, a display device, or the like. For example, the electronic device (100) may be a TV, a smart TV, or the like.

[0082] The terminal device (200) may be a device that remotely controls the electronic device (100). The terminal device (200) may be described as a remote control device.

[0083] According to one or more embodiments, the terminal device (200) may be a device including a display. The present disclosure will be described based on a terminal device (200) including a display. The terminal device (200) may be described as a user device or a wearable device, etc. For example, the terminal device (200) may be described as a smartphone, a smartwatch, etc.

[0084] According to one or more embodiments, the terminal device (200) may be a device that does not include a display. The terminal device (200) may be a remote control device that controls the electronic device (100). The terminal device (200) may be a dedicated remote control device for the electronic device (100). An embodiment related to a dedicated remote control device is described in FIG. 37.

[0085] There may be various ways in which the electronic device (100) and the terminal device (200) communicate with each other.

[0086] As one or more examples, the electronic device (100) and the terminal device (200) may communicate using a Bluetooth communication method. The Bluetooth communication method may include a low-power Bluetooth communication method.

[0087] As one or more examples, the electronic device (100) and the terminal device (200) can communicate using an infrared communication method.

[0088] As one or more examples, the electronic device (100) and the terminal device (200) can communicate using a Wi-Fi communication method.

[0089] FIG. 2 is a block diagram illustrating an electronic device (100) according to one or more embodiments.

[0090] Referring to FIG. 2, the electronic device (100) may include a memory (110) that stores instructions, a communication interface (130) that connects to a terminal device (200), a display (140), and at least one processor (120) that includes processing circuitry.

[0091] At least one processor (120) can perform overall control operations of the electronic device (100). At least one processor (120) can perform a function of controlling overall operations of the electronic device (100).

[0092] The instructions, when individually or collectively executed by at least one processor, may perform the following actions:

[0093] At least one processor (120) controls the display (140) to display a screen and pointer UI (10) including a plurality of objects, and when touch data is received from a terminal device (200) operating in a first mode for transmitting touch data, performs a function corresponding to the touch data, and when motion data is received from a terminal device (200) operating in a second mode for transmitting motion data based on a preset event, controls the display (140) to move the pointer UI (10) based on the motion data. According to one or more embodiments, the at least one processor (120) may receive touch data from a terminal device (200) operating in the first mode through a communication interface (130). According to one or more embodiments, the at least one processor (120) may receive motion data from a terminal device (200) operating in the second mode through a communication interface (130).

[0094] At least one processor (120) can display a screen including at least one object on the display (140). At least one processor (120) can display a screen including a pointer UI (10) together with at least one object on the display (140).

[0095] The pointer UI (10) may be a UI displayed to control the electronic device (100) with the terminal device (200). The pointer UI (10) may be a UI for indicating a location corresponding to a user input. The pointer UI (10) may be a UI displayed to select at least one object displayed on the display (140). The pointer UI (10) may be described as a cursor UI, a movement UI, a selection UI, an indicator, etc.

[0096] At least one processor (120) can control the display (140) to display a pointer UI (10) when a notification is received indicating that an application for controlling an electronic device (100) is executed in the terminal device (200).

[0097] The operation of the terminal device (200) executing the application is described in FIGS. 4 to 7. The operation of displaying the pointer UI (10) is described in FIGS. 8 and 9.

[0098] The operation of connecting the electronic device (100) and the terminal device (200) is described in FIGS. 10 and 11.

[0099] The function corresponding to the touch data may include at least one of a function for moving the pointer UI (10), a function for selecting an object corresponding to the position where the pointer UI (10) is displayed among a plurality of objects, a function for displaying the home screen, a function for displaying the previous screen, or a function for scrolling the current screen.

[0100] As one or more examples, the touch data may include at least one of a drag signal, a tap signal, a long tap signal, a double tap signal, or an edge scroll signal. The touch data may include information defining various signals corresponding to user input.

[0101] At least one processor (120) can store a touch input table in the memory (110). At least one processor (120) can perform a function corresponding to received touch data based on the touch input table. A description thereof is provided in FIGS. 12 to 14.

[0102] At least one processor (120) can move the pointer UI (10) based on touch data. The touch data may include a drag signal. At least one processor (120) can move the pointer UI (10) based on the drag signal. Operations related thereto are described in FIGS. 15 and 16.

[0103] The preset event may include at least one of an event in which a preset area among the entire area of ​​the display (140) of the terminal device (200) is touched, an event in which a floating UI (20) displayed on the display (140) of the terminal device (200) is touched, an event in which any area among the entire area of ​​the display (140) of the terminal device (200) is touched for a threshold period of time, an event in which a preset physical button of the terminal device (200) is touched, and an event in which a preset UI displayed on the display (140) of the terminal device (200) is touched.

[0104] The expression 'touched' can be written as 'selected'.

[0105] The actions for events in which a preset area is touched are described in FIGS. 17 to 19.

[0106] The operation for the event of touching the floating UI (20) is described in FIGS. 20 and 21.

[0107] Events in which any area is touched for a critical time or longer are described in FIGS. 22 and 23.

[0108] The event of a preset physical button being touched is described in FIGS. 24 and 25.

[0109] The terminal device (200) can transmit motion data to the electronic device (100) without touching the touch screen (the display screen of the terminal device (200)) or the physical button (the operating interface of the terminal device (200)). The operations related to this are described in FIGS. 26 and 27.

[0110] At least one processor (120) may perform a function corresponding to the touch data when touch data including a user input received through a first area among the entire area of ​​the display (140) of the terminal device (200) is received.

[0111] The first area may be an area designated for transmitting touch data among the entire touch area (entire display area) of the terminal device (200).

[0112] At least one processor (120) can control the display (140) to move the pointer UI (10) based on the motion data when motion data is received based on a user input received through a second area of ​​the entire area of ​​the display (140) of the terminal device (200).

[0113] The second area may be an area that receives user input for transmitting motion data among the entire touch area (entire display area) of the terminal device (200).

[0114] The operation of processing user input differently according to the first and second areas distinguished in the terminal device (200) is described in FIGS. 17 to 19.

[0115] At least one processor (120) can control the display (140) to move the pointer UI (10) based on the motion data when motion data obtained while a user input for selecting a floating UI (20) displayed on the display (140) of the terminal device (200) is received.

[0116] The floating UI (20) may be a UI for performing a second mode for transmitting motion data. The floating UI (20) may be a UI displayed on the display of the terminal device (200). The floating UI (20) may be a trigger UI for initiating an operation for transmitting motion data. The floating UI (20) may be described as a mode switching UI, a motion UI, a response UI, a pointer control UI, a gyro UI, etc.

[0117] The operation of moving the floating UI (20) is described in Fig. 29.

[0118] The operation of displaying the pointer UI (10) in different forms depending on the area where user input is received is described in FIG. 30.

[0119] Motion data may include gyro data acquired through a gyro sensor (or gyroscope sensor) included in the terminal device (200).

[0120] According to one or more embodiments, at least one processor (120) may acquire motion data including gyro data. The terminal device (200) may include a gyro sensor. The terminal device (200) may acquire gyro data through the gyro sensor. The gyro data may include information about the real-time rotation direction (two-dimensional or three-dimensional) of the terminal device (200).

[0121] According to one or more embodiments, at least one processor (120) may obtain motion data including gyro data (or gyroscope data) and acceleration data. The terminal device (200) may include an acceleration sensor. The terminal device (200) may obtain acceleration data through the acceleration sensor. The acceleration data may include information about real-time acceleration of the terminal device (200). The acceleration data may represent linear movement or acceleration of the terminal device (200). The gyroscope data may represent rotational movement or acceleration of the terminal device (200) about one or more axes.

[0122] At least one processor (120) can adjust the movement speed of the pointer UI (10) based on movement pattern information of the pointer UI (10) stored in the memory (110).

[0123] The movement pattern information of the pointer UI (10) may include information indicating a pattern in which the pointer UI (10) is moved to select an object. Descriptions related to the movement pattern information of the pointer UI (10) are described in FIGS. 31 to 34.

[0124] At least one processor (120) can control the display (140) to display identification information corresponding to the terminal device (200) together with the pointer UI (10).

[0125] The identification information may be information for indicating the terminal device (200). The identification information may include at least one of model information of the terminal device (200) and the name of the terminal device (200).

[0126] Identification information may include information that identifies a user. Identification information may include at least one of an ID, account, email address, name, initials, nickname, representative image, and avatar image.

[0127] The operation of displaying identification information can be described in Fig. 37. A situation in which multiple devices are connected is described in Figs. 37 to 40.

[0128] When motion data is used, the pointer UI (10) can be moved without having to drag the movement direction through the terminal device (200). When motion data is used, the pointer UI (10) can be easily moved, thereby increasing user convenience.

[0129] According to one or more embodiments, a preset event associated with an action of transmitting touch data or an action of transmitting motion data may be determined by the terminal device (200). The corresponding embodiment is described as a basic embodiment.

[0130] According to one or more embodiments, the electronic device (100) may determine whether to move the pointer UI (10) using motion data based on a preset event. A description related to this is provided in FIG. 41.

[0131] FIG. 3 is a block diagram illustrating a specific configuration of the electronic device (100) of FIG. 2 according to one or more embodiments.

[0132] Referring to FIG. 3, the electronic device (100) may include at least one of a memory (110), at least one processor (120), a communication interface (130), a display (140), an operation interface (150), an input / output interface (160), a speaker (170), a microphone (180), and a camera (190).

[0133] The memory (110) may be implemented as an internal memory such as a ROM (e.g., an electrically erasable programmable read-only memory (EEPROM)) or RAM included in at least one processor (120), or may be implemented as a separate memory from at least one processor (120). The memory (110) may be implemented as a memory embedded in the electronic device (100) or as a memory detachable from the electronic device (100) depending on the purpose of data storage. For example, data for driving the electronic device (100) may be stored in a memory embedded in the electronic device (100), and data for expanding functions of the electronic device (100) may be stored in a memory detachable from the electronic device (100).

[0134] In the case of memory embedded in the electronic device (100), it may be implemented as at least one of volatile memory (e.g., dynamic RAM (DRAM), static RAM (SRAM), or synchronous dynamic RAM (SDRAM)), non-volatile memory (e.g., one time programmable ROM (OTPROM), programmable ROM (PROM), erasable and programmable ROM (EPROM), electrically erasable and programmable ROM (EEPROM), mask ROM, flash ROM, flash memory (e.g., NAND flash or NOR flash), etc.), hard drive, or solid state drive (SSD), and in the case of memory that can be detachably attached to the electronic device (100), it may be implemented in the form of a memory card (e.g., compact flash (CF), secure digital (SD), micro secure digital (Micro-SD), mini secure digital (Mini-SD), extreme digital (xD), multi-media card (MMC), etc.), external memory that can be connected to a USB port (e.g., USB memory), etc.

[0135] The memory (110) can store at least one instruction. Based on the instruction stored in the memory (110), at least one processor (120) can perform various operations.

[0136] At least one processor (120) may be implemented as a digital signal processor (DSP), a microprocessor, or a time controller (TCON) that processes digital signals. However, the present invention is not limited thereto, and may include one or more of a central processing unit (CPU), a micro controller unit (MCU), a micro processing unit (MPU), a controller, an application processor (AP), a graphics-processing unit (GPU), a communication processor (CP), or an advanced reduced instruction set computer (RISC) machines (ARM) processor, or may be defined by the relevant terminology. At least one processor (120) may be implemented as a system on chip (SoC) having a built-in processing algorithm, a large scale integration (LSI), or may be implemented in the form of a field programmable gate array (FPGA). At least one processor (120) may perform various functions by executing computer executable instructions stored in a memory.

[0137] The communication interface (130) is a configuration that performs communication with various types of external devices according to various types of communication methods. The communication interface (130) may include a wireless communication module or a wired communication module. Each communication module may be implemented in the form of at least one hardware chip.

[0138] A wireless communication module may be a module that communicates wirelessly with an external device. For example, the wireless communication module may include at least one of a Wi-Fi module, a Bluetooth module, an infrared communication module, or other communication modules.

[0139] Wi-Fi and Bluetooth modules can communicate via Wi-Fi and Bluetooth, respectively. When using a Wi-Fi or Bluetooth module, various connection information, such as the service set identifier (SSID) and session key, is first transmitted and received. This information is then used to establish a communication connection before various other information can be transmitted and received.

[0140] Infrared communication modules perform communication based on infrared communication (IrDA, infrared Data Association) technology, which transmits data wirelessly over short distances using infrared light, which is between visible light and millimeter waves.

[0141] In addition to the above-described communication method, other communication modules may include at least one communication chip that performs communication according to various wireless communication standards such as zigbee, 3G (3rd Generation), 3GPP (3rd Generation Partnership Project), LTE (Long Term Evolution), LTE-A (LTE Advanced), 4G (4th Generation), 5G (5th Generation), etc.

[0142] A wired communication module may be a module that communicates with an external device via a wire. For example, the wired communication module may include at least one of a Local Area Network (LAN) module, an Ethernet module, a paired cable, a coaxial cable, a fiber optic cable, or an Ultra Wide-Band (UWB) module.

[0143] According to various embodiments, the communication interface (130) may utilize the same communication module (e.g., a Wi-Fi module) to communicate with an external device such as a remote control device and an external server.

[0144] According to various embodiments, the communication interface (130) may utilize different communication modules to communicate with external devices, such as remote control devices, and external servers. For example, the communication interface (130) may utilize at least one of an Ethernet module or a Wi-Fi module to communicate with an external server, and may also utilize a Bluetooth module to communicate with an external device, such as a remote control device. However, this is only one or more embodiments, and the communication interface (130) may utilize at least one of various communication modules when communicating with multiple external devices or external servers.

[0145] The display (140) may be implemented as a display of various forms, such as a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display panel (PDP), etc. The display (140) may also include a driving circuit, a backlight unit, etc., which may be implemented as a form, such as an a-si TFT (amorphous silicon thin film transistor), an LTPS (low temperature poly silicon) TFT, an OTFT (organic TFT), etc. The display (140) may be implemented as a touch screen combined with a touch sensor, a flexible display, a three-dimensional display (3D display, three-dimensional display), etc. According to one or more embodiments of the present disclosure, the display (140) may include not only a display panel that outputs an image, but also a bezel that houses the display panel. In particular, according to one or more embodiments of the present disclosure, the bezel may include a touch sensor for detecting user interaction.

[0146] The operating interface (150) may be implemented as a device such as a button, a touch pad, a mouse, and a keyboard, or as a touch screen capable of performing the above-described display function and operating input function. The button may be a mechanical button, a touch pad, a wheel, or any other type of button formed in any area of ​​the front, side, or back of the main body of the electronic device (100).

[0147] The input / output interface (160) may be any one of HDMI (High Definition Multimedia Interface), MHL (Mobile High-Definition Link), USB (Universal Serial Bus), DP (Display Port), Thunderbolt, VGA (Video Graphics Array) port, RGB port, D-SUB (D-subminiature), and DVI (Digital Visual Interface). The input / output interface (160) may input / output at least one of audio and video signals. Depending on the implementation example, the input / output interface (160) may include a port that inputs / outputs only audio signals and a port that inputs / outputs only video signals as separate ports, or may be implemented as a single port that inputs / outputs both audio signals and video signals. The electronic device (100) may transmit at least one of the audio and video signals to an external device (e.g., an external display device or an external speaker) through the input / output interface (160). An output port included in the input / output interface (160) can be connected to an external device, and the electronic device (100) can transmit at least one of an audio and video signal to the external device through the output port.

[0148] The input / output interface (160) can be connected to a communication interface. The input / output interface (160) can transmit information received from an external device to the communication interface or transmit information received through the communication interface to the external device.

[0149] The speaker (170) may be a component that outputs various audio data as well as various notification sounds or voice messages.

[0150] The microphone (180) is a component that receives a user's voice or other sounds and converts them into audio data. The microphone (180) can receive the user's voice in an activated state. For example, the microphone (180) can be formed integrally on the upper side, the front side, the side side, etc. of the electronic device (100). The microphone (180) can include various components such as a microphone that collects the user's voice in analog form, an amplifier circuit that amplifies the collected user's voice, an A / D conversion circuit that samples the amplified user's voice and converts it into a digital signal, and a filter circuit that removes noise components from the converted digital signal.

[0151] The camera (190) is a device configured to capture a subject and generate a captured image, and the captured image is a concept that includes both moving images and still images. The camera (190) can acquire images for at least one external device and can be implemented with a camera, lens, infrared sensor, etc.

[0152] The camera (190) may include a lens and an image sensor. The type of lens may include a general-purpose lens, a wide-angle lens, a zoom lens, etc., and may be determined according to the type, characteristics, usage environment, etc. of the electronic device (100). The image sensor may include a complementary metal oxide semiconductor (CMOS) and a charge-coupled device (CCD).

[0153] FIG. 4 is a drawing for explaining an operation of displaying a pointer UI (10) on an electronic device (100) according to one or more embodiments.

[0154] Referring to the embodiment (400) of FIG. 4, the terminal device (200) can execute an application for controlling the electronic device (100). The terminal device (200) can display a remote control screen on the display of the terminal device (200). The remote control screen can include at least one of a power control UI (401) or a voice input UI (402).

[0155] When a user input is received through the power control UI (401), the terminal device (200) can transmit a control command (on or off) related to power control to the electronic device (100).

[0156] When a user input is received through the voice input UI (402), the terminal device (200) can perform an operation related to the voice recognition function.

[0157] According to one or more embodiments, a voice recognition function may be performed in an electronic device (100). When a user input is received through a voice input UI (402), the terminal device (200) may transmit a control command for performing the voice recognition function to the electronic device (100). The electronic device (100) may perform the voice recognition function based on the control command received from the terminal device (200). The electronic device (100) may activate a microphone for voice recognition. The electronic device (100) may receive (or recognize) the user's voice through the activated microphone.

[0158] According to one or more embodiments, a voice recognition function may be performed in the terminal device (200). When a user input is received through the voice input UI (402), the terminal device (200) may perform the voice recognition function. The terminal device (200) may activate a microphone of the terminal device (200). The terminal device (200) may receive (or recognize) the user's voice through the activated microphone. The terminal device (200) may transmit the received user's voice to the electronic device (100).

[0159] The electronic device (100) can display a pointer UI (10) related to the control of the terminal device (200). The pointer UI (10) can be an indicator that moves under the control of the terminal device (200). In FIG. 4, the pointer UI (10) is described as having a circular shape, but is not limited thereto. The shape and form of the pointer UI (10) can vary.

[0160] The location where the pointer UI (10) is displayed can be changed according to a user's operation command input through the terminal device (200). The terminal device (200) can receive a user input (or user command) for moving the pointer UI (10). The terminal device (200) can transmit the received user input to the electronic device (100).

[0161] The operation of displaying the pointer UI (10) is described in FIGS. 8 and 9.

[0162] FIG. 5 is a drawing for explaining an operation of displaying a pointer UI (10) on an electronic device (100) according to one or more embodiments.

[0163] Referring to the embodiment (500) of FIG. 5, the terminal device (200) can execute an application for controlling the electronic device (100). The terminal device (200) can display a remote control screen on the display of the terminal device (200). The remote control screen can include at least one of a power control UI (501), a search UI (502), a favorites UI (503), a volume control UI (504), a channel control UI (505), a touch input UI (506), and a voice input UI (507). The arrangement structure of the UIs can be changed. A mute UI can be arranged instead of the voice input UI (507).

[0164] A terminal device (200) can be connected to an electronic device (100). When connected to the terminal device (200), the electronic device (100) can display a pointer UI (10). The electronic device (100) can display the pointer UI (10) at a preset location within the entire area of ​​the display (140).

[0165] FIG. 6 is a drawing for explaining an operation of displaying a pointer UI (10) on an electronic device (100) according to one or more embodiments.

[0166] Referring to the embodiment (600) of FIG. 6, the terminal device (200) can execute an application for controlling the electronic device (100). The terminal device (200) can display a remote control screen on the display of the terminal device (200). The remote control screen can include at least one of a power control UI (601), a search UI (602), a setting UI (603), a voice recognition UI (604), a screen size adjustment UI (605), a movement operation UI (606), a previous screen display UI (607), a home screen display UI (608), a playback UI (609), a volume control UI (610), a mute UI (611), and a favorites UI (612).

[0167] FIG. 7 is a drawing for explaining an operation of controlling an electronic device (100) through a terminal device (200) according to one or more embodiments.

[0168] The terminal device (200) can receive a first user input for executing an application related to controlling the electronic device (100) (S705). The terminal device (200) can execute an application for controlling the electronic device (100) in response to receiving the first user input (S710).

[0169] The terminal device (200) can display a screen provided by the application (S715). The terminal device (200) can receive a second user input for controlling the electronic device (100) through the screen (S720). The terminal device (200) can transmit the second user input to the electronic device (100) (S721).

[0170] The electronic device (100) can receive a second user input from the terminal device (200). The electronic device (100) can control the electronic device (100) based on the second user input (S722).

[0171] For clarity, the first user input may be described as a user input, and the second user input may be described as a user command. The ordinal numbers preceding the user input and user command may vary depending on the embodiment.

[0172] FIG. 8 is a drawing for explaining an operation of displaying a pointer UI (10) on an electronic device (100) according to one or more embodiments.

[0173] Steps S805, S810, S815, S820, S821, and S822 of FIG. 8 may correspond to steps S705, S710, S715, S720, S721, and S722 of FIG. 7. Duplicate explanations are omitted.

[0174] When the application is executed, the terminal device (200) can transmit a notification indicating that the application is executed to the electronic device (100) (S811).

[0175] The electronic device (100) can receive a notification indicating that an application is running on the terminal device (200). When a preset notification is received, the electronic device (100) can display a pointer UI (10). The electronic device (100) can display a screen including the pointer UI (10).

[0176] FIG. 9 is a drawing for explaining an operation of displaying a pointer UI (10) after a preset event is identified according to one or more embodiments.

[0177] Steps S905, S910, S915, S920, S921, and S922 of FIG. 9 may correspond to steps S705, S710, S715, S720, S721, and S722 of FIG. 7. Duplicate explanations are omitted.

[0178] After displaying the screen provided by the application, the terminal device (200) can determine whether a preset event related to the transmission of motion data has been identified (S916).

[0179] The preset events related to the transmission of motion data may include at least one of an event of touching a preset area, an event of touching a floating UI, an event of touching an arbitrary area (or any area) for a threshold period of time or longer, an event of touching (pressing) a physical button, and an event of touching a preset UI. Descriptions thereof are provided in FIGS. 17 to 27.

[0180] When a preset event is identified (S916-Y), the terminal device (200) can transmit a notification indicating that the preset event has been identified to the electronic device (100) (S917).

[0181] The electronic device (100) can receive a notification from the terminal device (200) indicating that a preset event has been identified. When a notification indicating that a preset event has been identified is received, the electronic device (100) can display a pointer UI (10) (S918).

[0182] FIG. 10 is a drawing for explaining an operation of connecting an electronic device (100) and a terminal device (200) according to one or more embodiments.

[0183] Referring to the embodiment (1000) of FIG. 10, a guide UI indicating that an electronic device (100) can be controlled through a terminal device (200) may be provided to the user.

[0184] The electronic device (100) can identify whether it is in a state where it can be connected to the terminal device (200). If it is in a state where it can be connected to the terminal device (200), the electronic device (100) can display a guide UI (1010). The guide UI (1010) can include at least one of information indicating that the electronic device (100) can be controlled through the terminal device (200) (text information and / or image information) and information guiding the user to check the terminal device (200).

[0185] The terminal device (200) can identify whether it is in a state where it can be connected to the electronic device (100). If it is in a state where it can be connected to the electronic device (100), the terminal device (200) can display a guide UI (1020). The guide UI (1020) can include at least one of information (text information and / or image information) indicating that the electronic device (100) can be directly controlled through the terminal device (200) and information querying whether to control the electronic device (100). The guide UI (1020) can be displayed in full screen or as a pop-up screen.

[0186] FIG. 11 is a drawing for explaining an operation of connecting a pre-registered electronic device (100) and a terminal device (200) according to one or more embodiments.

[0187] Steps S1110, S1115, S1120, S1121, and S1122 of FIG. 11 may correspond to steps S710, S715, S720, S721, and S722 of FIG. 7. Duplicate explanations are omitted.

[0188] It is assumed that the electronic device (100) and the terminal device (200) are mutually registered. It is assumed that the electronic device (100) and the terminal device (200) have a history of being interconnected and store information about the other party.

[0189] The electronic device (100) can determine whether connection with the terminal device (200) is possible. The terminal device (200) can determine whether connection with the electronic device (100) is possible.

[0190] If it is identified that connection with the terminal device (200) is possible, the electronic device (100) can provide a first guide UI for connection with the terminal device (200) (S1101). The first guide UI may be the guide UI (1010) of FIG. 10.

[0191] If it is identified that connection with the electronic device (100) is possible, the terminal device (200) can provide a second guide UI for connection with the electronic device (100) (S1102). The second guide UI may be the guide UI (1020) of FIG. 10.

[0192] The terminal device (200) can identify whether a first user input for controlling the electronic device (100) is received through the second guide UI (S1103). If the first user input is received (S1103-Y), the terminal device (200) can transmit the first user input to the electronic device (100) (S1104).

[0193] The electronic device (100) can receive a first user input for controlling the electronic device (100) from the terminal device (200). The electronic device (100) can approve a connection to the terminal device (200) in response to the first user input (S1105). If the electronic device (100) approves a connection to the terminal device (200), the electronic device (100) can transmit a notification indicating the approval of the connection to the terminal device (200).

[0194] The terminal device (200) may receive a connection approval notification. Upon receiving the connection approval notification, the terminal device (200) may execute an application for controlling the electronic device (100) (S1110). After the application is executed, steps S1115, S1120, S1121, and S1122 may be performed.

[0195] FIG. 12 is a drawing for explaining a touch input table according to one or more embodiments.

[0196] Referring to FIG. 12, the touch input table (1200) may include functions corresponding to user input. The touch input table may include functions corresponding to each touch input (or touch type) representing the user input.

[0197] As one or more examples, when a user input (drag) is received, the electronic device (100) can perform a function of moving the pointer UI (10) based on the touch data.

[0198] As one or more examples, when a user input (tap) is received, the electronic device (100) may perform a function of selecting an object (or item) corresponding to the current pointer location.

[0199] As one or more examples, when a user input (long tap) is received, the electronic device (100) may perform a function of displaying a home screen (or a preset screen).

[0200] As one or more examples, when user input (double tap) is received, the electronic device (100) may perform a function of displaying the previous screen.

[0201] As one or more examples, when user input (edge ​​scroll) is received, the electronic device (100) can perform a scroll function for the currently displayed screen.

[0202] For example, when a user input related to a preset event is received, the electronic device (100) may perform a function of moving the pointer UI (10) based on gyro data. The preset event may include at least one of an event of touching a preset area, an event of touching a floating UI, an event of touching an arbitrary area for a threshold time or longer, an event of touching (pressing) a physical button, and an event of touching a preset UI. Descriptions related thereto are described in FIGS. 17 to 27.

[0203] An event that touches an arbitrary area for a threshold period of time may be different from a long tap. A long tap may indicate that a tap input is received for a first threshold period of time. A preset event may indicate that a tap input is received for a second threshold period of time. For example, the second threshold period may be longer than the first threshold period.

[0204] The function corresponding to touch input may vary depending on the mode of the terminal device (200). The first mode may be a mode for transmitting touch data. The second mode may be a mode for transmitting motion data.

[0205] When the terminal device (200) is in the first mode, a specific function can be performed based on some information (#01, #02, #03, #04, #05) in the touch input table.

[0206] When the terminal device (200) is in the second mode, a specific function can be performed based on some information (#06) in the touch input table.

[0207] The touch input table (1200) can be stored in the electronic device (100) and / or the terminal device (200). The operation of the electronic device (100) using the touch input table (1200) is described in FIG. 13. The operation of the terminal device (200) using the touch input table (1200) is described in FIG. 14.

[0208] FIG. 13 is a drawing for explaining an operation in which an electronic device (100) performs a function corresponding to touch data according to one or more embodiments.

[0209] Referring to FIG. 13, the terminal device (200) may operate in a first mode. The first mode may be a mode for transmitting touch data to the electronic device (100). The first mode may be described as a touch mode. The first mode may be a default mode. The terminal device (200) may operate in the first mode, which is the default mode, when a preset event is not identified.

[0210] The terminal device (200) can acquire touch data (S1332). The touch data may include various inputs related to the user's touch on the terminal device (200). The touch data may include a touch location indicating whether a location was touched by the user and a touch time indicating how long the user touched the location. The terminal device (200) can transmit the touch data to the electronic device (100) (S1333).

[0211] The electronic device (100) can receive touch data from the terminal device (200). The electronic device (100) can store a touch input table. The electronic device (100) can identify a function corresponding to the touch data based on the touch input table (S1334). The electronic device (100) can perform the identified function (S1335). Additional descriptions related to this are described in FIGS. 15 and 16.

[0212] FIG. 14 is a diagram for explaining an operation of a terminal device (200) identifying a function corresponding to touch data according to one or more embodiments.

[0213] Steps S1431, S1432, and S1435 of Fig. 14 may correspond to steps S1331, S1332, and S1335 of Fig. 13. Duplicate explanations are omitted.

[0214] The terminal device (200) can store a touch input table. The terminal device (200) can identify a function corresponding to touch data based on the touch input table (S1433). The terminal device (200) can transmit information indicating the identified function to the electronic device (100) (S1434). The information indicating the identified function can be described as function information, function data, control commands, etc.

[0215] The electronic device (100) can receive information about the identified function from the terminal device (200). The electronic device (100) can perform the identified function (S1435).

[0216] FIG. 15 is a drawing for explaining an operation of moving a pointer UI (10) based on touch data according to one or more embodiments.

[0217] Referring to the embodiment (1500) of FIG. 15, the terminal device (200) can receive a user input (drag). The terminal device (200) can transmit a signal indicating the user input (drag) to the electronic device (100).

[0218] The electronic device (100) can receive a signal indicating a user input (drag) from the terminal device (200). The electronic device (100) can change the display position of the pointer UI (10) based on the user input (drag). The electronic device (100) can identify a new position based on the current position of the pointer UI (10) and a correction value corresponding to the user input (drag). The new position can be described as a correction position or a change position. The electronic device (100) can display a screen for moving the pointer UI (10) from the current position to the new position.

[0219] FIG. 16 is a drawing for explaining an operation of moving a pointer UI (10) based on touch data according to one or more embodiments.

[0220] Steps S1631, S1632, and S1633 of FIG. 16 may correspond to steps S1331, S1332, and S1333 of FIG. 13. Duplicate explanations are omitted.

[0221] The electronic device (100) can receive touch data from the terminal device (200). The electronic device (100) can obtain location information of the pointer UI (10) based on the touch data (S1634). The location information can include the current location of the pointer UI (10) and the new location (or changed location or corrected location) of the pointer UI (10). The current location can be described as the first location. The new location can be described as the second location.

[0222] The electronic device (100) can obtain a drag signal from a user input included in touch data. The electronic device (100) can obtain a correction value (or movement value) of the pointer UI (10) based on the drag signal. The electronic device (100) can identify a new position of the pointer UI (10) that reflects the correction value based on the current position of the pointer UI (10).

[0223] The electronic device (100) can obtain location information of the pointer UI (10), including the current location of the pointer UI (10) and the new location of the pointer UI (10). The electronic device (100) can display the pointer UI (10) based on the location information. The electronic device (100) can change the display location of the pointer UI (10) to move from the current location to the new location.

[0224] According to one or more embodiments, the electronic device (100) can move the pointer UI (10) without (the concept of) location information. The electronic device (100) can move the pointer UI (10) using the current location of the pointer UI (10) and the new location of the pointer UI (10).

[0225] FIG. 17 is a drawing for explaining an operation of moving a pointer UI (10) by touching a preset area according to one or more embodiments.

[0226] Referring to FIG. 17, the terminal device (200) may operate in gyro mode. The gyro mode may be a mode for transmitting motion data from the terminal device (200) to the electronic device (100). The entire screen of the display of the terminal device (200) may include a preset first area (1710) and a preset second area (1720). The first area (1710) and the second area (1720) may be changed according to the user's settings.

[0227] The first region (1710) may be an region that receives user input for transmitting touch data to the electronic device (100). When a user input is received in the first region (1710), the terminal device (200) may transmit touch data representing the user input to the electronic device (100).

[0228] The second area (1720) may be an area that receives a user input for transmitting motion data to the electronic device (100). When a user input is received in the second area (1720), the terminal device (200) may obtain gyro data using a gyro sensor included in the terminal device (200). The terminal device (200) may transmit motion data including gyro data to the electronic device (100).

[0229] The electronic device (100) can receive motion data from the terminal device (200). The electronic device (100) can move the pointer UI (10) based on the motion data. The electronic device (100) can obtain a correction value of the pointer UI based on the motion data. The electronic device (100) can identify a new location of the pointer UI (10) by applying the correction value to the current location of the pointer UI (10). The electronic device (100) can display the pointer UI (10) moving from the current location to the new location.

[0230] According to one or more embodiments, the first region (1710) and the second region (1720) may be displayed in the same color.

[0231] According to one or more embodiments, the first region (1710) and the second region (1720) may be displayed in different colors. The first region (1710) may be displayed in a first color, and the second region (1720) may be displayed in a second color (different from the first color). The colors may be described as background colors.

[0232] FIG. 18 is a drawing for explaining an operation of moving a pointer UI (10) by touching a preset area according to one or more embodiments.

[0233] Referring to FIG. 18, the terminal device (200) can obtain a user input for touching a preset area (S1840). The preset area may be an area that receives a user input for transmitting motion data obtained by the terminal device (200) to the electronic device (100). The preset area may be the second area (1720) of FIG. 17.

[0234] When a user input is received through the second area (1720), the terminal device (200) can operate in the second mode (or gyro mode) (S1841).

[0235] When a user input is received through the second area (1720), the terminal device (200) can obtain gyro data through the gyro sensor of the terminal device (200). The terminal device (200) can obtain motion data including the gyro data (S1842). The terminal device (200) can transmit the motion data to the electronic device (100) (S1843).

[0236] The electronic device (100) can receive motion data from the terminal device (200). The electronic device (100) can obtain location information of the pointer UI (10) based on the motion data (S1844). The location information of the pointer UI (10) can include the current location of the pointer UI (10) and the new location of the pointer UI (10). The electronic device (100) can obtain a correction value related to the movement of the pointer UI (10) based on the motion data. The electronic device (100) can calculate the new location of the pointer UI (10) by applying the correction value to the current location of the pointer UI (10). The electronic device (100) can obtain location information of the pointer UI (10) including the current location of the pointer UI (10) and the new location of the pointer UI (10).

[0237] The electronic device (100) can display a pointer UI (10) based on location information (S1845). The electronic device (100) can display a pointer UI (10) that moves from a current location to a new location based on location information.

[0238] According to one or more embodiments, the terminal device (200) may receive a user input touching a preset area for a threshold time or longer. When a user input touching the preset area for a threshold time or longer is received, the terminal device (200) may operate in a second mode.

[0239] FIG. 19 is a diagram for explaining an operation of processing touch data and an operation of processing motion data according to one or more embodiments.

[0240] Steps S1931, S1932, S1933, S1934, and S1935 of FIG. 19 may correspond to steps S1631, S1632, S1633, S1634, and S1635 of FIG. 16. Steps S1940, S1941, S1942, S1943, S1944, and S1945 of FIG. 19 may correspond to steps S1840, S1841, S1842, S1843, S1844, and S1845. Duplicate explanations are omitted.

[0241] The terminal device (200) can obtain user input in a preset first area (S1930). The preset first area may be the first area (1710) of FIG. 17.

[0242] When a user input is received in a preset first area, the terminal device (200) can operate in a first mode (or touch mode) (S1931). The terminal device (200) can receive a user input through the preset first area. The terminal device (200) can obtain touch data representing the user input received in the first area (S1932). When the touch data is obtained, steps S1933, S1934, and S1935 can be performed.

[0243] The terminal device (200) can obtain user input in a preset second area (S1940). The preset second area may be the second area (1720) of FIG. 17.

[0244] When a user input is received in a preset second area, the terminal device (200) can operate in a second mode (or gyro mode) (S1941). The terminal device (200) can receive a user input through the preset second area. The terminal device (200) can acquire motion data (or gyro data) while the user input is received in the second area (S1942). When the motion data is acquired, steps S1943, S1944, and S1945 can be performed.

[0245] FIG. 20 is a drawing for explaining a plotting UI (20) according to one or more embodiments.

[0246] Referring to the embodiment (2000) of FIG. 20, the terminal device (200) can display a floating UI (20). The floating UI (20) may be a UI for operating the terminal device (200) in a second mode (or gyro mode). The floating UI (20) may be displayed in a pop-up form on a screen displayed on the terminal device (200).

[0247] The terminal device (200) can determine whether a user input is received at a location where the floating UI (20) is displayed. The terminal device (200) can receive a user input through the floating UI (20). When a user input touching the floating UI (20) is received, the terminal device (200) can transmit motion data (gyro data) to the electronic device (100).

[0248] The electronic device (100) can receive motion data from the terminal device (200). The electronic device (100) can move the pointer UI (10) based on the motion data.

[0249] According to one or more embodiments, the terminal device (200) may receive a user input that continuously touches the floating UI (20). At a point when the floating UI (20) is not touched, the terminal device (200) may stop the second mode and operate in the first mode.

[0250] According to one or more embodiments, the terminal device (200) may identify an area corresponding to a location where the floating UI (20) is displayed as a preset area of ​​FIG. 18.

[0251] FIG. 21 is a drawing for explaining an operation of controlling a pointer UI (10) displayed on an electronic device (100) through a floating UI (20) according to one or more embodiments.

[0252] Steps S2141, S2143, S2144, and S2145 of FIG. 21 can perform steps S1841, S1843, S1844, and S1845 of FIG. 18. Duplicate descriptions are omitted.

[0253] The terminal device (200) can display a floating UI (20) for executing the second mode (S2140-1). The terminal device (200) can obtain a user input touching the floating UI (20) (S2140-2).

[0254] When a user input touching the floating UI (20) is received, the terminal device (200) can operate in the second mode (or gyro mode) (S2141). The terminal device (200) can obtain motion data (or gyro data) (S2142). While the user touches the floating UI (20), the terminal device (200) can obtain motion data representing the movement of the terminal device (200). When the motion data is obtained, the terminal device (200) can perform steps S2143, S2144, and S2145.

[0255] According to one or more embodiments, the terminal device (200) may transmit only motion data acquired while the user touches the floating UI (20) to the electronic device (100). In order to move the pointer UI (10) using the motion data, the user must be touching the floating UI (20).

[0256] According to one or more embodiments, after performing an operation in the second mode through the floating UI (20), user input to the floating UI (20) may not continue to be received. The terminal device (200) may transmit motion data to the electronic device (100) even if user input is not continuously received. To change from the second mode to the first mode, the user may touch the floating UI (20) once again. An operation of transmitting motion data to the electronic device (100) even without touching a specific area of ​​the terminal device (200) is described in FIG. 26.

[0257] FIG. 22 is a drawing for explaining displaying a pointer UI (10) based on a preset time-recognized touch according to one or more embodiments.

[0258] Referring to the embodiment (2200) of FIG. 22, the terminal device (200) can receive user input for a threshold period of time for any area. If a user input that continuously touches a location for a threshold period of time is received, the terminal device (200) can operate in a second mode. While operating in the second mode, the terminal device (200) can transmit motion data to the electronic device (100).

[0259] The electronic device (100) can receive motion data from the terminal device (200). The electronic device (100) can move the pointer UI (10) based on the motion data.

[0260] The user input received during the threshold time may differ from a typical long-tap input. If a user input is received that touches a specific location for a longer time than the time for a long-tap, the electronic device (100) may operate in a second mode.

[0261] FIG. 23 is a drawing for explaining displaying a pointer UI (10) based on a preset time-recognized touch according to one or more embodiments.

[0262] Steps S2341, S2343, S2344, and S2345 of FIG. 23 can perform steps S1841, S1843, S1844, and S1845 of FIG. 18. Duplicate descriptions are omitted.

[0263] The terminal device (200) can obtain a user input touching an arbitrary area (or an arbitrary location) for a threshold time or longer (S2340). The threshold time can be changed according to the user's settings. When a user input touching an arbitrary area for a threshold time or longer is received, the terminal device (200) can operate in a second mode (S2341).

[0264] The terminal device (200) can acquire motion data (or gyro data) by touching an arbitrary area (S2342). While the user touches an arbitrary area of ​​the terminal device (200), the terminal device (200) can acquire motion data representing the movement of the terminal device (200). Once the motion data is acquired, steps S2343, S2344, and S2345 can be performed.

[0265] FIG. 24 is a drawing for explaining an operation of moving a pointer UI (10) using a physical button of a terminal device (200) according to one or more embodiments.

[0266] Referring to the embodiment (2400) of FIG. 24, the terminal device (200) can receive a user input of touching (or pressing) a preset physical button (2410). When a user input of pressing the preset physical button (2410) is received, the terminal device (200) can operate in a second mode. When operating in the second mode, the terminal device (200) can transmit motion data to the electronic device (100). The electronic device (100) can move the pointer UI (10) based on the motion data transmitted from the terminal device (200).

[0267] FIG. 25 is a drawing for explaining an operation of moving a pointer UI (10) using a physical button of a terminal device (200) according to one or more embodiments.

[0268] Steps S2541, S2543, S2544, and S2545 of FIG. 25 can perform steps S1841, S1843, S1844, and S1845 of FIG. 18. Duplicate explanations are omitted.

[0269] The terminal device (200) can obtain a user input touching a preset physical button (S2540). When a user input touching a preset physical button is received, the terminal device (200) can operate in a second mode (S2541).

[0270] The terminal device (200) can acquire motion data (or gyro data) while a preset physical button is touched (S2542). While the user touches the preset physical button of the terminal device (200), the terminal device (200) can acquire motion data representing the movement of the terminal device (200). Once the motion data is acquired, steps S2543, S2544, and S2545 can be performed.

[0271] According to one or more embodiments, the terminal device (200) may transmit to the electronic device (100) only motion data acquired while the user touches a preset physical button. In order to move the pointer UI (10) using the motion data, the user must touch the preset physical button.

[0272] According to one or more embodiments, after performing an operation in the second mode through a preset physical button, a user input for the preset physical button may not continue to be received. The terminal device (200) may transmit motion data to the electronic device (100) even if the user input is not continuously received. To change from the second mode to the first mode, the user may touch the preset physical button once again. An operation of transmitting motion data to the electronic device (100) even without touching a specific area of ​​the terminal device (200) is described in FIG. 26.

[0273] FIG. 26 is a drawing for explaining an operation of moving a pointer UI (10) by identifying a preset event according to one or more embodiments.

[0274] Referring to the embodiment (2600) of FIG. 26, the terminal device (200) can transmit motion data to the electronic device (100) without touching the touch area of ​​the terminal device (200). The motion data can be transmitted to the electronic device (100) while the terminal device (200) is operating in the second mode. To maintain the second mode, the user may not need to touch a specific area (or a specific location or a specific physical button). To exit the second mode, the terminal device (200) must receive a new user input.

[0275] FIG. 27 is a drawing for explaining an operation of moving a pointer UI (10) by identifying a preset event according to one or more embodiments.

[0276] Steps S2741, S2743, S2744, and S2745 of FIG. 27 can perform steps S1841, S1843, S1844, and S1845 of FIG. 18. Duplicate descriptions are omitted.

[0277] Referring to FIG. 27, the terminal device (200) can identify a preset event (S2740). The preset event may represent an event for executing the second mode. The preset event may include at least one of an event of touching a preset area, an event of touching a floating UI, an event of touching an arbitrary area for a threshold time or longer, an event of touching (pressing) a physical button, and an event of touching a preset UI.

[0278] When a preset event is identified, the terminal device (200) can operate in a second mode. While operating in the second mode, the terminal device (200) can acquire motion data. The terminal device (200) can maintain the second mode even when the user's touch input is not continuously received. The terminal device (200) can acquire motion data in the absence of the user's touch input (S2742). When the motion data is acquired, steps S2743, S2744, and S2745 can be performed.

[0279] For example (see FIGS. 17, 18, and 19), when a user input touching a preset area is received, the terminal device (200) can change from the first mode to the second mode. While no user input is received, the terminal device (200) can operate in the second mode. When a user input touching the preset area again is received, the terminal device (200) can change from the second mode to the first mode.

[0280] For example (see FIGS. 20 and 21), when a user input touching the floating UI (20) is received, the terminal device (200) can change from the first mode to the second mode. While no user input is received, the terminal device (200) can operate in the second mode. When a user input touching the floating UI (20) again is received, the terminal device (200) can change from the second mode to the first mode.

[0281] For example (see FIGS. 22 and 23), when a user input touching an arbitrary area for a threshold time or longer is received, the terminal device (200) can change from the first mode to the second mode. While no user input is received, the terminal device (200) can operate in the second mode. When a user input touching an arbitrary area for a threshold time or longer is received, the terminal device (200) can change from the second mode to the first mode.

[0282] For example (see FIGS. 24 and 25), when a user input touching a preset physical button is received, the terminal device (200) can change from the first mode to the second mode. While no user input is received, the terminal device (200) can operate in the second mode. When a user input touching the preset physical button again is received, the terminal device (200) can change from the second mode to the first mode.

[0283] FIG. 28 is a drawing for explaining a mode of a terminal device (200) according to one or more embodiments.

[0284] Referring to the embodiment (2810) of FIG. 28, the terminal device (200) can operate in a first mode (or touch mode). While operating in the first mode, the terminal device (200) can display a screen in a first color (or a first background color).

[0285] Referring to the embodiment (2820) of FIG. 28, the terminal device (200) can operate in a second mode (or gyro mode). While operating in the second mode, the terminal device (200) can change the color of the touch area. While operating in the second mode, the terminal device (200) can display the screen in a second color (or a second background color).

[0286] The first and second colors may be different. The second color may be more prominent than the first color. The user may not focus their gaze directly on the terminal device (200). By displaying the screen in the second color, the user can easily recognize that the terminal device (200) is operating in the second mode.

[0287] FIG. 29 is a drawing for explaining a movable state of a plotting UI (20) according to one or more embodiments.

[0288] The terminal device (200) of Fig. 29 may correspond to the terminal device (200) described in Fig. 5. Duplicate description is omitted.

[0289] Referring to the embodiment (2910) of FIG. 29, the terminal device (200) can display a floating UI (20). While the floating UI (20) is displayed, the terminal device (200) can receive user input.

[0290] The terminal device (200) can move the display position of the floating UI (20). When a preset event is identified, the terminal device (200) can provide the user with a state for moving the floating UI (20). The state for moving the floating UI (20) can be described as a mode for moving the floating UI (20). The state for moving the floating UI (20) can be described as a floating UI (20) movement mode.

[0291] A preset event may represent an event in which a user input is received to move the floating UI (20).

[0292] According to one or more embodiments, the preset event may mean an event in which a user input is received for a separate setting item for moving the floating UI (20).

[0293] According to one or more embodiments, the preset event may be an event of pressing the floating UI (20) for a threshold time or longer.

[0294] In some of the embodiments described above, it is described that motion data is transmitted to the electronic device (100) only while touching the floating UI (20). The situation where the floating UI (20) is continuously touched and the situation where an event for moving the position of the floating UI (20) is identified may overlap. To solve this problem, the terminal device (200) can simultaneously perform an operation of checking whether the floating UI (20) is touched for a threshold time or longer and an operation of checking the change value of gyro data.

[0295] As one or more examples, when a user input is received that touches the floating UI (20) for a first threshold time, the terminal device (200) may operate in a second mode.

[0296] As one or more examples, if a change value of gyro data is within a threshold range (or below a threshold value) while a user input touching the floating UI (20) for a second threshold time is received, the terminal device (200) can identify that a preset event for moving the floating UI (20) has occurred. The terminal device (200) can perform (or operate) a mode for moving the floating UI (20). The second threshold time can be equal to or greater than the first threshold time.

[0297] When a user has the intention to move the pointer UI (10) by touching the floating UI (20), the change value of the motion data (or gyro data) sensed by the terminal device (200) may be large.

[0298] When a user has the intention to move the floating UI (20) by touching the floating UI (20), the change value of the motion data (or gyro data) sensed by the floating UI (20) may be small.

[0299] Referring to Example (2920), when a preset event for moving the floating UI (20) is identified, the terminal device (200) can perform a mode for moving the floating UI (20). The terminal device (200) can provide a UI to indicate a state in which the floating UI (20) can be moved. As one or more examples, the terminal device (200) can change the color of the touch input UI (506).

[0300] In the embodiment (2910), the terminal device (200) can display the touch input UI (506) in a first color (or a first background color).

[0301] In embodiment (2920), the terminal device (200) may display the touch input UI (506) in a second color (or a second background color). The second color may be different from the first color.

[0302] When the second color and the first color are different, the user can easily recognize the movable state of the floating UI (20). In the movable state of the floating UI (20), touch areas other than the touch input UI (506) can be set as non-responsive areas.

[0303] The movable area of ​​the floating UI (20) may be limited to a preset area. In the embodiment (2920), the floating UI (20) may be moved only within an area corresponding to the touch input UI (506). This is to clearly determine which UI the user is selecting.

[0304] FIG. 30 is a drawing for explaining an operation of displaying a pointer UI (10) corresponding to an area of ​​a terminal device (200) touched by a user according to one or more embodiments.

[0305] The terminal device (200) of FIG. 30 may correspond to the terminal device (200) of FIG. 17. Duplicate descriptions will be omitted. The entire screen of the display of the terminal device (200) may include a preset first area (1710) and a preset second area (1720). The first area (1710) may be an area that receives a user input for transmitting touch data to the electronic device (100). The second area (1720) may be an area that receives a user input for transmitting motion data to the electronic device (100).

[0306] Referring to the embodiment (3010) of FIG. 30, the terminal device (200) can receive a user input touching a first area (1710). The terminal device (200) can perform a first mode. The electronic device (100) can display the pointer UI (10) in a first color (or a first background color) while the terminal device (200) is performing in the first mode. When touch data is received, the electronic device (100) can display the pointer UI (10-1) in the first color.

[0307] Referring to the embodiment (3020) of FIG. 30, the terminal device (200) can receive a user input touching a second area (1720). The terminal device (200) can perform a second mode. The electronic device (100) can display the pointer UI (10-2) in a second color (or a second background color) while the terminal device (200) is performing in the second mode. When motion data is received, the electronic device (100) can display the pointer UI (10) in the second color.

[0308] FIG. 31 is a drawing for explaining a user pattern for moving a pointer UI (10) according to one or more embodiments.

[0309] Referring to the embodiment (3100) of FIG. 31, the electronic device (100) can identify the user's touch pattern. Assume a situation where the pointer UI (10) moves from object A to object B. To move from object A to object B, a drag signal in the right direction must be input. Depending on the user's touch habits, the drag input may be input for an excessively long time. If the drag input is long, the user must input the drag in the opposite direction.

[0310] The terminal device (200) can receive a drag signal (3101) in a first direction and a drag signal (3102) in a second direction. The second direction may be the opposite direction to the first direction.

[0311] The terminal device (200) can obtain touch data including a first drag signal (3101) and a second drag signal (3102). The terminal device (200) can transmit the touch data to the electronic device (100).

[0312] The electronic device (100) can obtain a first drag signal (3101) and a second drag signal (3102) based on touch data. The electronic device (100) can move the pointer UI (10) in a first direction (3111) based on the first drag signal (3101). The electronic device (100) can move the pointer UI (10) in a second direction (3112) based on the second drag signal (3102).

[0313] In a situation where object B is selected from object A, the electronic device (100) can identify a first pattern in which the user of the terminal device (200) continuously inputs drag signals in opposite directions. The first pattern may not be limited to a situation in which object B is selected. The first pattern may be identified in a situation in which any object is selected. The electronic device (100) can identify the number of first patterns. If the number of first patterns is a threshold number, the electronic device (100) can store the identified first patterns as movement pattern information of the pointer UI (10).

[0314] The electronic device (100) may provide a user with a first pattern with a function of controlling the movement speed of the pointer UI (10). The electronic device (100) may control the movement speed of the pointer UI (10) to prevent unnecessary drag signals (3102) from being input. The electronic device (100) may reduce the movement speed of the pointer UI (10) at a specific location.

[0315] Assume a situation where a pointer UI (10) moves from object A to object B. The electronic device (100) can store (or obtain) the location (3121) of object A and the location (3122) of object B in advance. Object A can be described as a first object. Object B can be described as a second object.

[0316] The electronic device (100) can move the pointer UI (10) in the right direction (in the direction corresponding to the drag signal) at a first speed from the current position (3121) of the pointer UI (10).

[0317] When the pointer UI (10) is located within a critical range from the position (3122) of the B object, the electronic device (100) can move the pointer UI (10) in the right direction (in the direction corresponding to the drag signal) at a second speed from the initial position (3123) where the pointer UI (10) is within the critical range. The second speed may be smaller than the first speed.

[0318] When the movement speed of the pointer UI (10) is slowed down, the number of situations in which the user must input a drag signal (3102) in the opposite direction may be reduced.

[0319] FIG. 32 is a drawing for explaining a user pattern for moving a pointer UI (10) according to one or more embodiments.

[0320] Referring to the embodiment (3200) of FIG. 32, the electronic device (100) can identify the user's touch pattern. Assume a situation where the pointer UI (10) moves from object A to object B. To move from object A to object B, a drag signal in the right direction must be input. Depending on the user's touch habits, the drag input may be input too briefly. If the drag input is too short, the user must input an additional drag in the same direction.

[0321] The terminal device (200) can receive a drag signal (3201) in a first direction and a drag signal (3202) in a second direction. The second direction may be the same direction as the first direction. Therefore, the second direction may be described as the first direction.

[0322] The terminal device (200) can obtain touch data including a first drag signal (3201) and a second drag signal (3202). The terminal device (200) can transmit the touch data to the electronic device (100).

[0323] The electronic device (100) can obtain a first drag signal (3201) and a second drag signal (3202) based on touch data. The electronic device (100) can move the pointer UI (10) in a first direction (3211) based on the first drag signal (3201). The electronic device (100) can move the pointer UI (10) in a second direction (3212) based on the second drag signal (3202).

[0324] In a situation where object B is selected from object A, the electronic device (100) can identify a second pattern in which the user of the terminal device (200) continuously inputs a drag signal in the same direction. The second pattern may not be limited to a situation in which object B is selected. The second pattern can be identified in a situation in which any object is selected. The electronic device (100) can identify the number of second patterns. If the number of second patterns is a threshold number, the electronic device (100) can store the identified second patterns as movement pattern information of the pointer UI (10).

[0325] The electronic device (100) may provide a user with a second pattern with a function of controlling the movement speed of the pointer UI (10). The electronic device (100) may control the movement speed of the pointer UI (10) to prevent unnecessary drag signals (3202) from being input. The electronic device (100) may increase the movement speed of the pointer UI (10) at a specific location.

[0326] Assume a situation where a pointer UI (10) moves from object A to object B. The electronic device (100) can store (or obtain) the location (3221) of object A and the location (3222) of object B in advance. Object A can be described as a first object. Object B can be described as a second object.

[0327] The electronic device (100) can move the pointer UI (10) in the right direction (in the direction corresponding to the drag signal) at a first speed from the current position (3221) of the pointer UI (10).

[0328] When the pointer UI (10) is located within a critical range from the position (3222) of the B object, the electronic device (100) can move the pointer UI (10) in the right direction (in the direction corresponding to the drag signal) at a second speed from the initial position (3223) where the pointer UI (10) is within the critical range. The second speed may be greater than the first speed.

[0329] When the movement speed of the pointer UI (10) is slowed down, the number of situations in which the user must input a drag signal (3202) in the same direction may be reduced.

[0330] FIG. 33 is a diagram for explaining an operation of changing the movement speed of a pointer UI based on the number of objects, according to one or more embodiments.

[0331] Referring to FIG. 33, the electronic device (100) can store movement pattern information of the pointer UI (S3301). The electronic device (100) can display a screen including multiple objects (S3302).

[0332] The terminal device (200) can operate in the first mode (S3331). The terminal device (200) can acquire touch data (S3332). The terminal device (200) can transmit touch data to the electronic device (100) (S3333).

[0333] The electronic device (100) can receive touch data from the terminal device (200). The electronic device (100) can obtain location information of the pointer UI (10) based on the touch data (S3334). Step S3334 may correspond to step S1634 of FIG. 16. Duplicate description is omitted.

[0334] The electronic device (100) can identify the number of objects included in the screen (S3335). The electronic device (100) can identify whether the number of objects is greater than or equal to a threshold number (S3336).

[0335] If the number of objects is greater than or equal to a threshold number (S3336-Y), the electronic device (100) may indicate that the pointer UI (10) should be moved based on a first speed (S3337). The electronic device (100) may fix the movement speed of the pointer UI (10) to the first speed. The first speed may be a basic speed.

[0336] The electronic device (100) can change the movement speed of the pointer UI (10) based on the number of objects. If the number of objects is greater than a threshold number, the movement speed of the pointer UI (10) may not be changed. This is because if the movement speed of the pointer UI (10) changes frequently, the user may feel uncomfortable.

[0337] If the number of objects is less than a threshold number (S3336-N), the electronic device (100) can change the movement speed of the pointer UI (10) based on the movement pattern information of the pointer UI (10) and the position information of the pointer UI (10) (S3338). The electronic device (100) can display the pointer UI (10) to be moved based on the changed movement speed.

[0338] In one or more examples, the movement speed may be changed to a second speed that is lower than the first speed. A description of this is provided in FIG. 31.

[0339] In one or more examples, the movement speed may be changed to a third speed greater than the first speed. A description thereof is provided in FIG. 32. The third speed may correspond to the second speed of FIG. 32.

[0340] According to one or more embodiments, the change in movement speed may be fixed. The electronic device (100) may identify the user's pattern based on movement pattern information of the pointer UI (10). The electronic device (100) may change the average movement speed of the pointer UI (10) based on the user's pattern. The electronic device (100) may move the pointer UI (10) based on the changed movement speed.

[0341] According to one or more embodiments, the change in movement speed may be variable based on the position of the object. A description thereof is provided in FIGS. 31 and 32.

[0342] According to one or more embodiments, the first mode described in FIG. 33 may be replaced with a second mode. Touch data may be replaced with motion data. This is because the pointer UI may be moved by motion data as well as touch data.

[0343] FIG. 34 is a drawing for explaining an operation of controlling the movement speed of a pointer UI based on the distance between an electronic device (100) and a terminal device (200), according to one or more embodiments.

[0344] Steps S3401, S3402, S3431, S3432, S3433, S3434, S3435, S3436, S3437, and S3439 of FIG. 34 may correspond to steps S3301, S3302, S3331, S3332, S3333, S3333, S3335, S3336, S3337, and S3339 of FIG. 33. Duplicate explanations are omitted.

[0345] When touch data is received from the terminal device (200), the electronic device (100) can identify the strength of the signal including the touch data. The electronic device (100) can obtain (or identify or calculate) distance information between the electronic device (100) and the terminal device (200) based on the signal strength of the touch data (S3433-2).

[0346] The electronic device (100) can perform steps S3434, S3435, S3436, and S3737.

[0347] If the number of objects is less than the threshold number (S3436-N), the electronic device (100) can change the movement speed of the pointer UI (10) based on the distance information acquired in step S3433-2, the movement pattern information of the pointer UI (10), and the position information of the pointer UI (10) (S3438). The electronic device (100) can display that the pointer UI (10) is to be moved based on the changed movement speed (S3439).

[0348] According to one or more embodiments, the first mode described in FIG. 34 may be replaced with a second mode. Touch data may be replaced with motion data, as the pointer UI may be moved by motion data as well as touch data.

[0349] FIG. 35 is a drawing for explaining a user's grab method according to one or more embodiments.

[0350] Referring to the embodiment (3500) of FIG. 35, the terminal device (200) can acquire the gripping posture in which the user is holding the terminal device (200). The terminal device (200) can store the preset gripping posture. The preset gripping posture can be described as the default gripping posture. The gripping posture can be described as gripping posture information.

[0351] According to one or more embodiments, the default grip posture may be a posture that takes into account the posture of average users. The default grip posture may be a preset posture.

[0352] According to one or more embodiments, the basic gripping posture may be an average posture of the terminal device (200). The terminal device (200) may analyze gyro data of the terminal device (200) to obtain a gripping posture indicating the posture in which the user is holding the terminal device (200). After obtaining the gripping posture for a preset period (or number of times), the terminal device (200) may store the user's average gripping posture as the basic gripping posture.

[0353] The terminal device (200) can compare the stored basic grip posture with the current grip posture. If the current grip posture and the basic grip posture are different, the terminal device (200) can display a guide screen (3501) related to the grip posture.

[0354] As one or more examples, the terminal device (200) can identify the gripping posture based on the average change value of the gyro data. The terminal device (200) can compare the gripping posture by comparing the basic average change value with the current average change value. The terminal device (200) can identify the difference between the basic average change value and the current average change value. If the difference is greater than or equal to a threshold value, the terminal device (200) can display a guide screen (3501).

[0355] The guide screen (3501) may include at least one of a UI (3510) explaining why a change in the grip posture is necessary, a UI (3520) guiding a change in the grip posture, and a UI (3530) indicating a default grip posture.

[0356] The UI (3530) indicating the basic grip posture may include text or an image. The UI (3530) indicating the basic grip posture allows the user to easily recognize the correct grip posture.

[0357] FIG. 36 is a drawing for explaining an operation of changing the position of a floating UI (20) according to one or more embodiments.

[0358] Embodiment (3600) of Fig. 36 may correspond to embodiment (3500) of Fig. 35 in relation to the grip posture. Duplicate explanation is omitted.

[0359] The terminal device (200) can compare the stored basic grip posture with the current grip posture. If the current grip posture and the basic grip posture are different, the terminal device (200) can display a guide screen (3601) related to the grip posture.

[0360] The guide screen (3601) may include at least one of a UI (3610) for guiding the position movement of the plotting UI (20) and a UI (3620) for guiding a recommended position.

[0361] The UI (3610) for guiding the position movement of the floating UI (20) may include at least one of information indicating that the user's current grip posture is different from the basic grip posture and information guiding the position movement of the floating UI (20).

[0362] The UI (3620) guiding the recommended location may include text or images. The UI (3620) guiding the recommended location allows the user to easily recognize the location where the floating UI (20) can be easily touched.

[0363] The terminal device (200) can store the user's touch history. Based on the user's touch history, the terminal device (200) can obtain dominant hand information indicating whether the user is right-handed or left-handed. The terminal device (200) can determine a recommended location based on the dominant hand information. The recommended locations for right-handed and left-handed users may differ.

[0364] FIG. 37 is a drawing for explaining an embodiment in which a terminal device (200) and a remote control device are simultaneously recognized by an electronic device (100) according to one or more embodiments.

[0365] Referring to the embodiment (3700) of FIG. 37, the electronic device (100) may be connected to a plurality of devices capable of controlling the electronic device (100). The plurality of devices may be a terminal device (200) and a remote control device (300). The remote control device (300) may be a dedicated control device corresponding to the electronic device (100).

[0366] The electronic device (100) can display a pointer UI (10, 11) for each of a plurality of connected devices. The electronic device (100) can display a pointer UI (10) corresponding to a terminal device (200). The electronic device (100) can display a pointer UI (11) corresponding to a remote control device (300). The pointer UI (10) can be described as a first pointer UI. The pointer UI (11) can be described as a second pointer UI.

[0367] The electronic device (100) can display the pointer UI (10) in a first form (or first shape).

[0368] The electronic device (100) can display the pointer UI (11) in a second form (or second shape). The second form may be different from the first form.

[0369] According to one or more embodiments, the electronic device (100) may include identification information (user1, remote) indicating the subject of the pointer UI. The pointer UI (10) may include identification information (user1) indicating the terminal device (200). The identification information may be displayed as an image or text. The pointer UI (11) may include identification information (remote) indicating the remote control device (300).

[0370] According to one or more embodiments, when multiple devices are identified as connected, the electronic device (100) may display only the pointer UI corresponding to one device according to a preset priority.

[0371] As one or more examples, the highest priority may be applied to a remote control device (300) corresponding to an electronic device (100).

[0372] As one or more examples, the highest priority may be applied to the terminal device (200) registered to the electronic device (100).

[0373] As one or more examples, the highest priority may be applied to the device that was most recently connected to the electronic device (100).

[0374] FIG. 38 is a drawing for explaining an embodiment in which multiple terminal devices (201, 202) are simultaneously recognized by an electronic device (100) according to one or more embodiments.

[0375] Referring to embodiment 3800 of FIG. 38, the electronic device (100) may be connected to a plurality of devices capable of controlling the electronic device (100). The plurality of devices may be a terminal device (201) and a terminal device (202). The terminal device (201) may be described as a first terminal device. The terminal device (202) may be described as a second terminal device.

[0376] When multiple terminal devices (201, 202) are identified as being connected, the electronic device (100) can display a pointer UI (10-1, 10-2) for each of the multiple connected terminal devices (201, 202). The electronic device (100) can display a pointer UI (10-1) corresponding to the terminal device (201). The electronic device (100) can display a pointer UI (10-2) corresponding to the terminal device (202). The pointer UI (10-1) can be described as a first pointer UI. The pointer UI (10-2) can be described as a second pointer UI.

[0377] The electronic device (100) can display the pointer UI (10-1) in a first form (or first shape).

[0378] The electronic device (100) can display the pointer UI (10-2) in a second form (or second shape). The second form may be different from the first form.

[0379] According to one or more embodiments, the electronic device (100) may include identification information (user1, user2) indicating the subject of the pointer UI. The pointer UI (10-1) may include identification information (user1) indicating the terminal device (201). The identification information may be displayed as an image or text. The pointer UI (10-2) may include identification information (user2) indicating the terminal device (202).

[0380] FIG. 39 is a drawing for explaining an embodiment in which multiple terminal devices (201, 202) are simultaneously recognized by an electronic device (100) according to one or more embodiments.

[0381] S3905-1, S3910-1, S3911-1, S3912-1, S3915-1, S3920-2, S3921-1, S3922-1 of FIG. 39 may correspond to S805, S810, S811, S812, S815, S820-2, S821, S822 of FIG. 8. S3905-2, S3910-2, S3911-2, S3912-2, S3915-2, S3920-2, S3921-2, S3922-2 of FIG. 39 may correspond to S805, S810, S811, S812, S815, S820-2, S821, S822 of FIG. 8. Duplicate explanations are omitted.

[0382] The first terminal device (201) can receive a first user input for executing an application related to the electronic device (100) (S3905-1). When the first user input is received, the first terminal device (201) can execute the application (S3910-1). The first terminal device (201) can transmit an execution notification of the application to the electronic device (100) (S3911-1).

[0383] The electronic device (100) can receive an application execution notification from the first terminal device (201). The electronic device (100) can display a first pointer UI (10-1) corresponding to the first terminal device (201).

[0384] The first terminal device (201) can display a screen provided by the application (S3915-1). The first terminal device (201) can receive a second user input for controlling the electronic device (100) through the screen (S3920-1). The first terminal device (201) can transmit the second user input to the electronic device (100) (S3921-1).

[0385] The electronic device (100) can receive a second user input from the first terminal device (201). The electronic device (100) can control the electronic device (100) based on the second user input (S3922-1).

[0386] The second terminal device (202) can receive a third user input for executing an application related to the electronic device (100) (S3905-2). When the third user input is received, the second terminal device (202) can execute the application (S3910-2). The second terminal device (202) can transmit an execution notification of the application to the electronic device (100) (S3911-2).

[0387] The electronic device (100) can receive an application execution notification from the second terminal device (202). The electronic device (100) can display a first pointer UI (10-2) corresponding to the second terminal device (202).

[0388] The second terminal device (202) can display a screen provided by the application (S3915-2). The second terminal device (202) can receive a fourth user input for controlling the electronic device (100) through the screen (S3920-2). The second terminal device (202) can transmit the fourth user input to the electronic device (100) (S3921-2).

[0389] The electronic device (100) can receive a fourth user input from the second terminal device (202). The electronic device (100) can control the electronic device (100) based on the fourth user input (S3922-2).

[0390] FIG. 40 is a drawing for explaining an operation of differently displaying a pointer UI (10-1, 10-2) according to a terminal device (201, 202) according to one or more embodiments.

[0391] Referring to the embodiment (4000) of FIG. 40, the electronic device (100) may be connected to a plurality of devices capable of controlling the electronic device (100). The plurality of devices may be a terminal device (201) and a terminal device (202). The terminal device (201) may be described as a first terminal device. The terminal device (202) may be described as a second terminal device.

[0392] When multiple terminal devices (201, 202) are identified as connected, the electronic device (100) can display separate screens (4010, 4020) corresponding to each of the multiple terminal devices (201, 202).

[0393] The electronic device (100) can display pointer UIs (10-1, 10-2) for each of the plurality of connected terminal devices (201, 202) on separate screens. The electronic device (100) can display the pointer UI (10-1) corresponding to the terminal device (201) on a first screen (4010) corresponding to the first terminal device (201). The electronic device (100) can display the pointer UI (10-2) corresponding to the terminal device (202) on a second screen (4020) corresponding to the second terminal device (202). The pointer UI (10-1) can be described as the first pointer UI. The pointer UI (10-2) can be described as the second pointer UI.

[0394] The content included in the first screen (4010) and the second screen (4020) may be the same. The difference between the first screen (4010) and the second screen (4020) may be the location and shape of the pointer UI.

[0395] The electronic device (100) can display the pointer UI (10-1) in a first form (or first shape).

[0396] The electronic device (100) can display the pointer UI (10-2) in a second form (or second shape). The second form may be different from the first form.

[0397] According to one or more embodiments, the electronic device (100) may include identification information (user1, user2) indicating the subject of the pointer UI. The pointer UI (10-1) may include identification information (user1) indicating the terminal device (201). The identification information may be displayed as an image or text. The pointer UI (10-2) may include identification information (user2) indicating the terminal device (202).

[0398] FIG. 41 is a drawing for explaining an operation in which an electronic device (100) receives touch data and motion data from a terminal device (200).

[0399] Referring to FIG. 41, the terminal device (200) can obtain touch data and motion data (S4105). The terminal device (200) can transmit the touch data and motion data to the electronic device (100) (S4110).

[0400] The electronic device (100) can receive touch data and motion data from the terminal device (200). The electronic device (100) can determine whether a preset event is identified based on at least one of the touch data and the motion data (S4115).

[0401] The preset events related to the transmission of motion data may include at least one of an event of touching a preset area, an event of touching a floating UI, an event of touching an arbitrary area for a threshold time or longer, an event of touching (pressing) a physical button, and an event of touching a preset UI.

[0402] In relation to an event of touching a physical button, the touch data may include operation data of the terminal device (200). The operation data may include data related to an input to a physical button of the terminal device (200).

[0403] If a preset event is not identified (S4115-N), the electronic device (100) can perform a function corresponding to the touch data (S4120). The function corresponding to the touch data is described in FIG. 12.

[0404] As one or more examples, the electronic device (100) can obtain location information of the pointer UI (10) based on touch data (S4121). The electronic device (100) can display the pointer UI (10) based on the location information (S4122). Steps S4121 and S4122 may correspond to steps S1634 and S1635 of FIG. 16. Duplicate descriptions are omitted.

[0405] When a preset event is identified (S4115-Y), the electronic device (100) can obtain location information of the pointer UI (10) based on motion data (S4125). The electronic device (100) can display the pointer UI (10) based on the location information (S4130). Steps S4125 and S4130 may correspond to steps S1844 and S1845 of FIG. 18. Duplicate descriptions are omitted.

[0406] FIG. 42 is a drawing for explaining a method of controlling an electronic device (100) according to one or more embodiments.

[0407] Referring to FIG. 42, a control method of an electronic device connected to a terminal device includes a step (S4205) of displaying a screen including a plurality of objects and a pointer UI (User Interface). The method may include a step of receiving touch data from a terminal device operating in a first mode or motion data from a terminal device operating in a second mode through a communication interface (130). The method may include a step (S4210) of performing a function corresponding to the touch data when touch data is received from a terminal device operating in a first mode for transmitting the touch data, and a step (S4215) of controlling a pointer UI to move based on the motion data when motion data is received from a terminal device operating in a second mode for transmitting the motion data based on a preset event.

[0408] The control method may include a step of displaying a pointer UI when a notification indicating that an application for controlling an electronic device has been executed on the terminal device is received.

[0409] The function corresponding to the touch data may include at least one of the following: the function of moving the pointer UI, the function of selecting an object corresponding to the location where the pointer UI is displayed among multiple objects, the function of displaying the home screen, the function of displaying the previous screen, or the function of scrolling the current screen.

[0410] Touch data may include at least one of a drag signal, a tap signal, a long tap signal, a double tap signal, or an edge scroll signal.

[0411] The preset event may include at least one of an event in which a preset area of ​​the entire display area of ​​the terminal device is touched, an event in which a floating UI displayed on the display of the terminal device is touched, an event in which any area of ​​the entire display area of ​​the terminal device is touched for a threshold period of time or longer, an event in which a preset physical button of the terminal device is touched, and an event in which a preset UI displayed on the display of the terminal device is touched.

[0412] The step (S4210) of performing a function corresponding to touch data may include performing a function corresponding to the touch data when touch data including a user input received through a first area among the entire area of ​​the display of the terminal device is received, and the step (S4215) of controlling the pointer UI to move based on the motion data may include controlling the pointer UI to move based on the motion data when the motion data is received based on a user input received through a second area among the entire area of ​​the display of the terminal device.

[0413] Step (S4215) of controlling the pointer UI to move based on motion data may be performed by controlling the pointer UI to move based on the motion data when motion data obtained while a user input for selecting a floating UI displayed on a display of a terminal device is received.

[0414] The motion data may include gyro data acquired through a gyro sensor included in the terminal device.

[0415] The control method includes a step of adjusting a movement speed of the pointer UI based on movement pattern information of the pointer UI stored in the electronic device, and the movement pattern information of the pointer UI may include information indicating a pattern in which the pointer UI is moved to select an object.

[0416] The control method may include a step of displaying identification information corresponding to the terminal device together with a pointer UI.

[0417] Embodiments of the methods and devices described herein enhance the functionality of a computer by enabling the control of an electronic device with a display through a terminal device utilizing touch input. The problem of controlling an electronic device with a display through a terminal device utilizing touch input is a problem emerging in the computing and networking fields. Therefore, the embodiments of this specification were developed based on computer technology to address this problem arising in the field of computer networks.

[0418] The methods according to the various embodiments of the present disclosure described above can be implemented in the form of an application that can be installed on an existing electronic device.

[0419] The methods according to the various embodiments of the present disclosure described above can be implemented only with a software upgrade or a hardware upgrade for an existing electronic device.

[0420] The various embodiments of the present disclosure described above may also be performed through an embedded server provided in an electronic device, or an external server of at least one of the electronic device and the display device.

[0421] According to an example embodiment of the present disclosure, the various embodiments described above may be implemented as software including instructions stored in a machine-readable storage medium that can be read by a machine (e.g., a computer). The device may include an electronic device according to the disclosed embodiments, which is a device that can call instructions stored in the storage medium and operate according to the called instructions. When the instructions are executed by a processor, the processor may directly or under the control of the processor use other components to perform a function corresponding to the instructions. The instructions may include code generated or executed by a compiler or interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' means that the storage medium does not contain signals and is tangible, but does not distinguish between whether data is stored semi-permanently or temporarily in the storage medium.

[0422] According to one or more embodiments of the present disclosure, the methods according to the various embodiments described above may be provided as included in a computer program product. The computer program product may be traded as a commodity between sellers and buyers. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or online through an application store. In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0423] Each of the components (e.g., modules or programs) according to the various embodiments described above may be composed of a single or multiple entities, and some of the sub-components described above may be omitted, or other sub-components may be further included in various embodiments. Alternatively or additionally, some components (e.g., modules or programs) may be integrated into a single entity, which may perform the same or similar functions as those performed by each of the respective components prior to integration. Operations performed by modules, programs or other components according to various embodiments may be executed sequentially, in parallel, iteratively or heuristically, or at least some operations may be executed in a different order, omitted, or other operations may be added.

[0424] Although the preferred embodiments of the present disclosure have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above, and various modifications may be made by a person having ordinary skill in the art to which the present disclosure pertains without departing from the scope of the present disclosure as claimed in the claims, and such modifications should not be understood individually from the technical idea of ​​the present disclosure.

Claims

1. In electronic devices, Memory that stores instructions; Communication interface connecting to a terminal device; display; and At least one processor comprising processing circuitry, The above instructions, when individually or collectively executed by the at least one processor, Control the display to display a screen and pointer UI (User Interface) including multiple objects, Through the above communication interface, touch data is received from the terminal device operating in the first mode or motion data is received from the terminal device operating in the second mode, When the above touch data is received, a function corresponding to the touch data is performed, When the above motion data is received, the display is controlled to move the pointer UI based on the motion data, An electronic device wherein the terminal device operates in the second mode based on a preset event.

2. In paragraph 1, The above instructions, when individually or collectively executed by the at least one processor, Receiving a notification from the terminal device through the communication interface, wherein the notification indicates that an application for controlling the electronic device has been executed, An electronic device that controls the display to display the pointer UI when the above notification is received.

3. In paragraph 1, The function corresponding to the above touch data is: An electronic device comprising at least one of a function of moving the pointer UI, a function of selecting an object corresponding to a location where the pointer UI is displayed among the plurality of objects, a function of displaying a home screen, a function of displaying a previous screen, or a function of scrolling a current screen.

4. In paragraph 3, The above touch data is, An electronic device comprising at least one of a drag signal, a tap signal, a long tap signal, a double tap signal, or an edge scroll signal.

5. In paragraph 1, The above preset events are: An electronic device comprising at least one of an event in which a preset area of ​​the display of the terminal device is touched, an event in which a floating UI displayed on the display of the terminal device is touched, an event in which one area of ​​the display of the terminal device is touched for a threshold period of time or longer, an event in which a preset physical button of the terminal device is touched, and an event in which a preset UI displayed on the display of the terminal device is touched.

6. In paragraph 1, The above instructions, when individually or collectively executed by the at least one processor, When the touch data including the user input received through the first area of ​​the display of the terminal device is received, a function corresponding to the touch data is performed, An electronic device that controls the display so that the pointer UI moves based on the motion data when the motion data is received based on a user input received through the second area of ​​the display of the terminal device.

7. In paragraph 1, The above instructions, when individually or collectively executed by the at least one processor, An electronic device that controls the display to move the pointer UI based on the motion data obtained while receiving a user input for selecting a floating UI displayed on the display of the terminal device.

8. In paragraph 1, The above motion data is, An electronic device comprising gyroscope data obtained through a gyro sensor included in the terminal device or acceleration data obtained through an acceleration sensor included in the terminal device.

9. In paragraph 1, The above instructions, when individually or collectively executed by the at least one processor, Adjusting the movement speed of the pointer UI based on the movement pattern information of the pointer UI stored in the memory, The movement pattern information of the above pointer UI is, An electronic device comprising information indicating a pattern in which the pointer UI is moved to select an object.

10. In paragraph 1, The above instructions, when individually or collectively executed by the at least one processor, An electronic device that controls the display to display identification information corresponding to the terminal device together with the pointer UI.

11. In a method for controlling an electronic device connected to a terminal device, A step of displaying a screen and a pointer UI (User Interface) including multiple objects; A step of receiving touch data from the terminal device operating in the first mode or receiving motion data from the terminal device operating in the second mode; When the above touch data is received, a step of performing a function corresponding to the touch data; and When the motion data is received, a step of controlling the pointer UI to move based on the motion data is included; A control method in which the terminal device operates in the second mode based on a preset event.

12. In paragraph 11, The above control method is, A step of receiving a notification from the terminal device, wherein the notification indicates that an application for controlling the electronic device has been executed, A control method comprising: a step of displaying the pointer UI when the above notification is received; 13. In paragraph 11, The function corresponding to the above touch data is: A control method comprising at least one of a function of moving the pointer UI, a function of selecting an object corresponding to a location where the pointer UI is displayed among the plurality of objects, a function of displaying a home screen, a function of displaying a previous screen, or a function of scrolling the current screen.

14. In paragraph 13, The above touch data is, A control method comprising at least one of a drag signal, a tap signal, a long tap signal, a double tap signal, or an edge scroll signal.

15. In paragraph 11, The above preset events are: A control method comprising at least one of an event in which a preset area of ​​the display of the terminal device is touched, an event in which a floating UI displayed on the display of the terminal device is touched, an event in which one area of ​​the display of the terminal device is touched for a threshold period of time or longer, an event in which a preset physical button of the terminal device is touched, and an event in which a preset UI displayed on the display of the terminal device is touched.

Images