Electronic device and control method thereof
The electronic device and control method improve content manipulation by using guide UIs on both the device and external devices, addressing the lack of precision with external controllers, and enhancing user experience by adjusting UI positions and transparencies based on user inputs and device sizes.
Patent Information
- Application Number
- PCT/KR2024/001167
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-11
- Filing Date
- 2024-01-25
- Publication Date
- 2024-11-14
AI Technical Summary
Users face difficulties in accurately manipulating content on electronic devices when a dedicated controller is not available or connected, and existing solutions for using external devices as controllers are not precise enough.
An electronic device and control method that communicates with external devices to display a guide UI corresponding to user touch inputs, allowing for accurate manipulation by adjusting the position and transparency of control UIs on both the electronic device and external devices based on user inputs and device sizes.
Enables accurate and intuitive content manipulation on electronic devices using external devices by providing a guide UI that enhances touch precision and user experience, even without a dedicated controller.
Smart Images

Figure KR2024001167_14112024_PF_FP_ABST
Abstract
Description
Electronic device and method of controlling the same
[0001] The present disclosure relates to an electronic device and a control method thereof, and more particularly, to an electronic device and a control method thereof that transmit UI information to an external device so that a control UI is displayed on the external device.
[0002] Electronic devices that include displays, such as TVs, can display a variety of content. Depending on the content, a dedicated controller (e.g., a gamepad) may be required. Users can use the dedicated controller to manipulate the content displayed on the TV.
[0003] If a dedicated controller for manipulating content cannot be connected to the TV (for example, a dedicated controller has not been purchased or the dedicated controller lacks power), the user will have the problem of not being able to manipulate content.
[0004] If a dedicated controller cannot be connected to the TV, the electronic device can display a virtual control UI on an external device (e.g., a smartphone). Users can manipulate content through the control UI displayed on the external device instead of the dedicated controller.
[0005] When manipulating content through the control UI of an external device rather than a dedicated controller, there is a problem that it is difficult to achieve accurate touch.
[0006] The present disclosure is designed to improve the above-described problem, and an object of the present disclosure is to provide an electronic device that displays a guide UI corresponding to a user's touch and a method for controlling the same.
[0007] According to one embodiment, an electronic device includes a communication interface for communicating with an external device, a display, at least one memory for storing commands, and at least one processor operatively connected to the communication interface, the display, and the at least one memory, wherein the at least one processor executes the commands to, when content is output to the display, transmit first UI information corresponding to a UI (User Interface) related to control of the content to the external device through the communication interface, and control the display to output a first control UI together with the content based on the first UI information, and when a user input is received through a second control UI output to the external device based on the first UI information, control the display to output a guide UI at an input position corresponding to the user input together with the content and the first control UI based on a first UI output position of the first control UI, a second UI output position of the second control UI, and input information corresponding to the user input.
[0008] The at least one processor executing the above commands can obtain second UI information based on the first UI information and size information of the external device through the communication interface, and transmit the second UI information to the external device through the communication interface.
[0009] The at least one processor executing the above commands can obtain third UI information based on the first UI information and the size information of the electronic device, and control the display to output the first control UI together with the content based on the third UI information.
[0010] The at least one processor executing the commands may, when the user input is received through the second control UI output to the external device based on the second UI information, obtain the first UI output position based on the third UI information, obtain the second UI output position based on the second UI information, and obtain the input position based on the input information.
[0011] The at least one processor executing the above commands can control the display to output at least one of the first control UI or the guide UI together with the content based on a preset transparency.
[0012] The at least one processor executing the above commands can control the display to output the first control UI at the first UI output location corresponding to the user input when a user input for changing the first UI output location is received.
[0013] The at least one processor executing the above commands can obtain a device location of the external device and control the display to output the first control UI at the first UI output location corresponding to the device location.
[0014] The at least one processor executing the above commands can control the display to output the first control UI including at least one button among the plurality of buttons included in the first UI information.
[0015] The at least one processor executing the above commands can control the display to change the color of the guide UI and output it based on whether the input location is in a preset area when the user input is received through the second control UI output to the external device.
[0016] The at least one processor executing the above commands may transmit a signal for controlling the second UI output position to the external device through the communication interface according to an input pattern obtained based on the input information corresponding to the plurality of user inputs when input information corresponding to a plurality of user inputs is received within a preset time through the second control UI outputted to the external device.
[0017] According to one embodiment, a method for controlling an electronic device includes the steps of: when content is output, transmitting first UI information corresponding to a UI (User Interface) related to control of the content to an external device; outputting a first control UI together with the content based on the first UI information; and when a user input is received through a second control UI output to the external device based on the first UI information, outputting a guide UI at an input position corresponding to the user input together with the content and the first control UI based on a first UI output position of the first control UI, a second UI output position of the second control UI, and input information corresponding to the user input.
[0018] The step of transmitting information corresponding to the UI to the external device may include obtaining second UI information based on the first UI information and size information of the external device, and transmitting the second UI information to the external device.
[0019] The step of outputting the first control UI may include obtaining third UI information based on the first UI information and size information of the electronic device, and outputting the first control UI together with the content based on the third UI information.
[0020] The step of outputting the guide UI may include, when the user input is received through the second control UI output to the external device based on the second UI information, obtaining the first UI output position based on the third UI information, obtaining the second UI output position based on the second UI information, and obtaining the input position based on the input information.
[0021] The above control method may further include a step of outputting at least one of the first control UI or the guide UI together with the content based on a preset transparency.
[0022] The above control method may further include a step of outputting the first control UI at the first UI output location corresponding to the user input when a user input for changing the first UI output location is received.
[0023] The step of outputting the first control UI may obtain the device location of the external device and output the first control UI to the first UI output location corresponding to the device location.
[0024] The step of outputting the first control UI may output the first control UI including at least one button among a plurality of buttons included in the first UI information.
[0025] The step of outputting the above guide UI may include, when the user input is received through the second control UI output to the external device, changing the color of the guide UI and outputting it based on whether the input location is in a preset area.
[0026] The above control method may further include a step of transmitting a signal for controlling the second UI output position to the external device according to an input pattern obtained based on the input information corresponding to the plurality of user inputs when input information corresponding to a plurality of user inputs is received within a preset time through the second control UI output to the external device.
[0027] The above and other aspects, features and advantages of specific embodiments of the present invention will become more apparent from the following description taken in conjunction with the accompanying drawings.
[0028] FIG. 1 is a drawing for explaining a system including an electronic device and an external device.
[0029] FIG. 2 is a drawing for explaining a control UI displayed on an electronic device and an external device.
[0030] FIG. 3 is a block diagram illustrating an electronic device according to an embodiment of the present disclosure.
[0031] Figure 4 is a block diagram for explaining the specific configuration of the electronic device of Figure 3.
[0032] Figure 5 is a drawing for explaining an operation of displaying a connection guide for an electronic device and an external device.
[0033] Figure 6 is a drawing for explaining an operation of displaying a guide UI according to user input.
[0034] Figure 7 is a drawing for explaining an operation of displaying a guide UI according to multiple user inputs.
[0035] FIG. 8 is a drawing for explaining an operation of displaying a control UI according to various embodiments.
[0036] FIG. 9 is a drawing for explaining an operation of displaying a control UI according to the relative positions of an electronic device and an external device.
[0037] Figure 10 is a drawing for explaining an operation of adjusting the number of multiple elements included in a control UI.
[0038] Figure 11 is a drawing for explaining how to adjust the transparency of the control UI.
[0039] FIG. 12 is a drawing for explaining a control UI corresponding to content according to various embodiments.
[0040] FIG. 13 is a drawing for explaining a control UI corresponding to a controller according to various embodiments.
[0041] Figure 14 is a drawing for explaining a communication method between an electronic device and an external device.
[0042] Figure 15 is a drawing for explaining areas related to selection of buttons (elements) in the control UI.
[0043] Figure 16 is a drawing for explaining areas related to each of a plurality of buttons (elements).
[0044] Figure 17 is a drawing for explaining an operation of controlling a control UI through voice recognition.
[0045] Figure 18 is a flowchart for explaining the operation of displaying the control UI and guide UI.
[0046] Figure 19 is a flowchart for explaining a second control UI displayed on an electronic device and a third control UI displayed on an external device.
[0047] Figure 20 is a flowchart for explaining the guide screen displayed in relation to the connection of electronic devices, controllers, and external devices.
[0048] Figure 21 is a flowchart for explaining the operation of displaying a guide UI.
[0049] Figure 22 is a flowchart for explaining the operation of displaying a guide UI when a button is not recognized.
[0050] Figure 23 is a flowchart for explaining the operation of displaying a guide UI corresponding to user input.
[0051] Figure 24 is a drawing for explaining an operation of changing the position of the control UI.
[0052] Figure 25 is a drawing for explaining an operation of analyzing a captured image including a user.
[0053] FIG. 26 is a flowchart for explaining a method of controlling an electronic device according to various embodiments.
[0054] Hereinafter, the present disclosure will be described in detail with reference to the attached drawings.
[0055] The terms used in the embodiments of this disclosure have been selected from widely used, current terms, taking into account the functions of this disclosure. However, these terms may vary depending on the intentions of those skilled in the art, precedents, the emergence of new technologies, etc. In certain cases, terms may be arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the description of the relevant disclosure. Therefore, the terms used in this disclosure should not be defined simply as names of terms, but rather based on the meanings of the terms and the overall content of this disclosure.
[0056] In this specification, expressions such as “has,” “can have,” “includes,” or “may include” indicate the presence of a feature (e.g., a number, function, operation, or component such as a part), and do not exclude the presence of additional features.
[0057] The expression "at least one of A and / or B" should be understood to mean either "A" or "B" or "A and B." As another example, "performing at least one of steps 1 and 2" or "performing at least one of steps 1 or 2" means the following three juxtapositions: (1) performing step 1, (2) performing step 2, and (3) performing steps 1 and 2.
[0058] As used herein, the expressions “first,” “second,” “first,” or “second,” etc., may describe various components, regardless of order and / or importance, and are only used to distinguish one component from another, but do not limit the components.
[0059] When it is said that a component (e.g., a first component) is “(operatively or communicatively) coupled with / to” or “connected to” another component (e.g., a second component), it should be understood that the component may be directly coupled to the other component, or may be connected through another component (e.g., a third component).
[0060] Singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, terms such as "comprise" or "consist of" are intended to indicate the presence of a feature, number, step, operation, component, part, or combination thereof described in the specification, but should be understood not to preclude the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.
[0061] Elements described as "modules" or "components" may be physically implemented by analog and / or digital circuits that include one or more of logic gates, integrated circuits, microprocessors, microcontrollers, memory circuits, passive electronic components, and active electronic components.
[0062] In this specification, the term user may refer to a person using an electronic device or a device using an electronic device (e.g., an artificial intelligence electronic device).
[0063] An embodiment of the present disclosure will be described in more detail with reference to the attached drawings below.
[0064] FIG. 1 is a drawing for explaining a system (1000) including an electronic device (100) and an external device (200).
[0065] Referring to FIG. 1, the system (1000) may include at least one of an electronic device (100) or an external device (200).
[0066] The electronic device (100) may include a display. The electronic device (100) may display content. The content may include image frames. The content may refer to still images or moving images. For example, the content may refer to games, broadcast content, educational content, etc. The content is not limited to any particular embodiment.
[0067] The external device (200) may include a display. The external device (200) may display a second control UI (20) related to content displayed on the electronic device (100) on the display. The second control UI (20) may include at least one UI for manipulating content displayed on the electronic device (100). The second control UI (20) may vary depending on the content displayed on the electronic device (100).
[0068] The second control UI (20) displayed on the external device (200) may be displayed as the first control UI (10) on the electronic device (100). In the description below, the first control UI (10) is described as being displayed on the electronic device (100). Depending on the implementation example, the first control UI (10) may not be displayed on the electronic device (100).
[0069] Control UI can be described as UI related to control, UI for control, UI for operation, UI related to operation, UI for operation, etc.
[0070] FIG. 2 is a drawing for explaining a control UI displayed on an electronic device (100) and an external device (200).
[0071] Referring to FIG. 2, it is assumed that content A is displayed on an electronic device (100). A dedicated controller may exist for manipulating content A. It is assumed that the dedicated controller is a controller (300). The controller (300) may mean at least one of a physical hardware device or a virtual UI (virtual control UI).
[0072] The controller (300) may include at least one button (or element). UI information corresponding to the controller (300) may include information related to the button of the controller (300). The UI information may include at least one of the shape, position, area, size, color, or transparency of the button. The area may indicate the area information described in FIG. 15. Area information may be included only in some cases and may not be included in some UI information.
[0073] Based on the UI information corresponding to the controller (300), the electronic device (100) or external device (200) can display a control UI.
[0074] According to various embodiments, the arrangement structure of the buttons included in the controller (300) may be displayed on the electronic device (100) or external device (200).
[0075] According to various embodiments, the arrangement structure of the buttons included in the controller (300) may be partially changed and displayed on the electronic device (100) or the external device (200). This is because the sizes of the electronic device (100) and the external device (200) are different. If the UI information corresponding to the controller (300) is displayed as is without changing, there is a problem that the user may experience inconvenience due to the arrangement or size of the control UI.
[0076] Accordingly, the UI information of the controller (300) may be partially modified so that the electronic device (100) or the external device (200) can display the control UI. The UI displayed on the electronic device (100) may be described as the first control UI (10), and the UI displayed on the external device (200) may be described as the second control UI (20).
[0077] The controller (300) may be described as a remote control device, joystick, game pad, operating pen, etc.
[0078] FIG. 3 is a block diagram illustrating an electronic device (100) according to one embodiment of the present disclosure.
[0079] Referring to FIG. 3, the electronic device (100) may include at least one of a communication interface (110), a display (120), or at least one processor (130).
[0080] The electronic device (100) may include various devices including a display. The electronic device (100) may be an electronic whiteboard, a TV, a desktop PC, a laptop, a smartphone, a tablet PC, a server, etc. The above-described examples are merely examples for describing the electronic device and are not necessarily limited to the above-described devices.
[0081] At least one processor (130) can perform overall control operations of the electronic device (100). Specifically, at least one processor (130) has a function of controlling overall operations of the electronic device (100).
[0082] At least one processor (130) may include at least one of a communication interface (110) for communicating with an external device (200), a display (120), or at least one processor (130).
[0083] At least one processor (130) can control the display (120) to transmit information corresponding to a UI (User Interface) related to control of the content to an external device (200) through a communication interface (110) when content is output to the display (120), and to output a first control UI together with the content based on first UI information corresponding to the UI. When a user input is received through a second control UI output to the external device (200) based on the information corresponding to the UI, at least one processor (130) can control the display (120) to output a guide UI at an input position corresponding to the user input together with the content and the first control UI based on the first UI output position of the first control UI, the second UI output position of the second control UI, and the information corresponding to the user input. At least one processor (130) can output the guide UI by overlapping the first control UI.
[0084] Information corresponding to a UI can be described as UI information. User input can be described as touch input. Information corresponding to user input can be described as touch information. The location corresponding to user input can be described as the location corresponding to touch information or as the touch location.
[0085] At least one processor (130) can display content on the display (120). The content can be selected by a user or selected according to a preset control command. The content can include image frames and can be provided to the user through the display (120) of the electronic device (100).
[0086] Content may be content that requires user interaction. Content may be content whose detailed structure (image information, audio information, etc.) changes based on user interaction. For example, content may refer to games or educational content. However, content is not limited to these and may refer to various types of content that require user interaction. Content may be described as an application.
[0087] To manipulate content, a controller (300) may be required. The controller (300) may refer to a dedicated controller or joystick required to manipulate content. The controller (300) may be described as a remote control device. When the controller (300) is connected to the electronic device (100), at least one processor (130) may receive user input from the controller (300).
[0088] At least one processor (130) can manipulate content based on user input. The act of manipulating content may refer to an act of changing an image (or object) included in the content based on the user input. For example, when a user input is received through the controller (300), at least one processor (130) may display a screen showing a character in the game moved based on the user input. The controller (300) may be a control device including at least one physical button.
[0089] At least one processor (130) can identify whether the output (or selected) content requires a controller (300). If the content requires a controller (300), at least one processor (130) can identify UI information corresponding to the controller (300).
[0090] UI information may include information related to at least one physical button included in the controller (300). The UI information may include at least one of the shape, position, area, size, color, or transparency of the button (or element). The area may represent the area information described in FIG. 15. Area information may be included only in some cases and may not be included in some UI information.
[0091] Even if a dedicated controller (300) is not connected to the content, at least one processor (130) can manipulate the content through an external device (200). The external device (200) may include a display. The external device (200) may display a second control UI (20) on the display of the external device (200) based on UI information received from the electronic device (100).
[0092] At least one processor (130) can display at least one of a first control UI (10) or a guide UI (30) on the display (120). The first control UI (10) and the guide UI (30) may be UIs that provide a touch position input through a second control UI (20) output to an external device (200).
[0093] The first control UI (10) may include a UI related to a button corresponding to the controller (300). The first control UI (10) may be a UI displayed on an electronic device (100). An example related to the first control UI (10) is described in FIG. 8.
[0094] The second control UI (20) may include a UI related to a button corresponding to the controller (300). The second control UI (20) may be a UI displayed on an external device (200). Examples related to the second control UI (20) are described in FIGS. 12 and 13.
[0095] The guide UI (30) may be a UI for notifying the user of the touch location. The guide UI (30) may be a UI displayed on the electronic device (100). Examples related to the guide UI (30) are described in FIGS. 6 and 7. For example, the guide UI (30) may be an image (or icon) representing the user's fingerprint.
[0096] The external device (200) can receive user input through the output second control UI (20). The external device (200) can transmit information (or touch information) related to the user input to the electronic device (100).
[0097] At least one processor (130) can receive information (or touch information) related to user input through an external device (200) and display a guide UI (30) at a location corresponding to the user input.
[0098] At least one processor (130) can identify a location corresponding to a user input (or a location corresponding to the touch information) based on the output location (or location information) of the first control UI (10), the output location (or location information) of the second control UI (20), and the touch location (or location information) included in the touch information.
[0099] The output location of the first control UI (10) may be the location (or coordinates) where the first control UI (10) is output among the entire display area of the electronic device (100).
[0100] The output location of the second control UI (20) may be the location (or coordinates) where the second control UI (20) is output among the entire display area of the external device (200).
[0101] The touch location included in the touch information may mean the location (or coordinates) where the user's touch is identified among the entire area of the display of the external device (200).
[0102] According to various embodiments, at least one processor (130) may obtain the output location of the second control UI (20) before touch information is received.
[0103] According to various embodiments, at least one processor (130) may receive (or obtain) the output location of the second control UI (20) together with touch information from an external device (200).
[0104] At least one processor (130) can obtain relative position information of a user input based on the output position and touch position of the second control UI (20). The relative position information can include information indicating at which position the user input was entered based on a specific button.
[0105] Relative position information may include distance values for the upper, lower, left, and right sides of a specific button. For example, relative position information may include information indicating that a user input was entered 1 cm to the left and 1 cm below the first button.
[0106] Relative position information may include angle and distance values relative to a specific button. For example, relative position information may include information indicating that a user input was entered at 225 degrees square root (1.414, etc.) cm relative to the first button.
[0107] At least one processor (130) can display a guide UI (30) based on the output position and relative position information of the first control UI (10). At least one processor (130) can determine a position at which the guide UI (30) is to be displayed, at a position that is a distance from the output position of the first control UI (10) by a value included in the relative position information. At least one processor (130) can display the guide UI (30) at the determined position.
[0108] According to various embodiments, the display size of the first control UI (10) and the display size of the second control UI (20) may be different. At least one processor (130) may identify a scaling factor by considering the display size of the first control UI (10) and the display size of the second control UI (20).
[0109] At least one processor (130) can display a guide UI (30) based on the output position, relative position information, and a scaling factor of the first control UI (10). The at least one processor (130) can obtain scaled relative position information by multiplying the relative position information by the scaling factor. The at least one processor (130) can determine a position at which the guide UI (30) is to be displayed based on the output position of the first control UI (10) and the scaled relative position information. The at least one processor (130) can display the guide UI (30) at the determined position.
[0110] The UI information may be first UI information. The first UI information may include information related to at least one UI representing a button included in the controller (300). The UI-related information may include at least one of the shape, position, size, color, or transparency of the button of the UI.
[0111] For example, the controller (300) may include four directional buttons and four function buttons. The controller (300) may include a total of eight buttons. The first UI information may include information related to the UI representing each of the eight buttons included in the controller (300).
[0112] At least one processor (130) can identify first UI information corresponding to content among a plurality of first UI information.
[0113] According to various embodiments, the first UI information may be information corresponding to content. At least one processor (130) may identify first UI information corresponding to selected content (or content being displayed). A mapping table may map content and first UI information corresponding to the content. The mapping table may include first UI information corresponding to each of a plurality of contents. At least one processor (130) may identify first UI information corresponding to the content based on the mapping table.
[0114] According to various embodiments, the first UI information may be information corresponding to the controller (300). At least one processor (130) may identify the controller (300) corresponding to the selected content (or the content being displayed) and identify the first UI information corresponding to the identified controller (300). The mapping table may map content, a controller corresponding to the content, and first UI information corresponding to the controller. The mapping table may include controller information corresponding to each of a plurality of contents and first UI information corresponding to each of the controllers. At least one processor (130) may identify the first UI information corresponding to the content based on the mapping table.
[0115] The mapping table may be stored in at least one of the electronic device (100), a server providing a service in the electronic device (100), or a server providing content.
[0116] At least one processor (130) can obtain second UI information based on first UI information and size information of an external device (200) through a communication interface (110), and transmit the second UI information to the external device (200) through the communication interface (110).
[0117] The first UI information may contain information related to the control UI for manipulating content, but may only contain information that does not take into account the display size. The first UI information may not include information indicating where on the display it is displayed.
[0118] At least one processor (130) can obtain new UI information (second UI information) so that the first UI information corresponding to the content can be appropriately displayed on the external device (200). At least one processor (130) can obtain second UI information by reflecting size information of the external device (200) into the first UI information. The second UI information can include at least one of the shape, position, size, color, or transparency of a button.
[0119] The second UI information may include information (or location information) indicating the location of a button. The location of the button may be information determined by considering the size information of the external device (200). The size information of the external device (200) may include the size of the display of the external device (200).
[0120] At least one processor (130) can transmit second UI information to an external device (200). The external device (200) can receive the second UI information from the electronic device (100). The external device (200) can display a second control UI (20) based on the position of a button included in the second UI information.
[0121] According to various embodiments, at least one processor (130) may transmit first UI information to an external device (200). The external device (200) may obtain second UI information based on the first UI information and size information of the external device (200).
[0122] At least one processor (130) can obtain third UI information based on the first UI information and the size information of the electronic device (100), and control the display (120) to output the first control UI together with the content based on the third UI information.
[0123] At least one processor (130) may obtain new UI information (third UI information) so that the first UI information corresponding to the content can be appropriately displayed on the electronic device (100). At least one processor (130) may obtain third UI information by reflecting size information of the electronic device (100) into the first UI information. The third UI information may include at least one of the shape, position, size, color, or transparency of a button.
[0124] The third UI information may include information (or location information) indicating the location of a button. The location of the button may be information determined by considering the size information of the electronic device (100). The size information of the electronic device (100) may include the size of the display of the electronic device (100).
[0125] At least one processor (130) may, when a user input is received through a second control UI output to an external device (200) based on second UI information, obtain an output position of the first control UI based on third UI information, obtain an output position of the second control UI based on second UI information, and obtain a position corresponding to the user input based on information corresponding to the user input.
[0126] At least one processor (130) can identify an output location of the first control UI (10) based on the third UI information, identify an output location of the second control UI (20) based on the second UI information, and identify a touch location based on the touch information when touch information is received through the second control UI (20) output to the external device (200) based on the second UI information.
[0127] The third UI information may include information indicating a location where a button is displayed in the entire area of the display (120) of the electronic device (100). At least one processor (130) may identify a location where the first control UI (10) included in the third UI information is displayed.
[0128] The second UI information may include information indicating the location where a button is displayed in the entire area of the display (120) of the external device (200). At least one processor (130) may identify the location where the second control UI (20) included in the second UI information is displayed.
[0129] Touch information may include information (touch location) indicating a location where a user input (touch input) is obtained. At least one processor (130) may identify a touch location included in the touch information.
[0130] At least one processor (130) can identify a location where a guide UI (30) is to be displayed among the entire area of the display (120) of the electronic device (100) based on the second UI information, the third UI information, and the touch information.
[0131] At least one processor (130) can identify a location where a guide UI (30) is to be displayed based on the output location of the second control UI (20) obtained from the second UI information, the output location of the first control UI (10) obtained from the third UI information, and the touch location.
[0132] According to various embodiments, at least one processor (130) may obtain the output location (or location information) of the first control UI (10) and the output location (or location information) of the second control UI (20) in a separate information form other than the second UI information and the third UI information. At least one processor (130) may obtain first location information indicating the output location of the first control UI (10). At least one processor (130) may obtain second location information indicating the output location of the second control UI (20).
[0133] At least one processor (130) can control the display (120) to output at least one of the first control UI or the guide UI together with (or overlapping) the content based on a preset transparency.
[0134] The content displayed on the display (120) may be an image viewed by the user. The first control UI (10) and the guide UI (30) may be UIs output while the content is displayed on the display (120). Therefore, the first control UI (10) and the guide UI (30) should not interfere with the user's content viewing behavior. At least one processor (130) may adjust the transparency of at least one of the first control UI (10) and the guide UI (30) and display it on the display (120).
[0135] At least one processor (130) can display a first control UI (10) based on a first transparency and display a guide UI (30) based on a second transparency.
[0136] In various embodiments, the first transparency and the second transparency may be the same.
[0137] Depending on the various embodiments, the first transparency and the second transparency may be different.
[0138] According to various embodiments, the first transparency may be higher than the second transparency. The service that the electronic device (100) provides to the user is to accurately inform the user of the location of the user's touch. Accordingly, at least one processor (130) may display the first control UI (10) and the guide UI (30) by setting the second transparency to be lower than the first transparency so that the guide UI (30) can be displayed more clearly than the first control UI (10).
[0139] At least one processor (130) can control the display (120) to output the first control UI at a location corresponding to the user input when a user input for changing the location of the first control UI is received.
[0140] The location where the first control UI (10) is displayed by the initial setting may be inconvenient for the user. The user can change the displayed first control UI (10). When a user input is received, at least one processor (130) can display the first control UI (10) at a location corresponding to the user input.
[0141] At least one processor (130) may display the first control UI (10) at a second position corresponding to the user input instead of the initially set first position. If the position of the first control UI (10) changes, the position at which the guide UI (30) is displayed may also automatically change. This is because the guide UI (30) is displayed based on the relative position information of the button and the user input.
[0142] At least one processor (130) can determine the output location of the first control UI (10) according to user input.
[0143] According to various embodiments, if a user input is received before the first control UI (10) is initially displayed, the first control UI (10) can be displayed at a location corresponding to the user input.
[0144] According to various embodiments, when a user input is received after displaying the first control UI (10) at a first location, the first control UI (10) may be displayed at a second location corresponding to the user input. At least one processor (130) may change the output location of the first control UI (10) from the first location to the second location.
[0145] According to various embodiments, when a user input for changing the size of the first control UI (10) is received, the first control UI (10) can be displayed in a size corresponding to the user input through the display (120).
[0146] According to various embodiments, at least one processor (130) may display a first control UI (10) based on at least one of a content type or a content situation (or content status).
[0147] At least one processor (130) can display the first control UI (10) based on at least one of a size corresponding to the content type or a position corresponding to the content type.
[0148] For example, if the content type is sports, at least one processor (130) can display the first control UI (10) based on a first size (or first position). If the content type is FPS (First-person shooter), the first control UI (10) can be displayed based on a second size (or second position).
[0149] At least one processor (130) can display the first control UI (10) based on at least one of a size corresponding to the content situation or a position corresponding to the content situation.
[0150] For example, if the content situation is a match situation, at least one processor (130) can display the first control UI (10) based on a first size (or first position). If the content situation is a non-match situation, the first control UI (10) can be displayed based on a second size (or second position). At least one processor (130) can analyze whether the content situation is a match situation based on the content.
[0151] At least one processor (130) can control the display (120) to output the first control UI at a location corresponding to the user input when a user input for changing the location of the first control UI is received.
[0152] At least one processor (130) can identify the location of the external device (200) and display the first control UI (10) at a location corresponding to the location of the external device (200) through the display (120).
[0153] At least one processor (130) can identify the location of an external device (200).
[0154] According to various embodiments, at least one processor (130) may receive location information of the external device (200) from the external device (200). At least one processor (130) may determine a location at which the first control UI (10) is to be displayed based on the received location information of the external device (200).
[0155] According to various embodiments, at least one processor (130) may identify the location of an external device (200) based on a captured image acquired through a camera (190). The captured image may include the external device (200).
[0156] A specific description of the operation of determining the output position of the first control UI (10) in relation to the position of the external device (200) is described in FIG. 9.
[0157] At least one processor (130) can obtain the location of the external device (200) and control the display (120) to output the first control UI at a location corresponding to the location of the external device (200).
[0158] At least one processor (130) can display a first control UI (10) by excluding some buttons from among a plurality of buttons included in UI information (first UI information) through the display (120).
[0159] The first UI information may include information related to a plurality of buttons (information related to the UI). At least one processor (130) may display a first control UI (10) that includes information related to only some of the plurality of buttons included in the first UI information.
[0160] Some buttons may be designated as primary buttons (or primary keys). Multiple buttons may be classified as primary buttons and general buttons.
[0161] At least one processor (130) may obtain third UI information including information related to some of the buttons among the plurality of buttons included in the first UI information. At least one processor (130) may display a first control UI (10) including only some of the buttons based on the third UI information.
[0162] For example, if there are UIs corresponding to 12 buttons in the first UI information, at least one processor (130) can display the first control UI (10) including UIs corresponding to 5 buttons.
[0163] Some buttons may be designated as primary buttons. In a typical situation, at least one processor (130) may display all buttons. When a preset event is identified, at least one processor (130) may display some buttons (or primary buttons).
[0164] Predefined events can be identified based on communication strength. A detailed description of this is provided in Figure 10.
[0165] According to various embodiments, at least one processor (130) may identify a primary button based on at least one of a content type or a content situation.
[0166] At least one processor (130) can identify a primary button corresponding to a content type. At least one processor (130) can identify a primary button based on a mapping table related to a primary button corresponding to a content type.
[0167] One processor (130) can identify a primary button based on a content context. At least one processor (130) can identify a primary button based on a mapping table associated with a primary button corresponding to a content context.
[0168] According to various embodiments, at least one processor (130) may identify a button with a high selectability based on at least one of a content type or a content situation as a primary button.
[0169] According to various embodiments, at least one processor (130) may identify a primary button based on selection history information regarding the button. At least one processor (130) may identify a button among a plurality of buttons whose selection frequency ratio is greater than or equal to a threshold ratio (e.g., 20%) as a primary button.
[0170] According to various embodiments, at least one processor (130) may display the first control UI (10) by emphasizing some (main) buttons among a plurality of buttons included in the first control UI (10). At least one processor (130) may display the first control UI (10) by emphasizing at least one of shape, size, color, and transparency of the main button. For example, the emphasizing method may include a method of displaying the button in a size larger than that of a general button, a method of displaying the button in a preset color (e.g., red), and a method of displaying the button in a transparency lower than that of a general button, etc.
[0171] At least one processor (130) may display the first control UI (10) by highlighting some (main) buttons among a plurality of buttons based on the content being executed. For example, at least one processor (130) may determine whether to highlight some buttons based on at least one of the content type, the content status, or the application for executing the content. The content type may refer to the type of content provided by the electronic device (100). For example, the content type may refer to games, movies, news, sports, documentaries, dramas, music, etc.
[0172] The content status may refer to the status of content provided by the electronic device (100). For example, the content status may refer to a state in which the content is being streamed in real time, a state in which the content is being called from pre-stored data, a state in which the content is being played in a specific mode, etc. The specific mode may be a mode in which a preset UI different from the default UI is provided to the user when playing or providing the content.
[0173] An application for executing content may refer to an application used to provide content through an electronic device (100). For example, an application for executing content may refer to a video playback application, a music playback application, a game application, etc.
[0174] At least one processor (130) can determine a button to be highlighted based on at least one of the content type, the content status, or the application for executing the content, and display the first control UI (10) by highlighting the determined button.
[0175] At least one processor (130) may determine a button with a high probability of selection as the button to be highlighted. At least one processor (130) may store a user's button selection history. At least one processor (130) may determine the most frequently selected button as the button to be highlighted based on the user's button selection history. The user's button selection history may be stored differently depending on the content. The electronic device (100) may store first history information related to first content and second history information related to second content.
[0176] For example, the content may be game content. At least one processor (130) may determine a button to highlight based on at least one of the game type and the game status, and display the first control UI (10) by highlighting the determined button.
[0177] At least one processor (130) can control the display (120) to change the color of the guide UI and output it based on whether a location corresponding to the user input is included in a preset area when a user input is received through the second control UI output to the external device (200).
[0178] At least one processor (130) can, when touch information is received through the second control UI (20) output to the external device (200), identify whether the touch location is included in a preset area and, through the display (120), change the color of the guide UI (30) and display it based on the identification result.
[0179] User input can be received through a second control UI (20) displayed on an external device (200). When a user touches a specific location on the display of the external device (200), the external device (200) can determine whether the user's touch location has selected a specific button included in the second control UI (20).
[0180] According to various embodiments, the electronic device (100) may determine whether a touch location has selected a specific button. To determine whether a user has selected a specific button, a preset area associated with the specific button may exist. If the touch location falls within the preset area, at least one processor (130) may determine that the specific button has been selected.
[0181] A preset area corresponding to a specific button may include multiple areas. The preset area may include at least one of a button area, an auxiliary button area, and an expandable area.
[0182] A button area may be an area where a button is identified as being selected.
[0183] The auxiliary button area may be an area where a button is preliminarily recognized as selected based on user intent, even though the button is not selected by the button area. The auxiliary button area may be changed.
[0184] The expandable area may refer to an area in which the auxiliary button area is expandable. When a user input is received in the expandable area, at least one processor (130) identifies that the user has touched the area around the button, but the button is not selected as a result.
[0185] If the touch location is included in the button area of the first button or the auxiliary button area of the first button, at least one processor (130) can determine that the user has selected (or touched) the first button.
[0186] If the touch location is included in the button area of the first button, at least one processor (130) can display the guide UI (30) in the first color.
[0187] If the touch location is included in the auxiliary button area of the first button, at least one processor (130) can display the guide UI (30) in a second color.
[0188] If the touch location is included in the expandable area of the first button, at least one processor (130) can display the guide UI (30) in a third color.
[0189] In various embodiments, the first color, the second color, and the third color may all be different colors.
[0190] In various embodiments, the first color and the second color may be the same and the first color and the third color may be different.
[0191] According to various embodiments, if the touch location is included in the button area of the first button, at least one processor (130) may not display the guide UI (30). If the user touches the button area, a separate guide UI (30) may not be required.
[0192] A specific description related to the preset areas is described in Fig. 15. A plurality of preset areas corresponding to one button is described in Fig. 15. A plurality of preset areas corresponding to each of the plurality of buttons are described in Fig. 16. A description of displaying the guide UI (30) in different formats depending on which area the touch location is included in is described in Fig. 23.
[0193] According to various embodiments, an external device (200) may determine whether a touch location has selected a specific button. In the above description, the operation of comparing the touch location with a preset area is described as being performed by the electronic device (100). The subject of the described operation may be changed from the electronic device (100) to the external device (200).
[0194] At least one processor (130) may transmit a signal for controlling the output position of the second control UI to the external device (200) through a communication interface (110) based on an input pattern acquired based on the information corresponding to the plurality of user inputs when information corresponding to multiple user inputs is received within a preset time through the second control UI output to the external device (200).
[0195] At least one processor (130) can identify an input pattern (or touch pattern) based on the plurality of touch information when a plurality of touch information is received within a threshold time through the second control UI (20) output to the external device (200), generate a control signal for changing the output position of the second control UI (20) based on the input pattern, and transmit the control signal to the external device (200) through the communication interface (110).
[0196] At least one processor (130) can change the output location of the second control UI (20) based on an input pattern. At least one processor (130) can receive consecutive user inputs received within a threshold time.
[0197] At least one processor (130) can acquire a plurality of touch information related to consecutive user inputs received within a threshold time. At least one processor (130) can determine whether a preset input pattern is identified based on the plurality of touch information.
[0198] A preset input pattern may be a pattern in which a touch location that selects a specific button moves in a specific direction. A preset input pattern may be a pattern in which the location of the user input moves in the following order: button area, auxiliary button area, and expandable area.
[0199] For example, the consecutive user inputs may be a first user input, a second user input, and a third user input. When an event is identified in which the first user input is identified in the button area, the second user input is identified in the auxiliary button area, and the third user input is identified in the expandable area, the electronic device (100) may determine that a preset input pattern has been identified.
[0200] When a preset input pattern is identified, at least one processor (130) can determine an output location of the second control UI (20) based on a touch location of the most recent user input. At least one processor (130) can generate a control signal for displaying the second control UI (20) at the determined output location. At least one processor (130) can transmit the generated control signal to an external device (200). The external device (200) can receive the control signal from the electronic device (100) and display the second control UI (20) at the determined output location.
[0201] An electronic device (100) according to various embodiments can guide an accurate touch location by displaying a guide UI (30). A user can intuitively recognize that an operation through an external device (200) is inaccurate through the guide UI (30) or the color of the guide UI (30).
[0202] In the above description, various operations are described as being performed through the electronic device (100). However, some of the operations described above may be performed in the electronic device (100), and other operations may be performed in an external server.
[0203] According to various embodiments, the electronic device (100) is a device that outputs a first control UI (10) and a guide UI (30), and a specific judgment operation can be performed in an external server.
[0204] For example, referring to the operation of FIG. 19, steps S1905, S1910, S1920, S1925, S1930, S1940, S1945, S1955, and S1970 may be performed in the electronic device (100). Steps S1915, S1935, S1950, and S1965 may be performed in an external server. The electronic device (100) and the external server may be connected to each other for communication and may exchange information about each operation.
[0205] According to various embodiments, the electronic device (100) can control content based on user voice acquired through a microphone.
[0206] For example, a microphone may be included in the electronic device (100).
[0207] For example, a microphone may be included in an external device (200). When a user's voice is acquired through the microphone, the external device (200) may transmit the user's voice to the electronic device (100). Even if a game user does not use a microphone connected to the electronic device (100), the game user can easily input voice using the microphone of the external device (200).
[0208] The electronic device (100) can analyze the acquired user voice to obtain information (e.g., text information) corresponding to the user voice. The electronic device (100) can control content based on the information corresponding to the user voice. The electronic device (100) can control content based on an audio signal including the user voice.
[0209] For example, the electronic device (100) may change settings related to content provision based on information corresponding to the user's voice. The electronic device (100) may change at least one of the brightness, resolution, and sound volume of the content based on the user's voice.
[0210] For example, the electronic device (100) can change an object included in the content based on information corresponding to the user's voice. The electronic device (100) can change the location or appearance of an object, UI, etc. included in the content based on the user's voice.
[0211] For example, the electronic device (100) can perform an input operation for a chat function provided in content based on text corresponding to a user's voice. When a user speaks "Hello, friend" on a screen, the electronic device (100) can input text information corresponding to "Hello, friend" into a chat UI. The electronic device (100) can transmit the text information corresponding to "Hello, friend" to a content server (or external server) that provides a content chat function. The content server can receive the received text information, generate a chat screen, and transmit content including the generated chat screen back to the electronic device (100). The electronic device (100) can receive and display content including the generated chat screen.
[0212] FIG. 4 is a block diagram for explaining the specific configuration of the electronic device (100) of FIG. 3.
[0213] Referring to FIG. 4, the electronic device (100) may include at least one of a communication interface (110), a display (120), at least one processor (130), at least one memory (140), an operation interface (150), an input / output interface (160), a speaker (170), a microphone (180), and a camera (190).
[0214] The communication interface (110) is a configuration that performs communication with various types of external devices according to various types of communication methods. The communication interface (110) may include a wireless communication module or a wired communication module. Each communication module may be implemented in the form of at least one hardware chip.
[0215] A wireless communication module may be a module that communicates wirelessly with an external device. For example, the wireless communication module may include at least one of a Wi-Fi module, a Bluetooth module, an infrared communication module, or other communication modules.
[0216] Wi-Fi and Bluetooth modules can communicate via Wi-Fi and Bluetooth, respectively. When using a Wi-Fi or Bluetooth module, various connection information, such as the service set identifier (SSID) and session key, is first transmitted and received. This information is then used to establish a communication connection before various other information can be transmitted and received.
[0217] Infrared communication modules perform communication based on infrared communication (IrDA, infrared Data Association) technology, which transmits data wirelessly over short distances using infrared light, which is between visible light and millimeter waves.
[0218] In addition to the above-described communication method, other communication modules may include at least one communication chip that performs communication according to various wireless communication standards such as zigbee, 3G (3rd Generation), 3GPP (3rd Generation Partnership Project), LTE (Long Term Evolution), LTE-A (LTE Advanced), 4G (4th Generation), 5G (5th Generation), etc.
[0219] A wired communication module may be a module that communicates with an external device via a wire. For example, the wired communication module may include at least one of a Local Area Network (LAN) module, an Ethernet module, a paired cable, a coaxial cable, a fiber optic cable, or an Ultra Wide-Band (UWB) module.
[0220] According to various embodiments, the communication interface (110) may utilize the same communication module (e.g., a Wi-Fi module) to communicate with an external device such as a remote control device and an external server.
[0221] According to various embodiments, the communication interface (110) may utilize different communication modules to communicate with external devices, such as remote control devices, and external servers. For example, the communication interface (110) may utilize at least one of an Ethernet module or a Wi-Fi module to communicate with an external server, and may also utilize a Bluetooth module to communicate with an external device, such as a remote control device. However, this is merely an embodiment, and the communication interface (110) may utilize at least one of various communication modules when communicating with multiple external devices or external servers.
[0222] The display (120) may be implemented as a variety of displays such as a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display panel (PDP), etc. The display (120) may also include a driving circuit, a backlight unit, etc., which may be implemented as a form such as an a-si TFT (amorphous silicon thin film transistor), an LTPS (low temperature poly silicon) TFT, an OTFT (organic TFT), etc. The display (120) may be implemented as a touch screen combined with a touch sensor, a flexible display, a three-dimensional display (3D display, three-dimensional dispaly), etc. According to an embodiment of the present disclosure, the display (120) may include not only a display panel that outputs an image, but also a bezel that houses the display panel. In particular, according to an embodiment of the present disclosure, the bezel may include a touch sensor (not shown) for detecting user interaction.
[0223] At least one processor (130) may be implemented as a digital signal processor (DSP), a microprocessor, or a time controller (TCON) that processes digital signals. However, the present invention is not limited thereto, and may include one or more of a central processing unit (CPU), a micro controller unit (MCU), a micro processing unit (MPU), a controller, an application processor (AP), a graphics-processing unit (GPU), a communication processor (CP), or an advanced reduced instruction set computer (RISC) machines (ARM) processor, or may be defined by the relevant terminology. The processor (130) may be implemented as a system on chip (SoC) having a built-in processing algorithm, a large scale integration (LSI), or may be implemented in the form of a field programmable gate array (FPGA). At least one processor (130) may perform various functions by executing computer executable instructions stored in a memory.
[0224] The memory (140) may be implemented as an internal memory such as a ROM (e.g., an electrically erasable programmable read-only memory (EEPROM)) or RAM included in the processor (130), or may be implemented as a separate memory from the processor (130). In this case, the memory (140) may be implemented as a memory embedded in the electronic device (100) or as a memory that can be detachably attached to the electronic device (100) depending on the purpose of data storage. For example, data for driving the electronic device (100) may be stored in a memory embedded in the electronic device (100), and data for expanding the functions of the electronic device (100) may be stored in a memory that can be detachably attached to the electronic device (100).
[0225] In the case of memory embedded in the electronic device (100), it may be implemented as at least one of volatile memory (e.g., dynamic RAM (DRAM), static RAM (SRAM), or synchronous dynamic RAM (SDRAM)), non-volatile memory (e.g., one time programmable ROM (OTPROM), programmable ROM (PROM), erasable and programmable ROM (EPROM), electrically erasable and programmable ROM (EEPROM), mask ROM, flash ROM, flash memory (e.g., NAND flash or NOR flash), etc.), hard drive, or solid state drive (SSD), and in the case of memory that can be attached or detached to the electronic device (100), it may be implemented in the form of a memory card (e.g., compact flash (CF), secure digital (SD), micro secure digital (Micro-SD), mini secure digital (Mini-SD), extreme digital (xD), multi-media card (MMC), etc.), external memory that can be connected to a USB port (e.g., USB memory), etc.
[0226] The memory (140) can store at least one instruction. Based on the instruction stored in the memory (140), the processor (130) can perform various operations.
[0227] The operating interface (150) may be implemented as a device such as a button, a touch pad, a mouse, and a keyboard, or as a touch screen capable of performing the above-described display function and operating input function. The button may be a mechanical button, a touch pad, a wheel, or any other type of button formed in any area of the front, side, or back of the main body of the electronic device (100).
[0228] The input / output interface (160) may be any one of HDMI (High Definition Multimedia Interface), MHL (Mobile High-Definition Link), USB (Universal Serial Bus), DP (Display Port), Thunderbolt, VGA (Video Graphics Array) port, RGB port, D-SUB (D-subminiature), and DVI (Digital Visual Interface). The input / output interface (160) may input / output at least one of audio and video signals. Depending on the implementation example, the input / output interface (160) may include a port that inputs / outputs only audio signals and a port that inputs / outputs only video signals as separate ports, or may be implemented as a single port that inputs / outputs both audio signals and video signals. The electronic device (100) may transmit at least one of the audio and video signals to an external device (e.g., an external display device or an external speaker) through the input / output interface (160). Specifically, an output port included in the input / output interface (160) can be connected to an external device, and the electronic device (100) can transmit at least one of an audio and video signal to the external device through the output port.
[0229] The input / output interface (160) can be connected to a communication interface. The input / output interface (160) can transmit information received from an external device to the communication interface or transmit information received through the communication interface to the external device.
[0230] The speaker (170) may be a component that outputs various audio data as well as various notification sounds or voice messages.
[0231] The microphone (180) is a component that receives a user's voice or other sounds and converts them into audio data. The microphone (180) can receive the user's voice in an activated state. For example, the microphone (180) can be formed integrally on the upper side, the front side, the side side, etc. of the electronic device (100). The microphone (180) can include various components such as a microphone that collects the user's voice in analog form, an amplifier circuit that amplifies the collected user's voice, an A / D conversion circuit that samples the amplified user's voice and converts it into a digital signal, and a filter circuit that removes noise components from the converted digital signal.
[0232] The camera (190) is a device configured to capture a subject and generate a captured image, wherein the captured image includes both moving images and still images. The camera (190) can acquire images for at least one external device and can be implemented with a camera, a lens, an infrared sensor, or the like.
[0233] The camera (190) may include a lens and an image sensor. The type of lens may include a general-purpose lens, a wide-angle lens, a zoom lens, etc., and may be determined according to the type, characteristics, usage environment, etc. of the electronic device (100). The image sensor may include a complementary metal oxide semiconductor (CMOS) and a charge-coupled device (CCD).
[0234] According to various embodiments, the electronic device (100) may include a display (120). Specifically, the electronic device (100) may directly display an acquired image or content on the display (120).
[0235] According to various embodiments, the electronic device (100) may not include a display (120). The electronic device (100) may be connected to an external display device and may transmit images or content stored in the electronic device (100) to the external display device.
[0236] The electronic device (100) can transmit an image or content to an external display device along with a control signal for controlling the display of the image or content on the external display device. The external display device can be connected to the electronic device (100) via a communication interface (110) or an input / output interface (160). For example, the electronic device (100) may not include a display, such as a set-top box (STB).
[0237] The electronic device (100) may include only a small display capable of displaying simple information such as text information. The electronic device (100) may transmit images or content to an external display device via a communication interface (110) via a wired or wireless connection, or may transmit the images or content to the external display device via an input / output interface (160).
[0238] There may be various embodiments in which the electronic device (100) performs an action corresponding to a user voice signal received through a microphone (180).
[0239] According to various embodiments, the electronic device (100) may control the display (120) based on a user voice signal received through the microphone (180). For example, when a user voice signal for displaying content A is received, the electronic device (100) may control the display (120) to display content A.
[0240] According to various embodiments, the electronic device (100) may control an external display device connected to the electronic device (100) based on a user voice signal received through the microphone (180). Specifically, the electronic device (100) may generate a control signal for controlling the external display device so that an operation corresponding to the user voice signal is performed on the external display device, and transmit the generated control signal to the external display device. The electronic device (100) may store a remote control application for controlling the external display device. In addition, the electronic device (100) may transmit the generated control signal to the external display device using at least one communication method among Bluetooth, Wi-Fi, and infrared. For example, when a user voice signal for displaying content A is received, the electronic device (100) may transmit a control signal for controlling the display of content A on the external display device to the external display device. The electronic device (100) may refer to various terminal devices capable of installing a remote control application, such as a smartphone or an AI speaker.
[0241] According to various embodiments, the electronic device (100) may use a remote control device to control an external display device connected to the electronic device (100) based on a user voice signal received through the microphone (180). Specifically, the electronic device (100) may transmit a control signal to the remote control device for controlling the external display device so that an operation corresponding to the user voice signal is performed on the external display device. In addition, the remote control device may transmit the control signal received from the electronic device (100) to the external display device. For example, when a user voice signal for displaying content A is received, the electronic device (100) may transmit a control signal to the remote control device for controlling the display of content A on the external display device, and the remote control device may transmit the received control signal to the external display device.
[0242] The electronic device (100) can receive a user voice signal in various ways.
[0243] According to various embodiments, the electronic device (100) can receive a user voice signal through a microphone (180) included in the electronic device (100).
[0244] According to various embodiments, the electronic device (100) may receive a user voice signal from an external device including a microphone. The external device may refer to a remote control device or a smartphone, etc. The received user voice signal may be a digital voice signal, but may also be an analog voice signal depending on the implementation example. The electronic device (100) may receive the user voice signal via a wireless communication method such as Bluetooth or Wi-Fi.
[0245] The electronic device (100) can convert the user voice signal in various ways.
[0246] According to various embodiments, the electronic device (100) may obtain text information corresponding to a user voice signal from an external server. Specifically, the electronic device (100) may transmit the user voice signal (audio signal or digital signal) to the external server. The external server may refer to a voice recognition server. The voice recognition server may convert the user voice signal into text information using STT (Speech To Text). Then, the external server may transmit text information corresponding to the converted user voice signal to the electronic device (100).
[0247] According to various embodiments, the electronic device (100) can independently acquire text information corresponding to a user voice signal. Specifically, the electronic device (100) can directly apply a STT (Speech To Text) function to a digital voice signal to convert it into text information and transmit the converted text information to an external server.
[0248] An external server can transmit information to the electronic device (100) in a variety of ways.
[0249] According to various embodiments, an external server may transmit text information corresponding to a user voice signal to an electronic device (100). Specifically, the external server may be a server that performs a voice recognition function that converts a user voice signal into text information.
[0250] According to various embodiments, the external server may transmit at least one of text information corresponding to the user voice signal or search result information corresponding to the text information to the electronic device (100). Specifically, the external server may be a server that performs a search result providing function that provides search result information corresponding to the text information in addition to a voice recognition function that converts the user voice signal into text information. As an example, the external server may be a server that performs both a voice recognition function and a search result providing function. As another example, the external server may perform only a voice recognition function and the search result providing function may be performed by a separate server. The external server may transmit text information to a separate server to obtain a search result and obtain a search result corresponding to the text information from the separate server.
[0251] The electronic device (100) can communicate with external devices and external servers in various ways.
[0252] According to various embodiments, communication modules for communication with external devices and external servers may be implemented identically. For example, the electronic device (100) may communicate with an external device using a Bluetooth module, and the external server may also communicate using a Bluetooth module.
[0253] According to various embodiments, communication modules for communication with external devices and external servers may be implemented separately. For example, the electronic device (100) may communicate with external devices using a Bluetooth module and with external servers using an Ethernet modem or Wi-Fi module.
[0254] FIG. 5 is a drawing for explaining an operation of displaying a connection guide screen of an electronic device (100) and an external device (200).
[0255] Referring to FIG. 5, the electronic device (100) may display a connection guide screen with an external device in relation to the displayed content. When the content to be displayed on the electronic device (100) is specified (or selected), a device for manipulating the specified content may be required. If a dedicated controller (hereinafter, the controller (300)) exists, the controller (300) must be connected to the electronic device (100). If the controller (300) exists but is not connected to the electronic device (100), the electronic device (100) may display a control UI using the external device (200). The user may manipulate the content displayed on the electronic device (100) using the external device (200) instead of the controller (300).
[0256] The electronic device (100) may display a connection guide screen (or guide screen) for connecting to a controller (300) or an external device (200). The connection guide screen may include at least one of a UI (510) indicating an attempt to connect to a dedicated controller or an external device, or a UI (520) indicating a list of connectable devices.
[0257] The UI (510) may include text. Depending on the implementation example, the UI (510) may include an image, icon, etc. indicating an attempt to connect with a controller or external device, etc.
[0258] The UI (520) may include a list of external devices that can be connected to the current electronic device (100) and a button for receiving a command to initiate a connection with the external device.
[0259] The external device list may include both an external device (200) and a controller (300). If the list of external devices connectable to the electronic device (100) includes both an external device (200) and a controller (300), the electronic device (100) may display the controller (300) with priority. For example, the method of displaying with priority may mean displaying the device with priority at the top of the external device list or highlighting the device with priority in a preset color.
[0260] Figure 6 is a drawing for explaining an operation of displaying a guide UI according to user input.
[0261] Referring to FIG. 6, the electronic device (100) can display a first control UI (10) and the external device (200) can display a second control UI (20). While the second control UI (20) is displayed on the external device (200), the external device (200) can obtain (or receive) a user input (610). When the user input (610) is received, the external device (200) can transmit information (e.g., location information) related to the user input (610) to the electronic device (100).
[0262] The electronic device (100) can receive information related to a user input (610) and provide a guide UI (30) corresponding to the user input (610). The guide UI (30) can indicate where the user input (610) was received on the external device (200). The user can easily recognize the touch location through the guide UI (30).
[0263] The guide UI (30) can change at least one of shape, size, color, and transparency. The appearance information of the guide UI (30) can be changed based on the user's settings or content.
[0264] Figure 7 is a drawing for explaining an operation of displaying a guide UI according to multiple user inputs.
[0265] Referring to FIG. 7, the electronic device (100) can display a first control UI (10) and the external device (200) can display a second control UI (20). While the second control UI (20) is displayed on the external device (200), the external device (200) can obtain (or receive) a user input (711, 712). When the user input (711, 712) is received, the external device (200) can transmit information (e.g., location information) related to the user input (711, 712) to the electronic device (100).
[0266] The electronic device (100) can receive information related to a user input (711, 712) and display a guide UI (31, 32) corresponding to the user input (711, 712). The guide UI (31, 32) can indicate where the user input (711, 712) was received on the external device (200). The user can easily recognize the touch location through the guide UI (31, 32).
[0267] In FIG. 6 and FIG. 7, a configuration for displaying the first control UI (10) is described.
[0268] According to various embodiments, the electronic device (100) may provide the first control UI (10) only when a preset event occurs in relation to the second control UI (20). The preset event may be an event in which the user's touch is determined to be an incorrect touch.
[0269] A preset event may refer to an event in which a user does not touch a preset button area. The preset event may be an event in which a user selects at least one area among the auxiliary button area (1512), the expandable area (1513), or the remaining area (1514) in the second control UI (20).
[0270] For example, if the user's touch is identified in one of the auxiliary button area (1512), the expandable area (1513), or the remaining area (1514) in the second control UI (20), the electronic device (100) may display the first control UI (10).
[0271] For example, if the user's touch is identified in the auxiliary button area (1512) or the expandable area (1513), the electronic device (100) may display the first control UI (10).
[0272] For example, when a user's touch is identified in the auxiliary button area (1512), the electronic device (100) may display the first control UI (10).
[0273] For example, when a user's touch is identified in the expandable area (1513), the electronic device (100) may display the first control UI (10).
[0274] The first control UI (10) may be displayed overlapping the content screen provided by the electronic device (100). The display of the first control UI (10) may interfere with the user's viewing of the content. The electronic device (100) may display the first control UI (10) only in situations where the user's touch is determined to be inaccurate. When the first control UI (10) is displayed, the user can easily recognize that the touch input must be accurate.
[0275] FIG. 8 is a drawing for explaining an operation of displaying a control UI according to various embodiments.
[0276] Referring to FIG. 8, the electronic device (100) can display the first control UI (10) in various ways. The embodiments disclosed in FIG. 8 should not be construed as limiting, and the first control UI (10) can be displayed on the electronic device (100) in various ways. The display location of the first control UI (10) can be determined based on at least one of user input and content.
[0277] According to an embodiment (810), the electronic device (100) can display the first control UI (10) in a specific area of the display (120). The specific area may be a preset area and may be determined according to predetermined criteria. According to an embodiment (810), the electronic device (100) can display the first control UI (10) while maintaining the relative positions of buttons (elements) included in the first UI information.
[0278] According to an embodiment (820), the electronic device (100) can display the first control UI (10) by changing the relative positions of buttons (elements) included in the first control UI (10). According to an embodiment (820), unlike in the embodiment (810), the electronic device (100) can display the first control UI (10) by narrowing the gap between the moving element and the function element (operation element).
[0279] According to an embodiment (830), the electronic device (100) can display the first control UI (10) by changing the relative positions of buttons (elements) included in the first control UI (10). According to an embodiment (830), unlike in the embodiment (810), the electronic device (100) can display the first control UI (10) by widening the gap between the moving element and the function element.
[0280] The spacing between the movable element and the functional element may vary based on the content displayed on the electronic device (100). The first control UI (10) should not interfere with the user viewing the content. Accordingly, the electronic device (100) may adjust the spacing between the movable element and the functional element by considering the screen configuration (or object) displayed in the content.
[0281] The electronic device (100) can identify the gap between the movable element and the functional element set based on the first UI information based on the position (or arrangement) of the element. The electronic device (100) can obtain third UI information by changing the position (or arrangement) of the element. Then, the electronic device (100) can display the first control UI (10) with the gap between the movable element and the functional element widened based on the changed third UI information.
[0282] FIG. 9 is a drawing for explaining an operation of displaying a control UI according to the relative positions of an electronic device (100) and an external device (200).
[0283] Referring to FIG. 9, the electronic device (100) can communicate with an external device (200). When the electronic device (100) is connected to the external device (200), the electronic device (100) can request the location of the external device (200). The external device (200) can transmit the location of the external device (200) to the electronic device (100) in response to the request of the electronic device (100).
[0284] The electronic device (100) can obtain a relative position between the electronic device (100) and the external device (200) by considering the position of the external device (200). The electronic device (100) can display the first control UI (10) based on the relative position of the external device (200). The electronic device (100) can determine the output position of the first control UI (10) based on the relative position of the external device (200). The electronic device (100) can display the first control UI (10) at the determined position.
[0285] Referring to embodiment (910), if the external device (200) is located on the left side (or first side) with respect to the electronic device (100), the electronic device (100) can display the first control UI (10) on the left side (or first side) of the display (120) of the electronic device (100) based on the relative position of the external device (200).
[0286] Referring to embodiment (920), if the external device (200) is located on the right side (or second side) with respect to the electronic device (100), the electronic device (100) can display the first control UI (10) on the right side (or second side) of the display (120) of the electronic device (100) based on the relative position of the external device (200).
[0287] Figure 10 is a drawing for explaining an operation of adjusting the number of multiple elements included in a control UI.
[0288] Referring to FIG. 10, the electronic device (100) can change the number of elements included in the first control UI (10). The elements included in the second control UI (20) and the elements included in the first control UI (10) may be different. The electronic device (100) can exclude (or delete) specific elements when acquiring third UI information.
[0289] The first UI information of the controller may include multiple elements. When receiving user input via an external device (200), the number of elements included in the first UI information and the number of elements included in the second control UI (20) must be the same. Since the first control UI (10) is output together with content from the electronic device (100), if all of the multiple elements are displayed, the user may experience inconvenience when viewing the content.
[0290] The electronic device (100) can determine a primary element (or essential element) from among a plurality of elements in the first UI information of the controller. The electronic device (100) can obtain third UI information including information related to the primary element. The electronic device (100) can display the first control UI (10) based on the third UI information.
[0291] Example 10 (1010) of FIG. 10 shows that the number of elements included in the second control UI (20) and the number of elements included in the first control UI (10) are the same.
[0292] An embodiment (1020) of FIG. 10 illustrates that the number of elements included in the second control UI (20) is different from the number of elements included in the first control UI (10). The electronic device (100) may determine a main element among a plurality of elements included in the first UI information and display the first control UI (10) including the determined main element. According to the embodiment (1020), the electronic device (100) may display the first control UI (10) including a moving element and some functional elements.
[0293] The electronic device (100) can determine at least one element to be included in the first control UI (10) based on the communication strength (or connection strength) between the electronic device (100) and the external device (200). The electronic device (100) can include a greater number of elements in the first control UI (10) as the communication strength becomes stronger.
[0294] The electronic device (100) can obtain the communication strength between the electronic device (100) and the external device (200). The electronic device (100) can determine at least one element to be included in the first control UI (10) based on the communication strength.
[0295] If the communication strength is greater than or equal to the threshold strength, the electronic device (100) can display a first control UI (10) including a plurality of elements included in the first UI information, as in the embodiment (1010).
[0296] If the communication strength is less than the threshold strength, the electronic device (100) can display a first control UI (10) including a main element among a plurality of elements included in the first UI information, as in the embodiment (1020).
[0297] The number of key elements may be a preset number. The number of key elements may be changed according to the user's settings. For example, if the external device (200) has 12 default buttons (elements), the electronic device (100) may adjust the number of buttons (elements) displayed on the UI based on the communication strength. The communication strength may be expressed as a connection speed.
[0298] According to one embodiment, if the communication strength is greater than or equal to a first threshold, the electronic device (100) may display all of the plurality of elements included in the first UI information as the first control UI (10).
[0299] According to one embodiment, when the communication strength is less than the first threshold and greater than the second threshold, the electronic device (100) may display a first number (e.g., 12) of main elements among all of the plurality of elements included in the first UI information as the first control UI (10).
[0300] According to one embodiment, if the communication strength is less than the second threshold and greater than the third threshold, the electronic device (100) may display a preset second number (e.g., 10) of primary elements among a plurality of elements included in the first UI information as the first control UI (10). The second number may be smaller than the first number.
[0301] According to one embodiment, if the communication strength is less than the third threshold and greater than the fourth threshold, the electronic device (100) may display a preset third number (e.g., eight) of main elements among all of the plurality of elements included in the first UI information as the first control UI (10). The third number may be smaller than the second number.
[0302] According to one embodiment, if the communication strength is less than the fourth threshold, the electronic device (100) may display a preset fourth number (e.g., five) of primary elements among all of the plurality of elements included in the first UI information as the first control UI (10). The fourth number may be less than the third number.
[0303] The electronic device (100) can adjust the number of elements provided based on the communication strength (or connection speed). As the number of elements provided decreases, user input is restricted, thus reducing the amount of data transmitted or received. With a reduced amount of data transmitted or received, the electronic device (100) can provide the first control UI (10) to the user without interruption, even when the communication strength is relatively weak. Fig. 11 is a diagram illustrating adjusting the transparency of the control UI.
[0304] Referring to FIG. 11, the electronic device (100) can determine the transparency of the first control UI (10). The electronic device (100) can determine the transparency of the first control UI (10) based on third UI information. The electronic device (100) can display the first control UI (10) together with the content screen. The electronic device (100) needs to not interfere with the user's viewing of content. The electronic device (100) can adjust the transparency of the first control UI (10) so as not to interfere with the user's viewing of content.
[0305] The electronic device (100) can display the first control UI (10) based on a preset transparency. The preset transparency can be changed according to the user's settings.
[0306] The electronic device (100) can determine transparency based on content. A transparency corresponding to each of the plurality of contents can be determined in advance. The transparency can be identified based on first UI information corresponding to each content.
[0307] The electronic device (100) can determine transparency based on color information of the content. The electronic device (100) can determine transparency based on color information of the screen provided by the content. When a content screen with a relatively dark color is displayed on the electronic device (100), the electronic device (100) can increase transparency. When a content screen with a relatively bright color is displayed on the electronic device (100), the electronic device (100) can decrease transparency.
[0308] According to the embodiment (1110) of FIG. 11, the electronic device (100) can display the first control UI (10) based on transparency (0%).
[0309] According to the embodiment (1120) of FIG. 11, the electronic device (100) can display the first control UI (10) based on transparency (30%).
[0310] According to the embodiment (1130) of FIG. 11, the electronic device (100) can display the first control UI (10) based on transparency (60%).
[0311] FIG. 12 is a drawing for explaining a control UI corresponding to content according to various embodiments.
[0312] Referring to FIG. 12, the external device (200) can display a second control UI (20) corresponding to the content. Different control UIs may exist for each content. If the content is created by different content providers, the control UIs of the controllers may differ. Even if the content is provided by the same provider, the control UIs of the controllers may differ if the content provided differs.
[0313] The electronic device (100) can obtain first UI information corresponding to specific content. The electronic device (100) can obtain second UI information by changing the first UI information based on size information of the external device (200). The electronic device (100) can display a second UI based on the second UI information. Since the first UI information corresponding to each content is different, the second control UI (20) displayed may be different depending on the content.
[0314] According to the embodiment (1210) of FIG. 12, the external device (200) can display a second control UI (20) corresponding to content A.
[0315] According to the embodiment (1220) of FIG. 12, the external device (200) can display a second control UI (20) corresponding to the B content.
[0316] According to the embodiment (1230) of FIG. 12, the external device (200) can display a second control UI (20) corresponding to the C content.
[0317] According to the embodiment (1240) of FIG. 12, the external device (200) can display a second control UI (20) corresponding to the D content.
[0318] At least one of the shape, position, area, size, color, and transparency of the element included in the second control UI (20) corresponding to each content may be different.
[0319] FIG. 13 is a drawing for explaining a control UI corresponding to a controller according to various embodiments.
[0320] Referring to Fig. 13, an external device (200) can display a second control UI (20) corresponding to the controller. Fig. 12 illustrates that the control UI varies depending on the content. In other implementation examples, different control UIs may exist depending on the controller. If the controllers are different, the control UIs may also vary.
[0321] The electronic device (100) can obtain first UI information corresponding to a specific controller. The electronic device (100) can obtain second UI information by changing the first UI information based on size information of the external device (200). The electronic device (100) can display a second UI based on the second UI information. Since the first UI information corresponding to each content is different, the second control UI (20) displayed may be different depending on the content.
[0322] According to the embodiment (1310) of FIG. 13, the external device (200) can display a second control UI (20) corresponding to the controller (301).
[0323] According to the embodiment (1320) of FIG. 13, the external device (200) can display a second control UI (20) corresponding to the controller (302).
[0324] According to the embodiment (1330) of FIG. 13, the external device (200) can display a second control UI (20) corresponding to the controller (303).
[0325] According to the embodiment (1340) of FIG. 13, the external device (200) can display a second control UI (20) corresponding to the controller (304).
[0326] FIG. 14 is a drawing for explaining a communication method between an electronic device (100) and an external device (200).
[0327] Referring to FIG. 14, an electronic device (100) and an external device (200) can be connected according to various communication methods.
[0328] The electronic device (100) and the external device (200) can be connected through an AP (Access Point) device (400). The AP device (400) may mean a router.
[0329] The electronic device (100) may include at least one of a Wi-Fi module (111), a web server module (112), a Bluetooth module (113), and a wired communication module (114). The Wi-Fi module (111), the web server module (112), the Bluetooth module (113), the wired communication module (114), etc. may be included in the communication interface (110) of the electronic device (100).
[0330] The external device (200) may include a WiFi module (211), a web client module (212), a Bluetooth module (213), and other communication modules (214).
[0331] According to various embodiments, the electronic device (100) and the external device (200) can be connected to the AP device (400) using a Wi-Fi module (111, 211).
[0332] According to various embodiments, the electronic device (100) and the external device (200) can perform Wi-Fi direct communication using the Wi-Fi module (111, 211). The electronic device (100) and the external device (200) can create a D2D (Device to Device) channel.
[0333] The web server module (112) may be a module that performs and manages a function of providing information requested by an external device (200) (or information to be transmitted to the external device (200)) statically or operatively. The web server module (112) may be described as a web server application. The web server module (112) may include component files (e.g., HTML files, JavaScript files, image files, etc.). The web server module (112) may communicate with the external device (200) based on at least one of a Wi-Fi module (111), a Bluetooth module (113), and a wired communication module (114).
[0334] The web client module (212) may be a module that performs and manages a function of requesting specific information from the electronic device (100) (or receiving specific information from the electronic device (100)). The web client module (212) may be described as a web client application. The web client module (212) may communicate with the electronic device (100) based on at least one of a Wi-Fi module (211), a Bluetooth module (213), and other communication modules (214).
[0335] The web server module (112) may be implemented as at least one of hardware or software. The web client module (212) may be implemented as at least one of hardware or software.
[0336] The Bluetooth module (113, 213) can connect communication between an electronic device (100) and an external device (200) using Bluetooth.
[0337] According to various embodiments, the electronic device (100) and the external device (200) can perform Bluetooth communication using the Bluetooth module (113, 213). The electronic device (100) and the external device (200) can create a D2D (Device to Device) channel.
[0338] The wired communication module (114) may be a module that communicates with an external device via a wire. For example, the wired communication module may include at least one of a LAN (Local Area Network) module, an Ethernet module, a pair cable, a coaxial cable, a fiber optic cable, or a UWB (Ultra Wide-Band) module.
[0339] The other communication module (214) may include at least one communication chip that performs communication according to various wireless communication standards such as zigbee, 3G (3rd Generation), 3GPP (3rd Generation Partnership Project), LTE (Long Term Evolution), LTE-A (LTE Advanced), 4G (4th Generation), 5G (5th Generation), etc. The other communication module (214) may also be applied to various wireless communication standards developed after 5G.
[0340] The electronic device (100) can communicate with the server (500) via a wired communication module (114). The external device (200) can communicate with the server (500) via another communication module (214). The electronic device (100) and the external device (200) can be interconnected using the server (500).
[0341] The communication speed using the D2D channel may be faster than the communication speed using the AP device (400). When a preset event is identified as occurring, the electronic device (100) may change the first connection method using the AP device (400) to the second connection method using the D2D channel.
[0342] When a preset event is identified while the electronic device (100) is connected to an external device (200) using a specific communication method (communication using an AP device (400)), the electronic device (100) may attempt at least one of Wi-Fi direct communication, Bluetooth communication, or communication using a server (500).
[0343] A preset event may be an event in which the communication strength between the electronic device (100) and the external device (200) is below a threshold strength. The communication strength may refer to the signal strength for communication using the AP device (400) between the electronic device (100) and the external device (200).
[0344] According to various embodiments, when a preset event is identified, the electronic device (100) may first attempt Wi-Fi Direct communication. If Wi-Fi Direct communication is determined to be impossible, the electronic device (100) may attempt Bluetooth communication. If Bluetooth communication is determined to be impossible, the electronic device (100) may attempt communication using the server (500). If communication using the server (500) is impossible, the electronic device (100) may connect to the external device (200) using a conventional communication method (communication using the AP device (400)).
[0345] According to various embodiments, when a preset event is identified, the electronic device (100) may first attempt Wi-Fi Direct communication. If Wi-Fi Direct communication is determined to be impossible, the electronic device (100) may attempt communication using the server (500). If communication using the server (500) is determined to be impossible, the electronic device (100) may attempt Bluetooth communication. If Bluetooth communication is impossible, the electronic device (100) may connect to the external device (200) using a conventional communication method (communication using the AP device (400).
[0346] According to various embodiments, when a preset event is identified, the electronic device (100) may first attempt Bluetooth communication. If Bluetooth communication is determined to be impossible, the electronic device (100) may attempt Wi-Fi Direct communication. If Wi-Fi Direct communication is determined to be impossible, the electronic device (100) may attempt communication using the server (500). If communication using the server (500) is impossible, the electronic device (100) may connect to the external device (200) using a conventional communication method (communication using the AP device (400)).
[0347] According to various embodiments, when a preset event is identified, the electronic device (100) may first attempt Bluetooth communication. If Bluetooth communication is determined to be impossible, the electronic device (100) may attempt communication using the server (500). If communication using the server (500) is determined to be impossible, the electronic device (100) may attempt Wi-Fi Direct communication. If communication using Wi-Fi Direct communication is impossible, the electronic device (100) may connect to the external device (200) using a conventional communication method (communication using the AP device (400)).
[0348] According to various embodiments, when a preset event is identified, the electronic device (100) may first attempt communication using the server (500). If communication using the server (500) is determined to be impossible, the electronic device (100) may attempt Wi-Fi Direct communication. If Wi-Fi Direct communication is determined to be impossible, the electronic device (100) may attempt Bluetooth communication. If Bluetooth communication is impossible, the electronic device (100) may connect to the external device (200) using a conventional communication method (communication using the AP device (400)).
[0349] According to various embodiments, when a preset event is identified, the electronic device (100) may first attempt communication using the server (500). If communication using the server (500) is determined to be impossible, the electronic device (100) may attempt Bluetooth communication. If Bluetooth communication is determined to be impossible, the electronic device (100) may attempt Wi-Fi Direct communication. If Wi-Fi Direct communication is impossible, the electronic device (100) may connect to the external device (200) using a conventional communication method (communication using the AP device (400)).
[0350] Figure 15 is a drawing for explaining areas related to selection of buttons (elements) in the control UI.
[0351] Referring to FIG. 15, an external device (200) can display a second control UI (20). The user can select (or touch) a specific button (element) through the output second control UI (20). Whether a specific button is selected can be determined based on the area corresponding to the specific button.
[0352] When the second control UI (20) is displayed based on the second UI information, the second UI information may include area information corresponding to the button. The area information may include information related to an area related to button selection.
[0353] According to the embodiment (1510) of FIG. 15, the area information may include a button area (1511), an auxiliary button area (1512), and an expandable area (1513). An area other than the button area (1511), the auxiliary button area (1512), and the expandable area (1513) among the entire area of the display of the external device (200) may be a remaining area (1514).
[0354] The button area (1511) may be an area where a button is identified as being selected.
[0355] The auxiliary button area (1512) may be an area that preliminarily recognizes that a button has been selected based on user intent, even though the button has not been selected by the button area (1511). The auxiliary button area (1512) may be changed.
[0356] The expandable area (1513) may refer to an area in which the auxiliary button area (1512) is expandable. When a user input is received in the expandable area (1513), the electronic device (100) identifies that the user has touched the area around the button, but as a result, the button is not selected.
[0357] The remaining area (1514) may be an area that does not include the button area (1511), the auxiliary button area (1512), and the expandable area (1513) among the entire area of the display. The remaining area (1514) may be described as a margin area, an empty area, etc.
[0358] According to the embodiment (1520) of FIG. 15, the button area (1521), the expandable area (1523), and the remaining area (1524) may correspond to the button area (1511), the expandable area (1513), and the remaining area (1514) of the embodiment (1510). The auxiliary button area (1522) of the embodiment (1520) may be more expanded than the auxiliary button area (1512) of the embodiment (1510).
[0359] According to another implementation example, the auxiliary button area (1512) of embodiment (1510) may be reduced.
[0360] Expansion or reduction of the auxiliary button area (1512) can be performed within the expandable area (1513, 1523). The expandable area (1513, 1523) can be an area used to determine the limit within which the auxiliary button area (1512) can be changed.
[0361] When the auxiliary button area (1512) of embodiment (1510) is expanded to the auxiliary button area (1522) of embodiment (1520), the expandable area (1513) of embodiment (1510) can be reduced (or reduced) like the expandable area (1523) of embodiment (1520).
[0362] The electronic device (100) may display a calibration guide screen to adjust the auxiliary button area (1512). The user may determine the expansion or reduction of the auxiliary button area (1512) through the calibration guide screen. The touch strength (or area) may vary depending on the length (or thickness) of the user's finger. Accordingly, the user may adjust the auxiliary button area (1512) to achieve a touch recognition suitable for the user.
[0363] When a user performs a calibration action, the electronic device (100) can store the user's calibration result and map the stored calibration result to content and an external device (200) and store it.
[0364] For example, when a correction result is obtained, the electronic device (100) can map A content - external device (200) - UI information reflecting the correction result and store the mapping table. The mapping table can be updated each time a user's correction result is obtained.
[0365] According to various embodiments, when an event is identified in which consecutive user inputs for a specific button are received within a threshold time, the electronic device (100) may expand a sub-button area corresponding to the specific button.
[0366] According to various embodiments, the electronic device (100) may perform expansion or reduction of the auxiliary button area (1512) based on the user's voice.
[0367] The electronic device (100) can receive a user's voice through a microphone included in the electronic device (100) or a microphone included in an external device (200). The electronic device (100) can change the size of the auxiliary button area (1512) according to the received user's voice.
[0368] For example, when a user voice command such as “expand auxiliary button area” is received, the electronic device (100) can expand the auxiliary button area (1512) by a preset size.
[0369] For example, when a user voice command such as “reduce auxiliary button area” is received, the electronic device (100) can reduce the auxiliary button area (1512) by a preset size.
[0370] The preset size can be changed according to the user's settings. The preset size may refer to a unit size that changes step by step among the changeable sizes of the auxiliary button area (1512). When the user inputs a command (or user voice) to expand or reduce the size of the auxiliary button area (1512), the electronic device (100) can change the auxiliary button area (1512) by the unit size.
[0371] For example, assume that the maximum changeable diameter is between 10 mm and 20 mm. Assume that the maximum diameter of the current auxiliary button area (1512) is 10 mm and the preset size (diameter) is 2 mm. When a user inputs "Expand auxiliary button area", the electronic device (100) can change the maximum diameter of the auxiliary button area (1512) to 12 mm.
[0372] Figure 16 is a drawing for explaining areas related to each of a plurality of buttons (elements).
[0373] Referring to FIG. 16, if there are multiple buttons in the second UI information, area information corresponding to each of the multiple buttons may be included in the second UI information. For example, the second UI information may include first area information corresponding to the first button and second area information corresponding to the second button.
[0374] According to the embodiment (1600) of Fig. 16, it is assumed that the second UI information includes four buttons. The second UI information may include area information corresponding to each of the four buttons.
[0375] The second UI information may include a first button area (1611) corresponding to the first button, a first auxiliary button area (1612) corresponding to the first button, and a first expandable area (1613) corresponding to the first button.
[0376] The second UI information may include a second button area (1621) corresponding to the second button, a second auxiliary button area (1622) corresponding to the second button, and a second expandable area (1623) corresponding to the second button.
[0377] The second UI information may include a third button area (1631) corresponding to the third button, a third auxiliary button area (1632) corresponding to the third button, and a third expandable area (1633) corresponding to the third button.
[0378] The second UI information may include a fourth button area (1641) corresponding to the fourth button, a 41st auxiliary button area (1642) corresponding to the fourth button, and a fourth expandable area (1643) corresponding to the fourth button.
[0379] An area excluding a plurality of areas (1611, 1612, 1613, 1621, 1622, 1623, 1631, 1632, 1633, 1641, 1642, 1643) included in the second UI information among the entire area of the display of the external device (200) may be a remaining area (1614).
[0380] In the embodiment of FIG. 16, the areas corresponding to each button are depicted as not overlapping with each other. Depending on the implementation, some areas may overlap despite different buttons. When an overlapping area between different buttons is selected by the user, the electronic device (100) may provide a guide UI (30) indicating that the different buttons overlap.
[0381] Figure 17 is a drawing for explaining an operation of controlling a control UI through voice recognition.
[0382] Referring to FIG. 17, the electronic device (100) may include a microphone (180). The electronic device (100) may obtain a user's voice input through the microphone (180). When the voice input is received, the electronic device (100) may perform an operation corresponding to the voice input.
[0383] A voice input may be an input that changes the first control UI (10). The electronic device (100) may adjust the shape, location, area, size, color, and transparency of the first control UI (10) based on the voice input. The area may represent the area information described in FIG. 15.
[0384] Voice input may be an input that changes the guide UI (30). The electronic device (100) may adjust the shape, position, size, color, and transparency of the guide UI (30) based on the voice input.
[0385] According to an implementation example, voice input can be obtained through an external device (200). The external device (200) can obtain the voice input and transmit it to the electronic device (100). The electronic device (100) can perform an operation corresponding to the voice input based on the voice input.
[0386] The analysis operation for voice input can be performed by a voice recognition function. The voice recognition function can be performed in an electronic device (100) or an external device (200).
[0387] Depending on the implementation example, the voice recognition function may be performed on a separate external server. The electronic device (100) or the external device (200) may receive a voice input and transmit it to the external server. The electronic device (100) or the external device (200) may receive result information (e.g., text) corresponding to the voice input analyzed by the external server and perform an operation corresponding to the voice input.
[0388] Figure 18 is a flowchart for explaining the operation of displaying the control UI and guide UI.
[0389] Referring to FIG. 18, the electronic device (100) can select (or determine) content (S1805).
[0390] The electronic device (100) can provide content by executing an application stored in the electronic device (100). The electronic device (100) can provide content selected by the user from among multiple contents. The electronic device (100) can guide the user's selection by displaying a content list.
[0391] The content list may include multiple contents that can be provided by the electronic device (100). A user may select one of the multiple contents included in the content list.
[0392] The content list can prioritize display of primary content among multiple content items that the electronic device (100) can provide. Prioritizing primary content can mean displaying primary content with priority at the top of the content list or highlighting primary content with priority in a preset color.
[0393] According to various embodiments, when a controller (300) is connected to an electronic device (100), the electronic device (100) can determine a content that can be operated by the controller (300) among a plurality of contents included in a content list as the main content.
[0394] According to various embodiments, when an external device (200) is connected to an electronic device (100), the electronic device (100) may determine, among a plurality of contents included in a content list, a content that can be operated by a controller as the main content. The external device (200) may represent a device including a display and may display control UIs of various controllers. Accordingly, the external device (200) may perform the role of various controllers. The plurality of contents may be divided into contents that can be operated by a controller and contents that do not require a controller. When an external device (200) is connected to an electronic device (100), the electronic device (100) may determine, among a plurality of contents included in a content list, a content that requires a controller as the main content.
[0395] According to various embodiments, when an external device (200) is connected to an electronic device (100), the electronic device (100) may determine, among a plurality of contents included in a content list, a content operable with a pre-registered controller (300) as the main content. The electronic device (100) may store a pre-registered controller (300) that has been connected to the electronic device (100) at least once. The electronic device (100) may determine, based on the storage history information, a content operable with a pre-registered controller (300) as the main content.
[0396] The electronic device (100) can display main content with priority in a content list. The user can easily recognize the main content displayed with priority.
[0397] The electronic device (100) can identify content selected by the user. The electronic device (100) can display the selected content (S1810).
[0398] The electronic device (100) can identify a control UI corresponding to the content (S1815). The electronic device (100) can display the identified control UI (S1820).
[0399] The electronic device (100) can identify a control UI corresponding to the selected content. Once the control UI corresponding to the selected content is identified, the electronic device (100) can transmit the identified control UI to an external device (200). The control UI can be transmitted via UI information.
[0400] If a control UI corresponding to the selected content is not identified, the electronic device (100) may transmit a basic control UI to the external device (200). The basic control UI may refer to a UI that includes essential buttons for general content manipulation. Even if the user cannot fully control the content through the basic control UI (or default UI), the user can manipulate essential functions (some functions) through the basic control UI.
[0401] The electronic device (100) can receive location information of a user input from an external device (200) (S1825). The electronic device (100) can provide a guide UI (30) based on the location information of the user input. The electronic device (100) can determine a specific location where the guide UI (30) is to be displayed based on the location information of the user input. The electronic device (100) can provide the guide UI (30) at the determined location.
[0402] FIG. 19 is a flowchart for explaining a second control UI displayed on an electronic device (100) and a third control UI displayed on an external device (200).
[0403] Referring to FIG. 19, the electronic device (100) can select content (S1905). The electronic device (100) can display the content (S1910). The electronic device (100) can identify a controller (300) corresponding to the content (S1915). The controller (300) may refer to a dedicated controller or a virtual controller corresponding to the content.
[0404] The electronic device (100) can identify whether the controller (300) is connected (S1920). If the controller (300) is connected (S1920-Y), the electronic device (100) can communicate with the controller (300) (S1925).
[0405] If the controller (300) is not connected (S1920-N), the electronic device (100) can identify whether it is connected to an external device (200) (S1930). If the external device (200) is not connected (S1930-N), the electronic device (100) can repeat steps S1920 and S1930.
[0406] When an external device (200) is connected (S1930-Y), the electronic device (100) can obtain first UI information of the controller (300) (S1935). The external device (200) can identify a controller (300) corresponding to content and obtain first UI information indicating information related to a button of the identified controller (300).
[0407] A controller mapping table may be used to obtain the first UI information. The mapping table may include information related to each button on the controller.
[0408] For example, it may include first UI information of a first controller, first UI information of a second controller, first UI information of a third controller, and first UI information of a fourth controller. When the first controller is connected to the electronic device (100), the electronic device (100) may obtain first UI information of the first controller based on the mapping table.
[0409] According to various embodiments, the mapping table may be stored in the electronic device (100).
[0410] According to various embodiments, the mapping table may be stored on an external server. The external server may be a server providing content or a server providing services related to the electronic device (100).
[0411] The electronic device (100) can request size information of the external device (200) from the external device (200) (S1940). The external device (200) can transmit size information of the external device (200) to the electronic device (100) in response to the request (S1945). The electronic device (100) can receive size information of the external device (200).
[0412] The electronic device (100) can obtain second UI information based on the first UI information and the size information of the external device (200) (S1950). The electronic device (100) can change the first UI information into second UI information based on the size information of the external device (200). Since the actual size of the controller (300) and the display size of the external device (200) are different, some of the detailed UI information included in the first UI information needs to be changed.
[0413] For example, the electronic device (100) can obtain second UI information by changing the size of a button included in the first UI information based on the size information of the external device (200).
[0414] The electronic device (100) can transmit second UI information to an external device (200) (S1955). The external device (200) can receive second UI information from the electronic device (100). The external device (200) can display a second control UI (20) based on the second UI information (S1960).
[0415] The electronic device (100) can obtain third UI information based on the first UI information and the size information of the electronic device (100) (S1965). The electronic device (100) can change the first UI information into third UI information based on the size information of the electronic device (100). Since the actual size of the controller (300) and the size of the display (120) of the electronic device (100) are different, some of the detailed UI information included in the first UI information needs to be changed.
[0416] The electronic device (100) can display the first control UI (10) based on the third UI information (S1970).
[0417] The size information of the electronic device (100) and the size information of the external device (200) may be described as the size information of the display (120) included in the electronic device (100) and the size information of the display included in the external device (200).
[0418] Depending on various embodiments, steps S1920 and S1925 may be omitted.
[0419] According to various embodiments, the electronic device (100) may display at least one of a screen including content, a screen including a first control UI (10), and a screen including a guide UI (30) by overlapping them. The screens may be described as layers.
[0420] According to various embodiments, a screen including a first control UI (10) and a screen including a guide UI (30) may be implemented as one screen.
[0421] According to various embodiments, the electronic device (100) may combine a screen including content, a screen including a first control UI (10), and a screen including a guide UI (30) into a single screen. The electronic device (100) may display the combined single screen.
[0422] FIG. 20 is a flowchart for explaining a guide screen displayed in relation to the connection of an electronic device (100), a controller, and an external device (200).
[0423] Referring to FIG. 20, steps S2005, S2010, S2015, S2020, S2025, S2030, S2035, S2040, S2045, S2050, S2055, S2060, S2065, and S2070 of FIG. 20 may correspond to steps S1905, S1910, S1915, S1920, S1925, S1930, S1935, S1940, S1945, S1950, S1955, S1960, S1965, and S1970 of FIG. 19. Therefore, redundant description is omitted.
[0424] After the controller (300) corresponding to the content is identified (S2015), the electronic device (100) can display a first guide screen (S2016). The first guide screen can include information guiding the connection of the controller (300) for manipulating the selected content.
[0425] If the controller (300) is not connected (S2020-N), the electronic device (100) may display a second guide screen (S2021). The second guide screen may include information guiding the connection of an external device (200) to manipulate content with a device other than the controller (300).
[0426] Depending on the various embodiments, the first guide screen and the second guide screen may be identical to a single screen. A description thereof is provided in FIG. 5.
[0427] Depending on the various embodiments, the first guide screen and the second guide screen may be different.
[0428] If the external device (200) is not connected (S2030-N), the electronic device (100) may display a third guide screen (S2031). The third guide screen may include information informing the user that a device required to manipulate content is not connected.
[0429] Depending on various embodiments, some of the operations of displaying the first guide screen, the second guide screen, and the third guide screen may be omitted.
[0430] Figure 21 is a flowchart for explaining the operation of displaying a guide UI.
[0431] Referring to Fig. 21, steps S2105 to S2170 may correspond to steps S1905 to S1970 of Fig. 19. Therefore, redundant descriptions are omitted. In Fig. 21, some steps are omitted and illustrated.
[0432] After the first control UI (10) is displayed on the electronic device (100) and the second control UI (20) is output from the external device (200), the external device (200) can obtain location information of the user input (S2175). The external device (200) can receive the user input and identify location information corresponding to the received user input. The external device (200) can transmit the location information of the user input to the electronic device (100) (S2176).
[0433] The electronic device (100) can provide a guide UI (30) based on the location information of the user input received from the external device (200) (S2180).
[0434] According to various embodiments, when a user input is received, the electronic device (100) may provide a guide UI (30) based on the location of the user input. For example, the guide UI (30) may be displayed whenever a user input is received.
[0435] Depending on various embodiments, the format (shape, size, color, transparency, etc.) of the guide UI (30) may be partially changed based on the location of the user input.
[0436] According to various embodiments, the electronic device (100) may provide the guide UI (30) only in the case of misrecognition. The case of misrecognition may mean an event in which a user input is identified as being received in the expandable area (1513) of FIG. 15.
[0437] Figure 22 is a flowchart for explaining the operation of displaying a guide UI when a button is not recognized.
[0438] Steps S2205 to S2276 of FIG. 22 may correspond to steps S2105 to S2176 of FIG. 21. Therefore, redundant description is omitted.
[0439] The electronic device (100) can identify whether the location of the user input is included in the first button area (S2281). If the location of the user input is included in the first button area (S2281-Y), the electronic device (100) can perform an operation corresponding to the first button (S2283).
[0440] If the location of the user input is not included in the first button area (S2281-N), the electronic device (100) can identify whether the location of the user input is included in the first sub-button area (S2284). If the location of the user input is included in the first sub-button area (S2284-Y), the electronic device (100) can perform an operation corresponding to the first button (S2286).
[0441] If the location of the user input is not included in the first sub-button area (S2284-N), the electronic device (100) can identify whether the location of the user input is included in the first expandable area (S2287).
[0442] If the location of the user input is included in the first expandable area (S2287-Y), the electronic device (100) can display a guide UI indicating misrecognition (S2288).
[0443] If the location of the user input is not included in the first expandable area (S2287-N), the electronic device (100) may display a guide UI indicating non-recognition (S2288).
[0444] Depending on the various embodiments, the guide UI indicating misrecognition and the guide UI indicating non-recognition may be different.
[0445] Depending on the various embodiments, the guide UI indicating misrecognition and the guide UI indicating non-recognition may be the same.
[0446] Figure 23 is a flowchart for explaining the operation of displaying a guide UI corresponding to user input.
[0447] Steps S2305 to S2376, S2381, S2383, S2384, S2386, and S2387 of FIG. 23 may correspond to steps S2205 to S2276, S2281, S2283, S2284, S2286, and S2287 of FIG. 22. Therefore, redundant descriptions are omitted.
[0448] If the location of the user input is included in the first button area (S2381-Y), the electronic device (100) can provide a guide UI (30) based on the first format (S2382). The electronic device (100) can perform an action corresponding to the first button (S2383).
[0449] If the location of the user input is included in the first sub-button area (S2384-Y), a guide UI (30) can be provided based on the second format (S2385). The electronic device (100) can perform an operation corresponding to the first button (S2386).
[0450] If the location of the user input is included in the first expandable area (S2387-Y), the electronic device (100) can provide a guide UI (30) based on the third format (S2388).
[0451] If the location of the user input is not included in the first expandable area (S2387-N), the electronic device (100) can provide a guide UI (30) based on the fourth format (S2389).
[0452] The first to fourth formats may include at least one of shape, position, size, color, and transparency.
[0453] According to various embodiments, the first to fourth formats may differ in at least one of shape, position, size, color, and transparency.
[0454] According to various embodiments, the first to fourth formats may all have the same shape, position, size, color, and transparency.
[0455] In various embodiments, the first format and the second format may be the same, the first format and the third format may be different, and the third format and the fourth format may be the same.
[0456] Depending on the various embodiments, the first format, the second format, and the third format may all be different.
[0457] In various embodiments, step S2389 may be omitted.
[0458] Guide UI is not limited to visual information, although it is described as images, icons, etc. Guide UI may include at least one of vibration, haptic, or audio signals.
[0459] According to various embodiments, when an event in which the location of a user input moves in the order of a button area, an auxiliary button area, and an expandable area is identified, the electronic device (100) may provide a notification to the user. The notification providing method may include at least one of vibration, haptic, or audio signal. The first to fourth formats may include at least one of vibration intensity, haptic intensity, and audio signal size. The electronic device (100) may generate a control signal for providing a notification and transmit it to an external device (200). The external device (200) may provide a notification based on the received control signal.
[0460] As the touch location moves in the order of button area, auxiliary button area, and expandable area, the output strength of the guide UI may also increase.
[0461] According to various embodiments, the output intensity of the guide UI may vary depending on the communication connection status, data status, or response speed between the electronic device (100) and the external device (200).
[0462] If the communication connection strength is below a critical strength or the response speed is below a critical speed, the electronic device (100) may not provide a guide UI such as a vibration, haptic, or audio signal. If the communication connection strength is below a critical strength or the response speed is below a critical speed, a guide UI (30) in the form of an image may be displayed.
[0463] When user input is received in the remaining area, the electronic device (100) may not provide a guide UI.
[0464] According to various embodiments, the electronic device (100) may display a guide UI (30) by displaying image information (or icons). If a preset event is identified while providing the guide UI (30) in the form of an image, the guide UI may be provided using at least one of a vibration, haptic, or audio signal other than an image.
[0465] A preset event may be an event that prevents the display of image information other than content on the electronic device (100). An event that prevents the display of image information (guide UI) may be an event determined by the content server providing the content itself as content that cannot display image information (guide UI). An event that prevents the display of image information (guide UI) may refer to an event that interferes with content viewing by displaying image information (guide UI).
[0466] Figure 24 is a drawing for explaining an operation of changing the position of the control UI.
[0467] Referring to FIG. 24, the electronic device (100) can change the output location of the second control UI (20). The electronic device (100) can change the location at which the second control UI (20) is displayed by changing the element location of the second UI information.
[0468] The electronic device (100) can change the output position of the second control UI (20) based on a preset event.
[0469] Example 24 (2410) of FIG. 24 shows that a second control UI (20) is displayed based on initially set second UI information.
[0470] Example 24 (2420) of FIG. 24 shows that a second control UI (20) is displayed based on changed second UI information.
[0471] According to various embodiments, the event changing from embodiment (2410) to embodiment (2420) may be an event in which a user input for changing the output location of the second control UI (20) is received. The electronic device (100) may change the output location of the second control UI (20) based on the user input (or command).
[0472] According to various embodiments, the event that changes from embodiment (2410) to embodiment (2420) may be an event in which the location of the user input moves in the order of a button area, an auxiliary button area, and an expandable area.
[0473] For example, when an event is identified in which a first user input is identified in a button area, a second user input is identified in an auxiliary button area, and a third user input is identified in an expandable area, the electronic device (100) can change the output location of the second control UI (20) based on the location of the third user input.
[0474] The first user input, the second user input, and the third user input may be consecutive inputs. The second user input may be an input acquired consecutively after the first user input was acquired. The third user input may be an input acquired consecutively after the second user input was acquired.
[0475] According to various embodiments, the event that changes from embodiment (2420) to embodiment (2410) may be an event in which the user's gaze is directed toward the external device (200). If the user's gaze is directed toward the external device (200), the electronic device (100) may determine that the user finds the changed position of the second control UI (20) uncomfortable. If the event in which the user's gaze is directed toward the external device (200) is identified, the electronic device (100) may change the position of the second control UI (20) to the initially set position.
[0476] To identify an event in which the user's gaze is directed toward an external device (200), the electronic device (100) may utilize a camera (190) or a gyro sensor. An example of utilizing a camera is described in FIG. 25.
[0477] A gyro sensor may be included in the external device (200). The external device (200) may obtain tilt information of the external device (200) through the gyro sensor. The external device (200) may transmit the tilt information to the electronic device (100). The electronic device (100) may receive the tilt information of the external device (200). Based on the tilt information of the external device (200), the electronic device (100) may identify whether the user's gaze is directed toward the external device (200). Although the user's gaze cannot be directly confirmed with the tilt information, the electronic device (100) may determine that the user feels uncomfortable with the changed position of the second control UI (20) through the external device (200) being tilted at a specific tilt.
[0478] According to various embodiments, if an event (misrecognition event) is identified in which the external device (200) tilts at a specific incline within a threshold time from the time the user input is received after the user input is located in the expandable area, the electronic device (100) may determine that the user feels uncomfortable with the changed position of the second control UI (20).
[0479] When the user's inconvenience is identified, the electronic device (100) can change the position of the second control UI (20) to the initially set position.
[0480] According to various embodiments, the event that changes from embodiment (2420) to embodiment (2410) may be an event in which no user input is received for a preset period of time. The changed position of the second control UI (20) may be a position that is somewhat more inconvenient for the user than the initially set position. When an event in which no user input is received for a preset period of time is identified, the electronic device (100) may change the position of the second control UI (20) to the initially set position.
[0481] According to various embodiments, the electronic device (100) may change the position of each of the plurality of buttons (elements) included in the second control UI (20). To change the position of each, the electronic device (100) may display a correction guide screen.
[0482] Figure 25 is a drawing for explaining an operation of analyzing a captured image including a user.
[0483] Referring to FIG. 25, the electronic device (100) may include a camera (190). The electronic device (100) may capture a photograph of a user (2510) operating an external device (200). The electronic device (100) may acquire a photographed image including the user (2510). The electronic device (100) may analyze the photographed image to identify whether the user's (2510) gaze is directed toward the external device (200).
[0484] The electronic device (100) can identify the gaze of the user (2510) based on the captured image. The electronic device (100) can identify the location of the external device (200) and the location of the iris of the user (2510) based on the captured image. The electronic device (100) can identify whether the user's gaze is directed toward the external device (200) based on the location of the external device (200) and the location of the iris of the user (2510).
[0485] When an event in which the user's gaze is directed toward an external device (200) is identified, the electronic device (100) can change the position of the changed second control UI (20) to the position of the initial setting.
[0486] FIG. 26 is a flowchart for explaining a control method of an electronic device (100) according to various embodiments.
[0487] Referring to FIG. 26, a method for controlling an electronic device communicating with an external device includes a step (S2605) of transmitting information corresponding to a UI (User Interface) related to control of content to an external device when content is output, a step (S2610) of outputting a first control UI together with content based on the information corresponding to the UI, and a step (S2615) of outputting a guide UI together with content and the first control UI at a location corresponding to user input based on an output location of the first control UI, an output location of the second control UI, and information corresponding to the user input when a user input is received through a second control UI output to the external device based on the information corresponding to the UI.
[0488] The information corresponding to the UI is first UI information, and the step (S2605) of transmitting the information corresponding to the UI to an external device may acquire second UI information based on the first UI information and size information of the external device, and transmit the second UI information to the external device.
[0489] The step of outputting the first control UI (S2610) may obtain third UI information based on the first UI information and the size information of the electronic device, and output the first control UI together with the content based on the third UI information.
[0490] The step of outputting the guide UI (S2615) may be such that when a user input is received through the second control UI output to the external device based on the second UI information, the output position of the first control UI may be obtained based on the third UI information, the output position of the second control UI may be obtained based on the second UI information, and the position corresponding to the user input may be obtained based on the information corresponding to the user input.
[0491] The control method may further include a step of outputting at least one of the first control UI or the guide UI together with the content based on a preset transparency.
[0492] The control method may further include a step of outputting the first control UI at a position corresponding to the user input when a user input for changing the position of the first control UI is received.
[0493] The step of outputting the first control UI (S2610) can acquire the location of the external device and output the first control UI at a location corresponding to the location of the external device.
[0494] The step of outputting the first control UI (S2610) may output the first control UI including at least one button among a plurality of buttons included in information corresponding to the UI.
[0495] The step of outputting the guide UI (S2615) may change the color of the guide UI and output it based on whether a location corresponding to the user input is included in a preset area when a user input is received through the second control UI output to an external device.
[0496] The control method may further include a step of transmitting a signal for controlling an output position of the second control UI to the external device based on an input pattern obtained based on the information corresponding to the plurality of user inputs when information corresponding to a plurality of user inputs is received within a preset time through the second control UI output to the external device.
[0497] The control method of an electronic device such as FIG. 26 can be executed on an electronic device having the configuration of FIG. 3 or FIG. 4, and can also be executed on an electronic device having other configurations.
[0498] The methods according to the various embodiments of the present disclosure described above can be implemented in the form of an application that can be installed on an existing electronic device.
[0499] The methods according to the various embodiments of the present disclosure described above can be implemented only with a software upgrade or a hardware upgrade for an existing electronic device.
[0500] The various embodiments of the present disclosure described above may also be performed through an embedded server provided in an electronic device, or an external server of at least one of the electronic device and the display device.
[0501] According to an example embodiment of the present disclosure, the various embodiments described above may be implemented as software including instructions stored in a machine-readable storage medium that can be read by a machine (e.g., a computer). The device may include an electronic device according to the disclosed embodiments, which is a device that can call instructions stored in the storage medium and operate according to the called instructions. When the instructions are executed by a processor, the processor may perform a function corresponding to the instructions directly or under the control of the processor using other components. The instructions may include code generated or executed by a compiler or interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. 'Non-transitory' means that the storage medium does not contain signals and is tangible, but does not distinguish between whether data is stored semi-permanently or temporarily in the storage medium.
[0502] According to one embodiment of the present disclosure, the method according to the various embodiments described above may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or online through an application store (e.g., Play Store™). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.
[0503] Each of the components (e.g., modules or programs) according to the various embodiments described above may be composed of a single or multiple entities, and some of the sub-components described above may be omitted, or other sub-components may be further included in various embodiments. Alternatively or additionally, some components (e.g., modules or programs) may be integrated into a single entity, which may perform the same or similar functions as those performed by each of the respective components prior to integration. Operations performed by modules, programs or other components according to various embodiments may be executed sequentially, in parallel, iteratively or heuristically, or at least some operations may be executed in a different order, omitted, or other operations may be added.
[0504] While specific embodiments of the present disclosure have been specifically illustrated and described, it will be understood that various changes in form and detail may be made therein without departing from the spirit and scope of the following claims.
Claims
1. In electronic devices, Communication interface for communicating with external devices; display; At least one memory for storing commands; and At least one processor operatively connected to said communication interface, said display and said at least one memory; At least one processor executing said instructions, When content is output to the above display, first UI information corresponding to the UI (User Interface) related to the control of the content is transmitted to the external device through the communication interface, Controlling the display to output the first control UI together with the content based on the first UI information; An electronic device that controls the display to output a guide UI at an input location corresponding to the user input together with the content and the first control UI based on the first UI output location of the first control UI, the second UI output location of the second control UI, and input information corresponding to the user input when a user input is received through the second control UI output to the external device based on the first UI information.
2. In paragraph 1, At least one processor executing said instructions, Through the above communication interface, second UI information is obtained based on the first UI information and the size information of the external device, An electronic device that transmits the second UI information to the external device through the communication interface.
3. In paragraph 2, At least one processor executing said instructions, Obtain third UI information based on the first UI information and the size information of the electronic device, An electronic device that controls the display to output the first control UI together with the content based on the third UI information.
4. In paragraph 3, At least one processor executing said instructions, An electronic device, wherein when the user input is received through the second control UI output to the external device based on the second UI information, the first UI output location is obtained based on the third UI information, the second UI output location is obtained based on the second UI information, and the input location is obtained based on the input information.
5. In paragraph 1, At least one processor executing said instructions, An electronic device that controls the display to output at least one of the first control UI or the guide UI together with the content based on a preset transparency.
6. In paragraph 1, At least one processor executing said instructions, An electronic device that controls the display to output the first control UI at the first UI output location corresponding to the user input when a user input for changing the first UI output location is received.
7. In paragraph 1, At least one processor executing said instructions, Obtain the device location of the external device, An electronic device that controls the display to output the first control UI at the first UI output location corresponding to the device location.
8. In paragraph 1, At least one processor executing said instructions, An electronic device that controls the display to output the first control UI including at least one button among a plurality of buttons included in the first UI information.
9. In paragraph 1, At least one processor executing said instructions, An electronic device that controls the display to change the color of the guide UI and output it based on whether the input location is in a preset area when the user input is received through the second control UI output to the external device.
10. In paragraph 1, At least one processor executing said instructions, An electronic device that, when input information corresponding to multiple user inputs is received within a preset time through the second control UI output to the external device, transmits a signal for controlling the second UI output position to the external device through the communication interface according to an input pattern obtained based on the input information corresponding to the multiple user inputs.
11. In a method for controlling an electronic device, When the content is output, a step of transmitting first UI information corresponding to a UI (User Interface) related to the control of the content to an external device; A step of outputting a first control UI together with the content based on the first UI information; and A control method, comprising: a step of outputting a guide UI at an input position corresponding to the user input together with the content and the first control UI based on a first UI output position of the first control UI, a second UI output position of the second control UI, and input information corresponding to the user input when a user input is received through a second control UI output to the external device based on the first UI information; 12. In paragraph 11, The step of transmitting information corresponding to the above UI to the external device is: Obtain second UI information based on the first UI information and the size information of the external device, A control method for transmitting the second UI information to the external device.
13. In paragraph 12, The step of outputting the above first control UI is: Obtain third UI information based on the first UI information and the size information of the electronic device, A control method for outputting the first control UI together with the content based on the third UI information.
14. In paragraph 13, The steps for outputting the above guide UI are: A control method wherein, when the user input is received through the second control UI output to the external device based on the second UI information, the first UI output location is obtained based on the third UI information, the second UI output location is obtained based on the second UI information, and the input location is obtained based on the input information.
15. In paragraph 11, The above control method is, A control method further comprising: a step of outputting at least one of the first control UI or the guide UI together with the content based on a preset transparency.
Citation Information
Patent Citations
Program and display system
JP2018143785A
Buoyancy of floating apparatus for solar generation of electric power on water
KR102015595B1
Active purge system and active purge method
KR1020200138903A
Loading and unloading device for a hollow roll
KR1020220165372A
Server, method and computer program for providing route information for logistics
KR102670807B1
