User terminal and control method therefor

Abstract

The present disclosure relates to: a user terminal capable of conveniently and quickly controlling, by using a single input device, each of a plurality of user terminals communicatively connected to each other; and a control method therefor. Provided is the user terminal comprising: an input device; a display; a communication unit for connection of a first communication connection and a second communication connection to a first external user terminal; and a control unit for controlling such that a first screen and an extended screen of the first screen are generated, the first screen is displayed on the display, the extended screen is transmitted to the first external user terminal through the first communication connection, and pointer display information of the input device and / or a user command input through the input device are / is transmitted to the first external user terminal through the second communication connection.

Description

User terminal and its control method

[0001] The present disclosure relates to a user terminal capable of sharing an input device with other user terminals, and more particularly, to a control method for controlling a plurality of user terminals that are connected to each other through communication with one input device.

[0002] User terminals can be divided into mobile / portable terminals and stationary terminals based on their mobility. Furthermore, user terminals can be divided into handheld terminals and vehicle-mounted terminals based on whether the user can carry them.

[0003] The functions of user terminals are becoming increasingly diverse. Examples include data and voice communications, camera photography and video recording, audio recording, music playback through speaker systems, and display of images and videos. Some terminals have added electronic game play capabilities or multimedia player functionality. Recent user terminals, in particular, can receive multicast signals that provide visual content such as broadcasts, videos, and television programs.

[0004] As the functions of such user terminals become more diverse, they are being implemented in the form of multimedia players with complex functions such as taking pictures or videos, playing music or video files, playing games, and receiving broadcasts.

[0005] User terminals can communicate with other user terminals via wired and / or wireless means. Each of the multiple user terminals connected to each other can be connected to a separate input device (e.g., a Human Interface Device (HID) such as a keyboard and mouse).

[0006] However, if each user terminal is equipped with a separate input device, the cost of providing multiple input devices for all user terminals can be significant. Furthermore, requiring users to selectively use only the appropriate input device to control each user terminal can be inconvenient.

[0007] This doesn't mean it's impossible to use a single input device for multiple user terminals. However, connecting a single input device to one user terminal, then disconnecting it and connecting it to another user terminal can also cause inconvenience to the user.

[0008] The present disclosure is proposed to solve such problems, and aims to provide a user terminal and a control method thereof that can conveniently and quickly control a plurality of user terminals that are connected to each other as a single input device.

[0009] In order to achieve the above object, the present disclosure can provide a user terminal including an input device, a display, a communication unit for a first communication connection and a second communication connection with a first external user terminal, and a control unit for generating a first screen and an extended screen of the first screen, displaying the first screen on the display, transmitting the extended screen to the first external user terminal through the first communication connection, and transmitting virtual coordinate information of the input device and a user command input through the input device to the first external user terminal through the second communication connection.

[0010] The above control unit can receive first display identification information from a first external user terminal via a first communication connection.

[0011] The above control unit can control to generate the extended screen based on the display resolution of the display identification data.

[0012] The control unit may receive an IP address and a product unique identifier of the first external user terminal from the first external user terminal via a wireless network, and, if the received product unique identifier is identical to the product unique identifier of the display identification data, control to match the IP address to the first external user terminal.

[0013] The control unit can perform PIN code authentication with a first external user terminal using the IP address, receive a client key code for automatic connection from the first external user terminal, and establish a second communication connection with the first external user terminal using the client key code for automatic connection.

[0014] The control unit may receive a mode status notification from a first external user terminal, and if the first external user terminal is determined to be operating as a main device through the mode status notification, request the first external user terminal to create an input communication channel in a second communication connection, and control the transmission of the virtual coordinate information of the input device and the user command input through the input device to the first external user terminal through the input communication channel.

[0015] The above extended screen can be a coordinate system of the above virtual coordinates.

[0016] The above control unit can control to establish a first communication connection and a second communication connection with a second external user terminal while a first communication connection and a second communication connection with a first external user terminal are established.

[0017] The first communication connection may be an HDMI (High Definition Multimedia Interface) or DP (DisplayPort) connection, and the second communication connection may be a Wi-Fi communication connection.

[0018] In order to achieve the above object, the present disclosure may provide a user terminal including an input device, a display, a communication unit for a first communication connection and a second communication connection with an external user terminal, and a control unit for generating a first screen and an extended screen of the first screen, displaying the first screen on the display, transmitting the extended screen to the external user terminal through the first communication connection, and transmitting at least one of pointer display information of the input device and a user command input through the input device to the external user terminal through the second communication connection.

[0019] The above control unit can receive display identification data from the external user terminal through the first communication connection.

[0020] The control unit may receive an IP address and a product unique identifier of the external user terminal from the external user terminal via a wireless network, and, if the received product unique identifier is identical to the product unique identifier of the display identification data, may control matching the IP address to the external user terminal, performing PIN code authentication with the external user terminal using the IP address, receiving a client key code for automatic connection from the external user terminal, and establishing a second communication connection with the external user terminal using the client key code for automatic connection.

[0021] The control unit may receive a mode status notification from the external user terminal, and if the external user terminal is determined to be operating as a main device through the mode status notification, request the external user terminal to create an input communication channel in a second communication connection, and control to transmit at least one of the pointer display information of the input device and the user command input through the input device to the external user terminal through the input communication channel.

[0022] The above control unit can control, in response to an SDP (Service Discovery Protocol) request signal received from the external user terminal, to transmit an SDP response signal to the external user terminal and perform Bluetooth pairing with the external user terminal as a second communication connection.

[0023] The control unit can determine whether the external user terminal operates in the first mode or the second mode based on a mode status notification signal received from the external user terminal.

[0024] The above control unit can control to display a pointer on the display when the external user terminal operates in the second mode, and not to display the pointer on the display when the external user terminal operates in the first mode.

[0025] The control unit may control the external user terminal to execute an operation corresponding to the user command when a user command is input through the input device while the external user terminal is operating in the second mode, and to transmit the user command to the external user terminal through a second communication connection without executing the operation corresponding to the user command when the user command is input through the input device while the external user terminal is operating in the first mode.

[0026] The control unit can control the external user terminal to display a pointer on the display while the external user terminal is operating in the first mode, and to transmit a control target selection notification to the external user terminal in response to the pointer being controlled through the input device to disappear by passing through an edge area of ​​the display.

[0027] The control unit may control the user to execute an operation corresponding to the user command when a user command is input through the input device while the pointer is displayed on the display, and to transmit the user command to the external user terminal through a second communication connection without executing the operation corresponding to the user command when the user command is input through the input device after the pointer passes through the edge area and disappears.

[0028] In addition, to achieve the above object, the present disclosure may provide a user terminal including an input device, a display, a communication unit for a first communication connection and a second communication connection with an external user terminal, and a control unit for controlling, while operating in a second mode, to display an extended screen received from the external user terminal through the first communication connection on the display, wherein the extended screen may include a first pointer, and while operating in the first mode, to display a second screen on the display, wherein the second pointer is generated based on pointer display information received through the second communication connection and is displayed on the second screen.

[0029] The control unit can control to create an input communication channel in response to an input communication channel creation request received from the external user terminal through a second communication connection and to transmit information of the created input communication channel to the external user terminal.

[0030] The above control unit can be controlled to perform an operation according to a user command received through the input communication channel.

[0031] The above control unit can control to transmit a mode status notification to the external user terminal whenever a mode switch is performed between the first mode and the second mode.

[0032] The above control unit can be controlled to perform an operation according to a user command received through a second communication connection while operating in the first mode.

[0033] The above control unit can control to display a second pointer based on the pointer display information received through the second communication connection on the second screen when receiving a control target selection notification from the external user terminal while operating in the first mode.

[0034] The control unit may control the external user terminal to transmit a control target release notification in response to the pointer passing through the edge area of ​​the display and disappearing.

[0035] In addition, to achieve the above purpose, the present disclosure may provide a method for controlling a user terminal, including the steps of establishing a first communication connection and a second communication connection with a first external user terminal, the step of generating a first screen and displaying it on a display, the step of generating an extended screen of the first screen and transmitting it to the external user terminal through the first communication connection, and the step of transmitting at least one of pointer display information of an input device and a user command input through the input device to the external user terminal through the second communication connection.

[0036] In addition, to achieve the above object, the present disclosure may provide a method for controlling a user terminal, including the steps of establishing a first communication connection and a second communication connection with an external user terminal, a step of displaying an extended screen received from the external user terminal through the first communication connection on a display while operating in a second mode, wherein the extended screen may include a first pointer, and a step of displaying a second screen on the display while operating in the first mode, wherein the second pointer is generated based on pointer display information received through the second communication connection and is displayed on the second screen.

[0037] The effects of the user terminal and the control method therefor according to the present disclosure are described as follows.

[0038] At least one of the embodiments of the present disclosure provides the advantage of eliminating the need to provide multiple input devices for all of the multiple user terminals by allowing one input device to be used to control each of the multiple user terminals.

[0039] In addition, according to at least one of the embodiments of the present disclosure, there is an advantage in that a user can control each of a plurality of user terminals with one input device, and thus can conveniently and quickly control each of the plurality of user terminals.

[0040] FIG. 1 is a block diagram illustrating a user terminal related to the present disclosure.

[0041] FIG. 2 illustrates a plurality of user terminals connected to each other in communication according to one aspect of the present disclosure.

[0042] FIG. 3 is an example of a user terminal operating as a sub-device according to one aspect of the present disclosure.

[0043] FIG. 4 and FIG. 5 are examples of a user terminal operating as a main device according to one aspect of the present disclosure.

[0044] FIG. 6 illustrates a communication connection process between multiple user terminals according to one aspect of the present disclosure.

[0045] FIG. 7 illustrates a mode switching process of a user terminal according to one aspect of the present disclosure.

[0046] FIGS. 8 and 9 illustrate a plurality of user terminals connected to each other in communication according to one aspect of the present disclosure.

[0047] FIG. 10 illustrates a communication connection process between multiple user terminals according to one aspect of the present disclosure.

[0048] FIG. 11 illustrates a communication connection process between multiple user terminals according to one aspect of the present disclosure.

[0049] FIG. 12 illustrates a mode switching process of a user terminal according to one aspect of the present disclosure.

[0050] FIG. 13 illustrates a first user terminal and a second user terminal that are connected to each other in communication according to one aspect of the present disclosure.

[0051] FIG. 14 is a control flowchart of a first user terminal and a second user terminal that are connected to each other in communication according to one aspect of the present disclosure.

[0052] FIG. 15 illustrates a first user terminal and a second user terminal that are connected to each other in communication according to one aspect of the present disclosure.

[0053] FIG. 16 is a control flowchart of a first user terminal and a second user terminal that are connected to each other in communication according to one aspect of the present disclosure.

[0054] FIG. 17 illustrates a first user terminal and a second user terminal that are connected to each other in communication according to one aspect of the present disclosure.

[0055] Hereinafter, embodiments disclosed in the present disclosure will be described in detail with reference to the attached drawings. Regardless of the drawing numbers, identical or similar components will be given the same reference numbers, and redundant descriptions thereof will be omitted. The suffixes "module" and "part" used for components in the following description are given or used interchangeably only for the convenience of writing the disclosure, and do not in themselves have distinct meanings or roles. In addition, when describing embodiments disclosed in the present disclosure, if it is determined that a specific description of a related known technology may obscure the gist of the embodiments disclosed in the present disclosure, a detailed description thereof will be omitted. In addition, the attached drawings are only for easily understanding the embodiments disclosed in the present disclosure, and the technical idea disclosed in the present disclosure is not limited by the attached drawings, and it should be understood that they include all modifications, equivalents, and substitutes included in the spirit and technical scope of the present invention.

[0056] These components may each be implemented as separate individual hardware modules, or may be implemented as two or more hardware modules, or two or more components may be implemented as one hardware module, and in some cases, they may also be implemented as software.

[0057] Terms that include ordinal numbers, such as first, second, etc., may be used to describe various components, but the components are not limited by these terms. These terms are used solely to distinguish one component from another.

[0058] When a component is referred to as being "connected" or "connected" to another component, it should be understood that it may be directly connected or connected to that other component, but that there may be other components intervening. Conversely, when a component is referred to as being "directly connected" or "connected" to another component, it should be understood that there are no other components intervening.

[0059] Singular expressions include plural expressions unless the context clearly dictates otherwise. In this disclosure, terms such as "comprises" or "have" are intended to indicate the presence of a feature, number, step, operation, component, part, or combination thereof described in the disclosure, but should be understood not to preclude the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0060] In this disclosure, the expression “at least one of A and B” may mean “A,” may mean “B,” or may mean both “A” and “B.”

[0061] The user terminal described in the present disclosure may include a mobile phone, a smart phone, a laptop computer, a digital broadcasting terminal, a personal digital assistant (PDA), a portable multimedia player (PMP), a navigation device, a slate PC, a tablet PC, an ultrabook, a wearable device (e.g., a smartwatch, a smart glass, a head mounted display (HMD)), and the like.

[0062] However, it will be readily apparent to those skilled in the art that the configuration according to the embodiments described in the present disclosure may also be applied to fixed terminals such as digital TVs, desktop computers, digital signage, etc., except in cases where it is applicable only to user terminals.

[0063] Referring to FIG. 1, FIG. 1 is a block diagram illustrating a user terminal related to the present disclosure.

[0064] The user terminal (100) may include a wireless communication unit (110), an input unit (120), a sensing unit (140), an output unit (150), an interface unit (160), a memory (170), a control unit (180), and a power supply unit (190). The components illustrated in FIG. 1 are not essential for implementing the user terminal, and thus the user terminal described in this specification may have more or fewer components than the components listed above.

[0065] More specifically, among the above components, the wireless communication unit (110) may include one or more modules that enable wireless communication between a user terminal (100) and a wireless communication system, between a user terminal (100) and another user terminal (100), or between a user terminal (100) and an external server. In addition, the wireless communication unit (110) may include one or more modules that connect the user terminal (100) to one or more networks.

[0066] This wireless communication unit (110) may include at least one of a broadcast reception module (111), a mobile communication module (112), a wireless Internet module (113), a short-range communication module (114), and a location information module (115).

[0067] The input unit (120) may include a camera (121) or a video input unit for inputting a video signal, a microphone (122) or an audio input unit for inputting an audio signal, and a user input unit (123, for example, a touch key, a mechanical key, etc.) for receiving information from a user. Voice data or image data collected by the input unit (120) may be analyzed and processed into a user's control command.

[0068] The sensing unit (140) may include one or more sensors for sensing at least one of information within the user terminal, information about the surrounding environment surrounding the user terminal, and user information. For example, the sensing unit (140) may include at least one of a proximity sensor (141), an illumination sensor (142), a touch sensor, an acceleration sensor, a magnetic sensor, a gravity sensor (G-sensor), a gyroscope sensor, a motion sensor, an RGB sensor, an infrared sensor (IR sensor), a fingerprint recognition sensor, an ultrasonic sensor, an optical sensor (e.g., a camera (see 121)), a microphone (see 122), a battery gauge, an environmental sensor (e.g., a barometer, a hygrometer, a thermometer, a radiation detection sensor, a heat detection sensor, a gas detection sensor, etc.), and a chemical sensor (e.g., an electronic nose, a healthcare sensor, a biometric recognition sensor, etc.). Meanwhile, the user terminal disclosed in this specification can utilize information sensed by at least two of these sensors in combination.

[0069] The output unit (150) is for generating output related to visual, auditory, or tactile sensations, and may include at least one of a display unit (151), an audio output unit (152), a haptic module (153), and an optical output unit (154). The display unit (151) may be formed as a layer structure with a touch sensor or formed as an integral part, thereby implementing a touch screen. This touch screen may function as a user input unit (123) that provides an input interface between the user terminal (100) and the user, and at the same time, may provide an output interface between the user terminal (100) and the user.

[0070] The interface unit (160) serves as a passageway for various types of external devices connected to the user terminal (100). This interface unit (160) may include at least one of a wired / wireless headset port, an external charger port, a wired / wireless data port, a memory card port, a port for connecting a device equipped with an identification module, an audio I / O (Input / Output) port, a video I / O (Input / Output) port, and an earphone port. In response to the external device being connected to the interface unit (160), the user terminal (100) may perform appropriate control related to the connected external device.

[0071] In addition, the memory (170) stores data that supports various functions of the user terminal (100). The memory (170) can store a plurality of application programs (or applications) running on the user terminal (100), data for the operation of the user terminal (100), and commands. At least some of these application programs can be downloaded from an external server via wireless communication. In addition, at least some of these application programs can exist on the user terminal (100) from the time of shipment for the basic functions of the user terminal (100) (e.g., call receiving, calling, message receiving, calling). Meanwhile, the application programs can be stored in the memory (170), installed on the user terminal (100), and driven by the control unit (180) to perform operations (or functions) of the user terminal.

[0072] In addition to the operations related to the above application program, the control unit (180) typically controls the overall operation of the user terminal (100). The control unit (180) can provide or process appropriate information or functions to the user by processing signals, data, information, etc. input or output through the components discussed above or by operating an application program stored in the memory (170).

[0073] In addition, the control unit (180) can control at least some of the components discussed with reference to FIG. 1 to drive an application program stored in the memory (170). Furthermore, the control unit (180) can operate at least two or more of the components included in the user terminal (100) in combination to drive the application program.

[0074] The power supply unit (190) receives external power and internal power under the control of the control unit (180) and supplies power to each component included in the user terminal (100). The power supply unit (190) includes a battery, and the battery may be a built-in battery or a replaceable battery.

[0075] At least some of the above components may cooperate with each other to implement the operation, control, or control method of the user terminal according to the various embodiments described below. In addition, the operation, control, or control method of the user terminal may be implemented on the user terminal by running at least one application program stored in the memory (170).

[0076] Below, before examining various embodiments implemented through the user terminal (100) discussed above, the components listed above will be examined in more detail with reference to FIG. 1.

[0077] First, regarding the wireless communication unit (110), the broadcast reception module (111) of the wireless communication unit (110) receives broadcast signals and / or broadcast-related information from an external broadcast management server via a broadcast channel. The broadcast channel may include a satellite channel or a terrestrial channel. Two or more of the broadcast reception modules may be provided in the mobile terminal (100) for simultaneous broadcast reception or broadcast channel switching for at least two broadcast channels.

[0078] The above broadcast management server may refer to a server that generates and transmits broadcast signals and / or broadcast-related information, or a server that receives previously generated broadcast signals and / or broadcast-related information and transmits them to a terminal. The above broadcast signals may include not only TV broadcast signals, radio broadcast signals, and data broadcast signals, but may also include broadcast signals in the form of TV broadcast signals or radio broadcast signals combined with data broadcast signals.

[0079] The above broadcast signal may be encoded according to at least one of the technical standards (or broadcasting methods, for example, ISO, IEC, DVB, ATSC, etc.) for transmitting and receiving digital broadcast signals, and the broadcast receiving module (111) may receive the digital broadcast signal using a method suitable for the technical specifications set forth in the above technical standards.

[0080] The above broadcast-related information may refer to information related to a broadcast channel, broadcast program, or broadcast service provider. The above broadcast-related information may also be provided via a mobile communications network. In such cases, it may be received by the mobile communications module (112).

[0081] The above broadcast-related information may exist in various forms, such as, for example, the Electronic Program Guide (EPG) of DMB (Digital Multimedia Broadcasting) or the Electronic Service Guide (ESG) of DVB-H (Digital Video Broadcast-Handheld). The broadcast signal and / or broadcast-related information received through the broadcast reception module (111) may be stored in the memory (170).

[0082] The mobile communication module (112) transmits and receives wireless signals with at least one of a base station, an external terminal, and a server on a mobile communication network constructed according to technical standards or communication methods for mobile communication (e.g., GSM (Global System for Mobile communication), CDMA (Code Division Multi Access), CDMA2000 (Code Division Multi Access 2000), EV-DO (Enhanced Voice-Data Optimized or Enhanced Voice-Data Only), WCDMA (Wideband CDMA), HSDPA (High Speed ​​Downlink Packet Access), HSUPA (High Speed ​​Uplink Packet Access), LTE (Long Term Evolution), LTE-A (Long Term Evolution-Advanced), 5G, etc.).

[0083] The above wireless signal may include various forms of data such as voice call signals, video call signals, or text / multimedia message transmission and reception.

[0084] The wireless Internet module (113) refers to a module for wireless Internet access, and can be built into or externally installed in the user terminal (100). The wireless Internet module (113) is configured to transmit and receive wireless signals in a communication network according to wireless Internet technologies.

[0085] Wireless Internet technologies include, for example, WLAN (Wireless LAN), Wi-Fi (Wireless-Fidelity), Wi-Fi (Wireless Fidelity) Direct, DLNA (Digital Living Network Alliance), WiBro (Wireless Broadband), WiMAX (World Interoperability for Microwave Access), HSDPA (High Speed ​​Downlink Packet Access), HSUPA (High Speed ​​Uplink Packet Access), LTE (Long Term Evolution), LTE-A (Long Term Evolution-Advanced), 5G, etc., and the wireless Internet module (113) transmits and receives data according to at least one wireless Internet technology, including Internet technologies not listed above.

[0086] From the perspective that wireless Internet access through WiBro, HSDPA, HSUPA, GSM, CDMA, WCDMA, LTE, LTE-A, 5G, etc. is achieved through a mobile communication network, the wireless Internet module (113) that performs wireless Internet access through the mobile communication network may be understood as a type of the mobile communication module (112).

[0087] The short-range communication module (114) is for short-range communication, and can support short-range communication using at least one of Bluetooth™, RFID (Radio Frequency Identification), Infrared Data Association (IrDA), UWB (Ultra Wideband), ZigBee, NFC (Near Field Communication), Wi-Fi (Wireless-Fidelity), Wi-Fi Direct, and Wireless USB (Wireless Universal Serial Bus) technologies. The short-range communication module (114) can support wireless communication between a user terminal (100) and a wireless communication system, between a user terminal (100) and another user terminal (100), or between a user terminal (100) and a network where another user terminal (100, or an external server) is located through a short-range wireless communication network (Wireless Area Networks). The short-range wireless communication network may be a short-range wireless personal area network (Wireless Personal Area Networks).

[0088] Here, the other user terminal (100) may be a wearable device (e.g., a smartwatch, smart glasses, or a head mounted display (HMD)) capable of exchanging data with (or linking with) the user terminal (100) according to the present disclosure. The short-range communication module (114) may detect (or recognize) a wearable device capable of communicating with the user terminal (100) around the user terminal (100). Furthermore, if the detected wearable device is a device authenticated to communicate with the user terminal (100) according to the present disclosure, the control unit (180) may transmit at least a portion of the data processed in the user terminal (100) to the wearable device via the short-range communication module (114). Accordingly, the user of the wearable device may utilize the data processed in the user terminal (100) via the wearable device. For example, according to this, when a call is received on the user terminal (100), the user can make a phone call through the wearable device, or when a message is received on the user terminal (100), the user can check the received message through the wearable device.

[0089] The location information module (115) is a module for obtaining the location (or current location) of the user terminal, and representative examples thereof include a GPS (Global Positioning System) module or a WiFi (Wireless Fidelity) module. For example, if the user terminal utilizes a GPS module, the user terminal can obtain the location of the user terminal using signals transmitted from GPS satellites. As another example, if the user terminal utilizes a Wi-Fi module, the user terminal can obtain the location of the user terminal based on information from a wireless access point (AP) that transmits or receives wireless signals with the Wi-Fi module. If necessary, the location information module (115) may perform the function of any of the other modules of the wireless communication unit (110) to obtain data regarding the location of the user terminal as a substitute or in addition. The location information module (115) is a module used to obtain the location (or current location) of the user terminal, and is not limited to a module that directly calculates or obtains the location of the user terminal.

[0090] Next, the input unit (120) is for inputting image information (or signal), audio information (or signal), data, or information input from a user. For inputting image information, the user terminal (100) may be equipped with one or more cameras (121). The camera (121) processes image frames such as still images or moving images obtained by an image sensor in a video call mode or a shooting mode. The processed image frames may be displayed on the display unit (151) or stored in the memory (170). Meanwhile, the plurality of cameras (121) equipped in the user terminal (100) may be arranged to form a matrix structure, and through the cameras (121) forming the matrix structure in this way, a plurality of image information having various angles or focal points may be input to the user terminal (100). In addition, the plurality of cameras (121) may be arranged in a stereo structure to obtain left and right images for implementing a three-dimensional image. Additionally, the plurality of cameras (121) may include a depth camera and / or a TOF (Time of Flight) camera for sensing a subject in three dimensions.

[0091] The microphone (122) processes external acoustic signals into electrical voice data. The processed voice data can be utilized in various ways depending on the function being performed (or the application being executed) on the user terminal (100). Meanwhile, the microphone (122) can implement various noise removal algorithms to remove noise generated during the process of receiving external acoustic signals.

[0092] The user input unit (123) is for receiving information from the user. When information is input through the user input unit (123), the control unit (180) can control the operation of the user terminal (100) to correspond to the input information. The user input unit (123) may include a mechanical input means (or a mechanical key, for example, a button located on the front, back, or side of the user terminal (100), a dome switch, a jog wheel, a jog switch, etc.) and a touch input means. As an example, the touch input means may be composed of a virtual key, a soft key, or a visual key displayed on a touch screen through software processing, or a touch key placed on a part other than the touch screen. Meanwhile, the virtual key or visual key may be displayed on the touch screen in various forms, and may be formed of, for example, graphics, text, icons, videos, or a combination thereof.

[0093] Meanwhile, the sensing unit (140) senses at least one of information within the user terminal, information regarding the surrounding environment surrounding the user terminal, and user information, and generates a sensing signal corresponding thereto. Based on this sensing signal, the control unit (180) can control the operation or behavior of the user terminal (100), or perform data processing, functions, or operations related to an application program installed in the user terminal (100). Representative sensors among the various sensors that can be included in the sensing unit (140) will be examined in more detail.

[0094] First, a proximity sensor (141) refers to a sensor that detects the presence or absence of an object approaching a predetermined detection surface or an object existing nearby without mechanical contact by using the force of an electromagnetic field or infrared rays. This proximity sensor (141) may be placed in an internal area of ​​a user terminal covered by the touch screen described above or near the touch screen.

[0095] Examples of proximity sensors (141) include a transmissive photoelectric sensor, a direct reflection photoelectric sensor, a mirror reflection photoelectric sensor, a high-frequency oscillation proximity sensor, a capacitive proximity sensor, a magnetic proximity sensor, and an infrared proximity sensor. In the case where the touch screen is capacitive, the proximity sensor (141) may be configured to detect the proximity of a conductive object by a change in an electric field caused by the object's proximity. In this case, the touch screen (or touch sensor) itself may be classified as a proximity sensor.

[0096] Meanwhile, for the convenience of explanation, an act of bringing an object close to the touch screen without touching it so that it is recognized that the object is located on the touch screen is called a "proximity touch," and an act of actually making contact with the touch screen is called a "contact touch." The position at which an object is touched close to the touch screen means the position at which the object corresponds vertically to the touch screen when the object is touched close to the touch screen. The proximity sensor (141) can detect a proximity touch and a proximity touch pattern (e.g., proximity touch distance, proximity touch direction, proximity touch speed, proximity touch time, proximity touch position, proximity touch movement status, etc.). Meanwhile, the control unit (180) processes data (or information) corresponding to the proximity touch action and proximity touch pattern detected through the proximity sensor (141) as described above, and further outputs visual information corresponding to the processed data on the touch screen. Furthermore, the control unit (180) can control the user terminal (100) so that different actions or data (or information) are processed depending on whether a touch on the same point on the touch screen is a proximity touch or a contact touch.

[0097] The touch sensor detects a touch (or touch input) applied to a touch screen (or display unit (151)) using at least one of various touch methods, such as a resistive method, a capacitive method, an infrared method, an ultrasonic method, and a magnetic field method.

[0098] As an example, a touch sensor may be configured to convert changes in pressure applied to a specific portion of a touch screen or electrostatic capacity generated at a specific portion, etc., into an electrical input signal. The touch sensor may be configured to detect a location, area, pressure at the time of touch, electrostatic capacity at the time of touch, etc., of a touch target that applies a touch to the touch screen on the touch sensor. Here, the touch target is an object that applies a touch to the touch sensor, and may be, for example, a finger, a touch pen or a stylus pen, a pointer, etc.

[0099] In this way, when there is a touch input to the touch sensor, the corresponding signal(s) is sent to the touch controller. The touch controller processes the signal(s) and then transmits the corresponding data to the control unit (180). As a result, the control unit (180) can determine which area of ​​the display unit (151) has been touched, etc. Here, the touch controller may be a separate component from the control unit (180) or may be the control unit (180) itself.

[0100] Meanwhile, the control unit (180) may perform different or identical controls depending on the type of touch target that touches the touch screen (or a touch key provided in addition to the touch screen). Whether to perform different or identical controls depending on the type of touch target may be determined based on the current operating status of the user terminal (100) or the running application program.

[0101] Meanwhile, the touch sensor and proximity sensor discussed above can independently or in combination sense various types of touches, such as short (or tap) touch, long touch, multi-touch, drag touch, flick touch, pinch-in touch, pinch-out touch, swipe touch, hovering touch, etc. on the touch screen.

[0102] An ultrasonic sensor can recognize the location information of a detection target using ultrasonic waves. Meanwhile, the control unit (180) can calculate the location of a wave generation source using information detected by a light sensor and multiple ultrasonic sensors. The location of the wave generation source can be calculated by taking advantage of the fact that light is much faster than ultrasonic waves, that is, the time it takes for light to reach the light sensor is much faster than the time it takes for ultrasonic waves to reach the ultrasonic sensor. More specifically, the location of the wave generation source can be calculated by using the time difference between the time at which ultrasonic waves arrive and the time at which light is a reference signal.

[0103] Meanwhile, the camera (121) as seen in the configuration of the input unit (120) includes at least one of a camera sensor (e.g., CCD, CMOS, etc.), a photo sensor (or image sensor), and a laser sensor.

[0104] A camera (121) and a laser sensor can be combined with each other to detect the touch of a detection target on a three-dimensional stereoscopic image. The photo sensor can be laminated on a display element, and the photo sensor is configured to scan the movement of a detection target close to the touch screen. More specifically, the photo sensor scans the contents placed on the photo sensor by using an electrical signal that changes according to the amount of light applied to the photo diode by mounting a photo diode and a TR (transistor) in rows / columns. That is, the photo sensor performs coordinate calculation of the detection target according to the amount of change in light, and through this, the location information of the detection target can be acquired.

[0105] The display unit (151) displays (outputs) information processed in the user terminal (100). For example, the display unit (151) can display execution screen information of an application program running in the user terminal (100), or UI (User Interface) or GUI (Graphical User Interface) information according to such execution screen information.

[0106] Additionally, the display unit (151) may be configured as a stereoscopic display unit that displays a stereoscopic image.

[0107] The above stereoscopic display unit can be applied with a three-dimensional display method such as a stereoscopic method (glasses method), an autostereoscopic method (glasses-free method), or a projection method (holographic method).

[0108] The audio output unit (152) can output audio data received from the wireless communication unit (110) or stored in the memory (170) in a call signal reception mode, call mode or recording mode, voice recognition mode, broadcast reception mode, etc. The audio output unit (152) also outputs audio signals related to functions performed in the user terminal (100) (e.g., call signal reception sound, message reception sound, etc.). The audio output unit (152) may include a receiver, a speaker, a buzzer, etc.

[0109] The haptic module (153) generates various tactile effects that can be felt by the user. A representative example of the tactile effect generated by the haptic module (153) may be vibration. The intensity and pattern of the vibration generated by the haptic module (153) may be controlled by the user's selection or the settings of the control unit. For example, the haptic module (153) may synthesize and output different vibrations or output them sequentially.

[0110] In addition to vibration, the haptic module (153) can generate various tactile effects, such as effects caused by stimulation such as pin arrangements moving vertically with respect to the contact skin surface, air injection or suction force through nozzles or suction ports, brushing against the skin surface, contact with electrodes, electrostatic force, and effects caused by reproduction of hot and cold sensations using elements capable of absorbing or exotherming heat.

[0111] The haptic module (153) can not only transmit tactile effects through direct contact, but can also be implemented so that the user can feel the tactile effects through the muscle senses of the fingers or arms, etc. Two or more haptic modules (153) may be provided depending on the configuration of the user terminal (100).

[0112] The light output unit (154) outputs a signal to notify the occurrence of an event using light from a light source of the user terminal (100). Examples of events occurring in the user terminal (100) may include message reception, call signal reception, missed call, alarm, schedule notification, email reception, and information reception through an application.

[0113] The signal output from the light output unit (154) is implemented by the user terminal emitting light of a single color or multiple colors to the front or rear. The signal output may be terminated when the user terminal detects the user's event confirmation.

[0114] The interface unit (160) acts as a passageway for all external devices connected to the user terminal (100). The interface unit (160) receives data from an external device, supplies power, and transmits it to each component inside the user terminal (100), or allows data inside the user terminal (100) to be transmitted to an external device. For example, a wired / wireless headset port, an external charger port, a wired / wireless data port, a memory card port, a port for connecting a device equipped with an identification module, an audio I / O (Input / Output) port, a video I / O (Input / Output) port, an earphone port, an HDMI (High Definition Multimedia Interface) port, a USB (Universal Serial Bus) port, a Thunderbolt port, a DisplayPort, etc. may be included in the interface unit (160). When the user terminal (100) is connected to the external device through the wireless communication unit (110), the wireless communication unit (110) can be understood as a type of interface unit (160).

[0115] Meanwhile, the identification module is a chip that stores various information for authenticating the usage rights of the user terminal (100), and may include a user identity module (UIM), a subscriber identity module (SIM), a universal subscriber identity module (USIM), etc. A device equipped with an identification module (hereinafter referred to as an "identification device") may be manufactured in the form of a smart card. Accordingly, the identification device may be connected to the terminal (100) through the interface unit (160).

[0116] In addition, the interface unit (160) may serve as a passage through which power from the cradle is supplied to the user terminal (100) when the user terminal (100) is connected to an external cradle, or may serve as a passage through which various command signals input by the user from the cradle are transmitted to the user terminal (100). Various command signals or the power input from the cradle may function as signals for recognizing that the user terminal (100) is correctly mounted on the cradle.

[0117] The memory (170) can store a program for the operation of the control unit (180), and can also temporarily store input / output data (e.g., phonebook, messages, still images, videos, etc.). The memory (170) can store data regarding various patterns of vibration and sound output when a touch input is made on the touch screen.

[0118] The memory (170) may include at least one type of storage medium among a flash memory type, a hard disk type, an SSD (Solid State Disk type), an SDD (Silicon Disk Drive type), a multimedia card micro type, a card type memory (e.g., SD or XD memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, and an optical disk. The user terminal (100) may also be operated in relation to web storage that performs the storage function of the memory (170) on the Internet.

[0119] Meanwhile, as previously discussed, the control unit (180) controls operations related to the application program and, typically, the overall operation of the user terminal (100). For example, if the state of the user terminal satisfies a set condition, the control unit (180) may execute or release a lock state that restricts the user's input of control commands for applications.

[0120] In addition, the control unit (180) may perform control and processing related to voice calls, data communications, video calls, etc., or may perform pattern recognition processing that can recognize handwriting input or drawing input performed on a touch screen as characters and images, respectively. Furthermore, the control unit (180) may control any one or a combination of the components described above in order to implement various embodiments described below on a user terminal (100) according to the present disclosure.

[0121] The power supply unit (190) receives external power and internal power under the control of the control unit (180) and supplies the power required for the operation of each component. The power supply unit (190) includes a battery, and the battery may be a built-in battery that is rechargeable and may be detachably connected to the terminal body for charging, etc.

[0122] Additionally, the power supply unit (190) may be equipped with a connection port, and the connection port may be configured as an example of an interface (160) to which an external charger that supplies power for charging a battery is electrically connected.

[0123] As another example, the power supply unit (190) may be configured to wirelessly charge the battery without using the above-described connection port. In this case, the power supply unit (190) may receive power from an external wireless power transmission device using at least one of an inductive coupling method based on a magnetic induction phenomenon and a magnetic resonance coupling method based on an electromagnetic resonance phenomenon.

[0124] Meanwhile, the various embodiments described below may be implemented in a recording medium readable by a computer or similar device, for example, using software, hardware, or a combination thereof.

[0125] Hereinafter, a user terminal configured as described above can be connected to other user terminals for communication, which will be further described with reference to FIG. 2. FIG. 2 illustrates a plurality of user terminals connected to each other for communication according to one aspect of the present disclosure.

[0126] Among the multiple user terminals in FIG. 2, one is referred to as a first user terminal (100A) and the other is referred to as a second user terminal (100B). Each of the first user terminal (100A) and the second user terminal (100B) may correspond to the user terminal (100) described in FIG. 1.

[0127] Let's assume that the first user terminal (100A) is a personal computer, such as a laptop, and the second user terminal (100B) is a smart monitor. This is merely an example, and the first user terminal (100A) and the second user terminal (100B) are not limited thereto. For example, the first user terminal (100A) may be a mobile terminal, such as a smartphone or tablet, and the second user terminal (100B) may be a TV. Alternatively, the first user terminal (100A) may be a smart monitor, and the second user terminal (100B) may be a personal computer.

[0128] Hereinafter, the names of each component of the first user terminal (100A) will be preceded by “first” and the drawing number thereof will be indicated by A, and the names of each component of the second user terminal (100B) will be preceded by “second” and the drawing number thereof will be indicated by B. For example, the control unit of the first user terminal (100A) will be indicated as the first control unit (180A), and the control unit of the second user terminal (100B) will be indicated as the second control unit (180B).

[0129] The first user terminal (100A) can be connected to the second user terminal (100B) via wired communication. The wired communication connection can be implemented using, for example, one of the following methods: HDMI, USB, Thunderbolt, DisplayPort, etc. (but not limited thereto).

[0130] Alternatively, the first user terminal (100A) may be connected wirelessly to the second user terminal (100B). The wireless communication connection may be implemented by one of the following methods (but not limited to): Bluetooth, UWB (Ultra-wideband), Miracast, AirPlay, etc.

[0131] Meanwhile, the first user terminal (100A) and the second user terminal (100B) may be driven by different operating systems. For example, the first user terminal (100A) may be driven by an operating system (hereinafter, referred to as the first operating system) such as Microsoft's Windows or Apple's macOS as a personal computer. The second user terminal (100B) may be driven by an operating system (hereinafter, referred to as the second operating system) such as the Linux-based webOS as a smart monitor. The first operating system of the first user terminal (100A) does not necessarily have to be different from the second operating system of the second user terminal (100B). The first operating system of the first user terminal (100A) may be the same as the second operating system of the second user terminal (100A).

[0132] The first user terminal (100A) can operate as a main device. Accordingly, the first user terminal (100A) can display its own screen, i.e., the first screen (200A), according to the first operating system of the first user terminal (100A), on the first display (151A), as illustrated in (2-1) of FIG. 2.

[0133] A second user terminal (100B) that is connected to the first user terminal (100A) in communication can operate as another main device when operating in the first mode, and can operate as a sub-device of the first user terminal (100A) when operating in the second mode.

[0134] When the second user terminal (100B) operates as a main device, it can display its own screen, i.e., the second screen (200B), according to the second operating system of the second user terminal (100B) on the second display (151B), as shown in (2-2) of FIG. 2.

[0135] When the second user terminal (100B) operates as a sub-device, it can receive an extended screen or a duplicate screen (300B) of the first screen (200A) from the first user terminal (100A) through a communication connection and display it on the second display (151B), as shown in (2-3) of FIG. 2. The extended screen refers to a screen connected to one side of the first screen (200A), and the duplicate screen may refer to a screen identical to the first screen (200A). Hereinafter, the second user terminal (100B) will be described assuming that it displays an extended screen of the first screen (200A) of the first user terminal (100A) when operating as a sub-device.

[0136] Hereinafter, a case in which a second user terminal (100B) operates as a sub-device will be described with reference to FIG. 3. FIG. 3 is an example in which a user terminal operates as a sub-device according to one aspect of the present disclosure.

[0137] As illustrated in (3-1) of FIG. 3, the first user terminal (100A) can display a first pointer (or cursor) (400A) on the first screen (200A). The first pointer (400A) may be generated by the first user terminal (100A) and controlled by a first input device (e.g., a first user input unit (123A)) of the first user terminal (100A). Examples of the first input device may include at least one of a keyboard, a mouse, a touchpad, etc. The first input device may be built into the first user terminal (100A) or may be externally connected to communicate with the first user terminal (100A). The user can input his or her user command into the first user terminal (400A) by controlling the first pointer (400A) using the first input device. For example, a user may use a first input device to move a first pointer (400A) over any clickable graphical object (e.g., an icon, menu, link, text input box, etc.) on a first screen (200A) to select (e.g., click) a desired graphical object.

[0138] As shown in (3-2) of FIG. 3, when the second user terminal (100B) operates as a sub-device, it can display an extended screen (300B) of the first screen (200A) of the first user terminal (100A).

[0139] Accordingly, the user can control the first pointer (400A) to move from the first screen (200A) to the extended screen (300B) using the first input device. Even when the first pointer (400A) is on the extended screen (300B), the user can still control the first pointer (400A) using the first input device to input his / her user command into the first user terminal (400A). For example, the user can select a desired graphic object by moving the first pointer (400A) onto any clickable graphic object (e.g., an icon, a menu, a link, a text input box, etc.) on the extended screen (300B) using the first input device.

[0140] The communication connection for transmitting the expanded screen (300B) of the first screen from the first user terminal (100A) described above to the second user terminal (100B) will be referred to as the first communication connection.

[0141] Hereinafter, with reference to FIGS. 4 and 5, a case in which a second user terminal (100B) operates as another main device will be described. FIGS. 4 and 5 are examples of a user terminal operating as a main device according to one aspect of the present disclosure.

[0142] As illustrated in FIG. 4, the first user terminal (100A) can display a first pointer (or cursor) (400A) on the first screen (200A), and as described above, the user can input his / her user command into the first user terminal (400A) by controlling the first pointer (400A) using the first input device. For example, the user can select a desired graphic object by moving the first pointer (400A) onto any clickable graphic object (e.g., an icon, a menu, a link, a text input box, etc.) on the first screen (200A) using the first input device.

[0143] Meanwhile, the second user terminal (100B) can display a second pointer (or cursor) (400B) on the second screen (200B). The second pointer (400B) may be generated by the second user terminal (100B) and controlled by a second input device (e.g., a second user input unit (123B)) of the second user terminal (100B). Examples of the second input device may include at least one of a keyboard, a mouse, a touchpad, etc. The second input device may be built into the second user terminal (100B) or may be externally connected to communicate with the second user terminal (100B). The user can control the second pointer (400B) using the second input device to input his / her user command into the second user terminal (400B). For example, a user can use a second input device to move a second pointer (400B) over any clickable graphical object (e.g., an icon, a menu, a link, a text input box, etc.) on a second screen (200B) to select a desired graphical object.

[0144] However, in this case, there is a disadvantage in that the second user terminal (100B) must also be equipped with a separate second input device. In addition, there is the inconvenience of having to use the first input device when the user inputs a user command to the first user terminal (100A), and having to use the second input device when the user inputs a user command to the second user terminal (100B). For example, when the user inputs a user command to the first user terminal (100A), the first pointer (400A) must be controlled using the first input device, and when the user inputs a user command to the second user terminal (100B), the second pointer (400B) must be controlled using the second input device.

[0145] To alleviate this inconvenience, it may be considered to allow a user to input user commands to a second user terminal (100B) using the first input device of the first user terminal (100A), even if the second user terminal (100B) operates as another main device. This will be described with reference to FIG. 5.

[0146] For example, as illustrated in FIG. 5, a first user terminal (100A) can generate a first screen (200A) and its extended screen (300B). The size of the extended screen (300B) of the first screen (200A) can be determined based on the display resolution in the display identification data received from the second user terminal (100B). The first user terminal (100A) can display the first screen (200A) and transmit the extended screen (300B) to the second user terminal (100B) via the first communication connection. However, since the second user terminal (100B) operates as a main device, it can display the second screen (200B) without displaying the extended screen (300B). The extended screen (300B) can correspond to a virtual screen for the second user terminal (100B).

[0147] As described above, the first user terminal (100A) can display a first pointer (or cursor) (400A) on the first screen (200A), and the user can input his / her user command to the first user terminal (400A) by controlling the first pointer (400A) using the first input device.

[0148] A user can control a first pointer (400A) to move from a first screen (200A) to a second screen (200B) (or the extended screen (300B)) using a first input device (e.g., a mouse, a touch pad, etc.). For example, if a user controls a first pointer (400A) to pass an edge area (E1) (e.g., a right edge area) of the first screen (200A) using the first input device, the first user terminal (100A) can control the first pointer (400A) to pass the edge area (E1) and disappear from the first screen (200A). In addition, the first user terminal (100A) can transmit information about the virtual coordinates of the first pointer (400A) within the virtual screen beyond the edge area of ​​the first screen (200A) to the second user terminal (100B). Since the virtual screen is determined based on the display resolution within the display identification data received from the second user terminal (100B), it can correspond to the second screen (200B) in terms of display coordinates. It can serve as a reference for the virtual coordinates of the virtual screen, i.e., a coordinate system.

[0149] Then, the second user terminal (100B) can receive information about the virtual coordinates of the first pointer (400A) and create and display the second pointer (400B) at the location of the second screen (200B) corresponding to the virtual coordinates of the first pointer (400A).

[0150] From a user perspective, the first pointer (400A) may appear to move to the right on the first screen (200A) and disappear as soon as it passes the right edge area (E1), and then the second pointer (400B) may appear on the left side of the second screen (200B) and move to the right.

[0151] A user can select (e.g., click) a desired graphic object by moving a second pointer (400B) onto any clickable graphic object (e.g., an icon, a menu, a link, a text input box, etc.) on a second screen (200B) using a first input device (123A) rather than a second input device (123B). When the first input device receives a command to select a graphic object (e.g., a click command) from the user while the second pointer (400B) is positioned on the graphic object on the second screen (200B), the first user terminal (100A) can transmit the selection command to the second user terminal (100B). The second user terminal (100B) can receive the selection command and perform an operation according to the selection command on the graphic object.

[0152] If the graphic object is a text input box, a typing command (e.g., a search word) input by a user through a first input device (e.g., a keyboard) after the selection command may be transmitted from the first user terminal (100A) to the second user terminal (100B). The second user terminal (100B) may receive the typing command and perform an operation according to the typing command for the text input box. For example, if the text input box is a search word input box, the second user terminal (100B) may perform a search operation corresponding to the search word.

[0153] At least one of the information regarding the virtual coordinates, the selection command, and the typing command described above may be transmitted via a second communication connection separate from the first communication connection between the first user terminal (100A) and the second user terminal (100B). Hereinafter, the establishment of the second communication connection between the first user terminal (100A) and the second user terminal (100B) will be described with reference to FIG. 6 . FIG. 6 illustrates a communication connection process between a plurality of user terminals according to one aspect of the present disclosure.

[0154] Hereinafter, for the sake of simplicity, the first user terminal (100A) and the second user terminal (100B) are described as being the subjects of the operation. However, in reality, at least one component of the first user terminal (100A) (e.g., the first control unit (180A)) and at least one component of the second user terminal (100B) (e.g., the second control unit (180B)) may be the subjects of the operation.

[0155] Let us assume that the first user terminal (100A) and the second user terminal (100B) are connected to the same wireless network. For example, the first user terminal (100A) and the second user terminal (100B) may be connected to the same access point. Alternatively, one of the first user terminal (100A) and the second user terminal (100B) may be connected to a soft AP (Access Point) provided by the other. A soft AP refers to a user terminal that implements the role of a wireless access point in software so that it can also function as another access point for other terminals and / or other home appliances.

[0156] The first user terminal (100A) may be a personal computer, such as a laptop, and the second user terminal (100B) may be a smart monitor, but is not limited thereto. Furthermore, the first user terminal (100A) may be a smart monitor, and the second user terminal (100B) may be a personal computer, such as a laptop.

[0157] First, a first communication connection can be established between a first user terminal (100A) and a second user terminal (100B) [S601]. For example, a wired communication connection can be established between the first user terminal (100A) and the second user terminal (100B) by one of the following methods (but not limited to): HDMI, USB, Thunderbolt, etc. Alternatively, a wired communication connection can be established between the first user terminal (100A) and the second user terminal (100B) by one of the following methods (but not limited to): Bluetooth, UWB (Ultra-wideband), Miracast, Airplay, etc.

[0158] When a first communication connection is established between a first user terminal (100A) and a second user terminal (100B), the first user terminal (100A) can receive display identification data (EDID: Extended display identification data) from the second user terminal (100B) [S603]. The display identification data may include display specification information regarding at least one of the manufacturer, model name, product unique identifier, display size, display resolution, etc. of the user terminal. The product unique identifier may include, for example, a product serial number.

[0159] The first user terminal (100A) can extract a product unique identifier from the display identification data of the second user terminal (100B) [S605]. The product unique identifier extracted from the display identification data will be referred to as a first product unique identifier.

[0160] Meanwhile, the first user terminal (100A) can search for at least one other device, including the second user terminal (100B), within the wireless network [S607]. For example, when the first user terminal (100A) connects to the wireless network, it can search for other devices within the wireless network based on UPnP (Universal Plug and Play) and / or SSDP (Simple Service Discovery Protocol).

[0161] The first user terminal (100A) can receive its device information from at least one other device, including the second user terminal (100B) [S609]. The device information may include an IP address and a product unique identifier.

[0162] The first user terminal (100A) can extract an IP address and a product unique identifier from device information [S611]. The product unique identifier extracted from the device information will be referred to as a second product unique identifier.

[0163] The steps S607 to S611 may be performed before the steps S601 to S605, or the steps S607 to S611 may be performed in parallel with the steps S601 to S605.

[0164] The first user terminal (100A) can compare the first product unique identifier with the second product unique identifier and match the extracted IP address with the second product unique identifier that is identical to the first product unique identifier to the second user terminal (100B) [S613]. That is, the first user terminal (100A) can know that the extracted IP address is the one assigned to the second user terminal (100B) within the wireless network.

[0165] The first user terminal may immediately establish a second communication connection using the extracted IP code, but may perform PIN code authentication as follows to increase the reliability of the second communication connection.

[0166] That is, the first user terminal (100A) can generate a PIN code for the second user terminal (100B) and transmit the PIN code to the second user terminal (100B) using the extracted IP information within the wireless network [S615].

[0167] And, the first user terminal (100A) can display an input box for the user to enter a PIN code [S617].

[0168] The second user terminal (100B) can receive and display the PIN code [S619].

[0169] The user can view the PIN code displayed on the second user terminal (100B) and input the PIN code into the input box displayed on the first user terminal (100A) [S621].

[0170] The first user terminal (100A) can retransmit the entered PIN code to the second user terminal (100B) [S623].

[0171] The second user terminal (100B) can perform PIN code authentication using the PIN code received again from the first user terminal (100A) [S625]. For example, the second user terminal (100B) can determine whether the PIN code displayed in step S619 and the PIN code received in step S623 are the same.

[0172] If these are identical, the second user terminal (100B) can generate a client key code for automatic connection and transmit it to the first user terminal (100A) [S627].

[0173] The first user terminal (100A) can receive the client key code and store it for future automatic connection with the second user terminal (100B) [S629].

[0174] The first user terminal (100A) can establish a second communication connection with the second user terminal (100B) on the wireless network using the stored client key code [S631]. The second communication connection may be a Wi-Fi communication connection.

[0175] Even if the second communication connection with the second user terminal (100B) is disconnected in the future, the first user terminal (100A) can immediately establish the second communication connection with the second user terminal (100B) on the wireless network by performing only the step S631 using the stored client key code without performing the steps S601 to S629 again.

[0176] The first user terminal (100A) and the second user terminal (100B) can communicate with each other on the wireless network using a secure web socket method as a second communication connection.

[0177] It goes without saying that the first user terminal (100A) can also establish the first communication connection and the second communication connection with each of at least one other user terminal as described above while maintaining the first communication connection and the second communication connection with the second user terminal (100B).

[0178] Hereinafter, with reference to FIG. 7, a description will be given of how the first user terminal (100A) controls all transitions between the first mode and the second mode of the second user terminal (100B) when the first communication connection and the second communication connection are established between the first user terminal (100A) and the second user terminal (100B). FIG. 7 illustrates a mode transition process of a user terminal according to one aspect of the present disclosure.

[0179] The first user terminal (100A) can display the first screen (200A) while operating as a main device [S701].

[0180] Meanwhile, the second user terminal (100B) can operate in the second mode and act as a sub-device of the first user terminal (100A) [S703]. Accordingly, the second user terminal (100B) can receive and display an extended screen (300B) of the first screen (200A) from the first user terminal (100A) [S703].

[0181] The first user terminal (100A) can subscribe to mode status notification to the second user terminal (100B) [S705].

[0182] In response to receiving the mode status alarm subscription, the second user terminal (100B) may periodically and / or whenever its mode is switched, transmit a mode status notification to the first user terminal (100A) about what its current mode is [S707].

[0183] The first user terminal (100A) can know the current mode of the second user terminal (100B) by receiving a mode status notification from the second user terminal (100B).

[0184] In FIG. 7, the second user terminal (100B) is illustrated as performing step S707 in response to a mode status notification subscription from the first user terminal (100A). However, the second user terminal (100B) may be configured to perform step S707 by default when a second communication connection is established with the first user terminal (100A).

[0185] The first user terminal (100A) can receive a mode switching command of the second user terminal (100B) from the user through the first input device (123A) [S709].

[0186] The first user terminal (100A) can transmit a mode switching command to the second user terminal (100B) [S711].

[0187] The second user terminal (100B) can operate as another main device by switching from the second mode to the first mode in response to a mode switching command received from the first user terminal (100A) [S713]. Accordingly, the second user terminal (100B) can display the second screen (200B).

[0188] The first user terminal (100A) can transmit a request for creating an input communication channel to the second user terminal (100B) to transmit a user command input through the first input device (123A) of the first user terminal (100A) to the second user terminal (100B) to control the second user terminal (100B) operating in the first mode [S715].

[0189] The second user terminal (100B) can create an input communication channel in response to a request for creating an input communication channel from the first user terminal (100A) and transmit information of the input communication channel to the first user terminal (100A) [S717, S719].

[0190] The first user terminal (100A) can receive a user command for the second user terminal (100B) via the first input device (123A) [S721]. The user command for the second user terminal (100B) can include at least one of the aforementioned virtual coordinate information, selection command, typing command, etc.

[0191] The first user terminal (100A) can transmit a user command to the second user terminal (100B) through an input communication channel [S723].

[0192] The second user terminal (100B) can perform an operation according to a user command [S725].

[0193] The first user terminal (100A) can receive a mode switching command of the second user terminal (100B) from the user through the first input device (123A) [S727].

[0194] The first user terminal (100A) can transmit a mode switching command to the second user terminal (100B) [S729].

[0195] The second user terminal (100B) can operate as a sub-device of the first user terminal (100A) by switching from the first mode to the second mode in response to a mode switching command received from the first user terminal (100A) [S703].

[0196] Steps S705, S707, S711, S715, S719, S723, and S709 in FIG. 7 may be performed through a second communication connection between the first user terminal (100A) and the second user terminal (100B), but at least one of them may also be performed through the first communication connection between the first user terminal (100A) and the second user terminal (100B).

[0197] In the above, the case where the first user terminal (100A) is connected to a first communication connection and a second communication connection with another user terminal, i.e., the second user terminal (100B), which can be switched between the first mode and the second mode, has been described. Hereinafter, the case where the first user terminal (100A) is connected to a first communication connection and a second communication connection with a plurality of other user terminals which can be switched between the first mode and the second mode will be described further with reference to FIGS. 8 and 9. FIGS. 8 and 9 illustrate a plurality of user terminals that are connected to each other in communication according to one aspect of the present disclosure.

[0198] In FIGS. 8 and 9, it is assumed that a first user terminal (100A) is connected to three other user terminals, namely a second user terminal (100B), a third user terminal (100C), and a fourth user terminal (100D), through a first communication connection and a second communication connection, respectively, which can be switched between a first mode and a second mode. It can be understood that the first user terminal (100A) acts as a host device, and the second user terminal (100B), the third user terminal (100C), and the fourth user terminal (100D) act as client devices.

[0199] Let us assume that the first user terminal (100A), the second user terminal (100B), the third user terminal (100C), and the fourth user terminal (100D) are arranged from left to right. The relative positions of the second user terminal (100B), the third user terminal (100C), and the fourth user terminal (100D) with respect to the first user terminal (100A) can be set by the user at the first user terminal (100A).

[0200] As illustrated in FIG. 8, the first user terminal (100A) can generate a first screen (200A), its first extended screen (300B), its second extended screen (300C), and its third extended screen (300D). Since the first user terminal (100A), the second user terminal (100B), the third user terminal (100C), and the fourth user terminal (100D) are arranged from left to right, the first screen (200A), the first extended screen (300B), the second extended screen (300C), and the third extended screen (300D) can also be generated to be arranged from left to right accordingly.

[0201] The size of the first extended screen (300B) may be determined based on the display resolution in the display identification data received from the second user terminal (100B), the size of the second extended screen (300C) may be determined based on the display resolution in the display identification data received from the third user terminal (100C), and the size of the third extended screen (300D) may be determined based on the display resolution in the display identification data received from the fourth user terminal (100D).

[0202] The first user terminal (100A) can display the first screen (200A).

[0203] A first user terminal (100A) can transmit a first extended screen (300B) to a second user terminal (100B) via a first communication connection between the first user terminal (100A) and the second user terminal (100B). The first extended screen (300B) may correspond to a first virtual screen for the second user terminal (100B).

[0204] A first user terminal (100A) can transmit a second extended screen (300C) to a third user terminal (100C) via a first communication connection between the first user terminal (100A) and the third user terminal (100C). The second extended screen (300C) may correspond to a second virtual screen for the third user terminal (100C).

[0205] The first user terminal (100A) can transmit the third extended screen (300D) to the fourth user terminal (100D) via the first communication connection between the first user terminal (100A) and the fourth user terminal (100D). The third extended screen (300D) may correspond to a third virtual screen for the fourth user terminal (100D).

[0206] FIGS. 8 and 9 illustrate that the second user terminal (100B) operates as the main device. Accordingly, the second user terminal (100B) can display the second screen (200B) without displaying the first extended screen (300B). The second screen (200B) may be a unique screen based on the operating system of the second user terminal (100B).

[0207] In FIGS. 8 and 9, it is illustrated that the third user terminal (100B) operates as a sub-device of the first user terminal (100A). Accordingly, the third user terminal (100C) can display the second extended screen (300C).

[0208] In FIGS. 8 and 9, it is illustrated that the fourth user terminal (100D) operates as the main device. Accordingly, the fourth user terminal (100D) can display the fourth screen (200D) without displaying the third extended screen (300C). The fourth screen (200D) may be a self-contained screen according to the operating system of the fourth user terminal (100D).

[0209] As described above, the first user terminal (100A) can display a first pointer (or cursor) (400A) on the first screen (200A), and the user can input his / her user command into the first user terminal (400A) by controlling the first pointer (400A) using the first input device.

[0210] As described in FIG. 5, a user can control a second user terminal (100B) by using a first input device (123A) of a first user terminal (100A) to display a second pointer (400B) on a second screen (200B) of a second user terminal (100B).

[0211] Hereinafter, with reference to FIG. 8, we will look at a case where a user operates the first input device (123A) so that the first pointer (400A) on the first screen (200A) moves to the second extended screen (300C).

[0212] A user can control a first pointer (400A) to move from a first screen (200A) to a second extended screen (or second virtual screen) (300C) via a first extended screen (or first virtual screen) (300B) using a first input device (e.g., a mouse, a touch pad, etc.).

[0213] For example, when a user controls a first pointer (400A) to pass through a first edge area (E1) between a first screen (200A) and a first extended screen (300B) and a second edge area (E2) between the first extended screen (300B) and a second extended screen (300C) using a first input device, the first user terminal (100A) can control the first pointer (400A) to pass through the edge area (E1) and disappear from the first screen (200A).

[0214] As described above in FIG. 5, while the first pointer (400A) passes over the first extended screen (300B), the second pointer (400B) can be displayed on the second screen (200B) of the second user terminal (100B) so as to correspond to the virtual coordinates of the first virtual screen (300B) of the first pointer (400A) instead of the first pointer (400A). Accordingly, while the first pointer (400A) passes over the first extended screen (300B), the second pointer (400B) can be displayed so as to pass over the second screen (200B) of the second user terminal (100B) so as to correspond thereto.

[0215] When the first pointer (400A) is positioned within the second extended screen (300C), the first user terminal (100A) transmits the second extended screen (300C) on which the first pointer (400A) is displayed to the third user terminal (100C), and the third user terminal (100C) can display the second extended screen (300C) on which the first pointer (400A) is displayed.

[0216] Hereinafter, with reference to FIG. 9, we will look at a case where a user operates the first input device (123A) so that the first pointer (400A) on the first screen (200A) moves to the third extended screen (300D).

[0217] A user can control a first pointer (400A) to move from a first screen (200A) to a first extended screen (or first virtual screen) (300B) and a second extended screen (or second virtual screen) (300C) sequentially to a third extended screen (or third virtual screen) (300D) by using a first input device (e.g., a mouse, a touch pad, etc.).

[0218] For example, when a user controls a first pointer (400A) to pass through a first edge area (E1) between a first screen (200A) and a first extended screen (300B), a second edge area (E2) between a first extended screen (300B) and a second extended screen (300C), and a third edge area (E3) between a second extended screen (300C) and a third extended screen (300D) using a first input device, the first user terminal (100A) can control the first pointer (400A) to pass through the edge area (E1) and disappear from the first screen (200A).

[0219] As described above in FIG. 5, while the first pointer (400A) passes over the first extended screen (300B), the second pointer (400B) can be displayed on the second screen (200B) of the second user terminal (100B) so as to correspond to the virtual coordinates of the first virtual screen (300B) of the first pointer (400A) instead of the first pointer (400A). Accordingly, while the first pointer (400A) passes over the first extended screen (300B), the second pointer (400B) can be displayed so as to pass over the second screen (200B) of the second user terminal (100B) so as to correspond thereto.

[0220] As described above in FIG. 8, while the first pointer (400A) passes over the second extended screen (300C), the first pointer (400A) can be displayed on the second extended screen (300C) of the third user terminal (100C). Accordingly, the first pointer (400A) can be displayed so as to pass over the second extended screen (300C) of the third user terminal (100C).

[0221] When the first pointer (400A) is positioned within the third extended screen (300D), the first user terminal (100A) can transmit information regarding the virtual coordinates of the first pointer (400A) within the third virtual screen (300D) to the fourth user terminal (100D). Since the third virtual screen (300D) is determined based on the display resolution within the display identification data received from the fourth user terminal (100D), it can correspond to the fourth screen (200D) in terms of display coordinates.

[0222] Then, the fourth user terminal (100D) can receive information about the virtual coordinates of the first pointer (400A) and create and display the second pointer (400B) at the location of the fourth screen (200D) corresponding to the virtual coordinates of the first pointer (400A).

[0223] From a user perspective, the first pointer (400A) may appear to move to the right on the second extended screen (300C) and disappear as soon as it passes the third edge area (E3), and then the second pointer (400B) may appear on the left side of the fourth screen (200D) and move to the right.

[0224] As described in FIG. 5, a user can control a second pointer (400B) in a fourth screen (200D) through a first input device (123A) of a first user terminal (100A), and control a fourth user terminal (100D) through the first input device (123A) and the second pointer (400B).

[0225] In the above, when the first user terminal (100A) and the second user terminal (100B) are connected to the same wireless network (e.g., a Wi-Fi network), the transmission of a user command received through the first input device (123A) of the first user terminal (100A) to the second user terminal (100B) has been described.

[0226] Hereinafter, when the first user terminal (100A) and the second user terminal (100B) are not connected to the same wireless network, a method of transmitting a user command received through the first input device (123A) of the first user terminal (100A) to the second user terminal (100B) will be described.

[0227] First, referring to FIG. 10, a second communication connection between a first user terminal (100A) and a second user terminal (100B) that does not use a wireless network (e.g., a Wi-Fi network) will be described. FIG. 10 illustrates a communication connection process between multiple user terminals according to one aspect of the present disclosure.

[0228] In Fig. 10, the explanation will be given assuming that both the first user terminal (100A) and the second user terminal (100B) have the Bluetooth communication function activated.

[0229] The first control unit (180A) of the first user terminal (100A) can execute an HID sharing application to enable its first input device (123A) to be used for another user terminal [S1001]. The HID sharing application can be executed in response to a user command via the first input device (123A), and can also be executed automatically when the first user terminal (100A) is powered on.

[0230] If the Bluetooth communication function is not activated in the first user terminal (100A), the first control unit (180A) of the first user terminal (100A) can automatically activate the Bluetooth communication function as the HID sharing application is executed.

[0231] The second control unit (180B) of the second user terminal (100B) can control to transmit an SDP (Service Discovery Protocol) request signal to at least one surrounding Bluetooth device including the first user terminal (100A) [S1003].

[0232] The first control unit (180A) of the first user terminal (100A) can receive an SDP request signal and control to transmit an SDP response signal to the SDP request signal to the second user terminal (100B) [S1005].

[0233] SDP (Service Discovery Protocol) is a protocol used to exchange service information between Bluetooth devices (e.g., client devices and server devices). For example, a client device can obtain information about which Bluetooth services a server device provides via SDP.

[0234] Accordingly, the second control unit (180B) of the second user terminal (100B) can recognize the first user terminal (100A) by receiving the SDP response signal and can obtain information on the Bluetooth service provided by the first user terminal (100A) [S1007].

[0235] The second control unit (180B) of the second user terminal (100B) can control the transmission of a Bluetooth pairing request signal to the first user terminal (100A) [S1009].

[0236] The first control unit (180A) of the first user terminal (100A) can receive a Bluetooth pairing request signal and control to transmit a Bluetooth pairing response signal to the Bluetooth pairing request signal to the second user terminal (100B) [S1011].

[0237] Through this, Bluetooth pairing can be completed between the first user terminal (100A) and the second user terminal (100B) [S1013].

[0238] When Bluetooth pairing is completed between the first user terminal (100A) and the second user terminal (100B), the second control unit (180B) of the second user terminal (100B) can open an L2CAP (Logical Link Control and Adaption Protocol) channel for the first user terminal (100A) [S1015]. The L2CAP channel can be used as the second communication connection described above.

[0239] In the above, it has been described that Bluetooth pairing is established between the first user terminal (100A) and the second user terminal (100B) by the second user terminal (100B) transmitting an SDP request signal and a Bluetooth pairing request signal to the first user terminal (100A). However, Bluetooth pairing may also be established between the first user terminal (100A) and the second user terminal (100B) by the first user terminal (100A) transmitting an SDP request signal and a Bluetooth pairing request signal to the second user terminal (100B). This will be described with reference to FIG. 11. FIG. 11 illustrates a communication connection process between a plurality of user terminals according to one aspect of the present disclosure.

[0240] In Fig. 11, the explanation will be given assuming that both the first user terminal (100A) and the second user terminal (100B) have the Bluetooth communication function activated.

[0241] The first control unit (180A) of the first user terminal (100A) can execute an HID sharing application to enable the first input device (123A) of the asset to be used for another user terminal [S1001].

[0242] The second control unit (180B) of the second user terminal (100B) can control to transmit a first SDP (Service Discovery Protocol) request signal to at least one surrounding Bluetooth device including the first user terminal (100A) [S1003].

[0243] The first control unit (180A) of the first user terminal (100A) can receive an SDP request signal and control to transmit a first SDP response signal to the first SDP request signal to the second user terminal (100B) [S1005].

[0244] The second control unit (180B) of the second user terminal (100B) can recognize the first user terminal (100A) by receiving the first SDP response signal and can obtain information on the Bluetooth service provided by the first user terminal (100A) [S1007].

[0245] Steps S1001 to S1007 are as described above.

[0246] The first control unit (180A) of the first user terminal (100A) that transmitted the first SDP response signal can transmit the second SDP request signal to the second user terminal (100B) [S1101].

[0247] The second control unit (180B) of the second user terminal (100B) can receive a second SDP request signal and control to transmit a second SDP response signal to the second SDP request signal to the first user terminal (100A) [S1103].

[0248] The first control unit (180A) of the first user terminal (100A) can recognize the second user terminal (100B) by receiving the second SDP response signal and can determine identification information (e.g., manufacturer and / or model name information) of the second user terminal (100B) [S1105].

[0249] In FIG. 11, steps S1101 to S1105 are illustrated as being executed after steps S1001 to S1007. However, the present disclosure is not limited thereto. For example, steps S1101 to S1105 may be executed before steps S1001 to S1007, or steps S1101 to S1105 and steps S1001 to S1007 may be executed substantially simultaneously.

[0250] The first control unit (180A) of the first user terminal (100A) can determine whether the first input device (123A) of the first user terminal (100A) can be used for the second user terminal (100B) based on the identification information of the second user terminal (100B) [S1107]. For example, the first memory (170) of the first user terminal (100A) can store information on whether the first input device (123A) can be shared by manufacturer and / or model, and the first control unit (180A) of the first user terminal (100A) can determine whether the first input device (123A) can be used for the second user terminal (100B) by searching the identification information of the second user terminal (100B) in the first memory (170).

[0251] If it is determined that the first input device (123A) of the first user terminal (100A) cannot be used for the second user terminal (100B), the first control unit (180A) of the first user terminal (100A) can control not to transmit a Bluetooth pairing request signal to the second user terminal (100B) [S1109].

[0252] If it is determined that the first input device (123A) of the first user terminal (100A) can be used for the second user terminal (100B), the first control unit (180A) of the first user terminal (100A) can control to transmit a Bluetooth pairing request signal to the second user terminal (100B) [S1111].

[0253] The second control unit (180B) of the second user terminal (100B) can receive a Bluetooth pairing request signal and control to transmit a Bluetooth pairing response signal to the Bluetooth pairing request signal to the first user terminal (100A) [S1113].

[0254] Through this, Bluetooth pairing can be completed between the first user terminal (100A) and the second user terminal (100B) [S1115].

[0255] When Bluetooth pairing is completed between a first user terminal (100A) and a second user terminal (100B), the first control unit (180A) of the first user terminal (100A) can open an L2CAP (Logical Link Control and Adaption Protocol) channel for the second user terminal (100B) [S1117]. The L2CAP channel can be used as the second communication connection described above.

[0256] Hereinafter, with reference to FIGS. 12 and 13, a description will be given of how the first user terminal (100A) controls all transitions between the first mode and the second mode of the second user terminal (100B) when the first communication connection and the second communication connection are established between the first user terminal (100A) and the second user terminal (100B). FIG. 12 illustrates a mode transition process of the user terminal according to one aspect of the present disclosure. FIG. 13 illustrates a first user terminal and a second user terminal that are communicatively connected to each other according to one aspect of the present disclosure.

[0257] The first user terminal (100A) can display the first screen (200A) while operating as a main device, as shown in (13-1) of FIG. 13 [S1201].

[0258] Meanwhile, the second user terminal (100B) can operate in the second mode and operate as a sub-device of the first user terminal (100A) [S1203]. Accordingly, the second user terminal (100B) can receive and display an extended screen (300B) of the first screen (200A) from the first user terminal (100A), as illustrated in (13-1) of FIG. 13 [S1203].

[0259] The second user terminal (100B) can periodically and / or whenever its mode is switched, transmit a mode status notification to the first user terminal (100A) regarding its current mode [S1207]. The first user terminal (100A) can know the current mode of the second user terminal (100B) by receiving the mode status notification from the second user terminal (100B).

[0260] If the mode switching of the second user terminal (100B) is performed by transmitting a mode switching command input through the first input device (123A) of the first user terminal (100A) to the second user terminal (100B) through the first communication connection or the second communication connection, the first user terminal (100A) can know the current mode of the second user terminal (100B) even if it does not receive a mode status notification from the second user terminal (100B). In this case, step S1207 may be omitted. This can also be applied to the following description.

[0261] In Fig. 12, it is illustrated that the second user terminal (100B) performs step S1207 by default when a second communication connection is established with the first user terminal (100A). However, when the second user terminal (100B) establishes a second communication connection with the first user terminal (100A), the second user terminal (100B) may be configured to receive a mode status notification subscription from the first user terminal (100A), and the second user terminal (100B) may perform step S1207 in response to the mode status notification subscription.

[0262] The first control unit (180A) of the first user terminal (100A) can control the first pointer (400A) to be displayed on the first screen (200A) or its extended screen (200B), as shown in (13-1) of FIG. 13, when the second user terminal (100B) operates in the second mode [S1208]. The user can move the first pointer (400A) between the first screen (200A) and its extended screen (200B) by manipulating the first input device (123A) of the first user terminal (100A).

[0263] The first user terminal (100A) can receive a first user command through the first input device (123A) of the first user terminal (100A) [S1209]. The first user command may be, for example, selecting (e.g., clicking) a desired graphic object (e.g., an icon, a menu, a link, a text input box, etc.) by moving the first pointer (400A) onto a desired clickable graphic object (e.g., an icon, a menu, a link, a text input box, etc.) on the first screen (200A) or its extended screen (200B) using the first input device (123A).

[0264] If the graphic object is a text input box, the first user command may include a typing command (e.g., entering a search word) entered by the user via a first input device (e.g., a keyboard) after the selection command.

[0265] The first control unit (180A) of the first user terminal (100A) can execute an operation according to the first user command [S1211]. For example, if the graphic object is a search word input box, the first control unit (180A) of the first user terminal (100A) can perform a search operation corresponding to the search word and control the display of the search result on at least one of the first screen (200A) and its extended screen (200B).

[0266] While the second user terminal (100B) is operating in the second mode, the first user terminal (100A) can receive a first mode switching command through the first input device (123A) of the first user terminal (100A) [S1213]. The first mode switching command may be, for example, a first shortcut key input received through the first input device (123).

[0267] The first user terminal (100A) can transmit the first mode switching command to the second user terminal (100B) via the first communication connection or the second communication connection [S1215].

[0268] The second user terminal (100B) can operate as another main device by switching from the second mode to the first mode in response to the first mode switching command received from the first user terminal (100A) [S1217]. Accordingly, as illustrated in (13-2) of FIG. 13, the first user terminal (100A) can operate as a main device and display the first screen (200A), while the second user terminal (100B) can operate as another main device and display the second screen (200B).

[0269] The second control unit (180B) of the second user terminal (100B) can control the second pointer (400B) to be displayed on the second screen (200B) [S1219]. That is, while the second user terminal (100B) operates in the first mode, the first pointer (400A) may not be displayed on the first screen (200A) and the second pointer (400B) may be displayed on the second screen (200B).

[0270] The first user terminal (100A) can receive a second user command through the first input device (123A) of the first user terminal (100A) [S1221]. The second user command may include at least one of pointer display information (e.g., pointer position information, pointer movement information, and / or pointer virtual coordinate information), a selection command, a typing command, etc.

[0271] The second user command may be, for example, moving the second pointer (400B) using the first input device (123A) over a desired clickable graphical object (e.g., an icon, menu, link, text input box, etc.) on the second screen (200B) to select (e.g., click) the desired graphical object.

[0272] If the graphic object is a text input box, the second user command may include a typing command (e.g., entering a search word) entered by the user via the first input device (e.g., a keyboard) after the selection command.

[0273] Additionally, the second user command may further include pointer display information regarding the display position (or movement direction) of the second pointer (400B) on the second screen (200B).

[0274] The first user terminal (100A) can transmit the second user command to the second user terminal (100B) via the second communication connection [S1223]. That is, the first control unit (180A) of the first user terminal (100A) can control the second user terminal (100B) to only transmit the second user command to the second user terminal (100B) and not to perform an operation according to the second user command itself. For example, if the second user command is a command regarding a search, the first user terminal (100A) can only transmit the second user command to the second user terminal (100B) and not perform a search operation.

[0275] The second user terminal (100B) can receive a second user command and perform an operation according to the second user command [S1225].

[0276] The second control unit (180B) of the second user terminal (100B) can control the display of the second pointer (400B) on the second screen (200B) to correspond to the pointer display information of the second user command, as shown in (13-2) of FIG. 13.

[0277] For example, if the graphic object is a search word input box, the second control unit (180B) of the second user terminal (100B) can perform a search operation corresponding to the search word according to a second user command and control the search result to be displayed on the second screen (200B).

[0278] While the second user terminal (100B) is operating in the first mode, the first user terminal (100A) can receive a second mode switching command through the first input device (123A) of the first user terminal (100A) [S1227]. The second mode switching command may be, for example, a second shortcut key input received through the first input device (123). The second shortcut key input may be the same as the first shortcut key input (i.e., toggle switching) or may be different. The first shortcut key input and the second shortcut key input are merely examples, and it is to be understood that other inputs may be used as the first mode switching command or the second mode switching command.

[0279] The first user terminal (100A) can transmit a second mode switching command to the second user terminal (100B) via the first communication connection or the second communication connection [S1229].

[0280] The second user terminal (100B) can operate as a sub-device by switching from the first mode to the second mode in response to a second mode switching command received from the first user terminal (100A) [S1203].

[0281] While the second user terminal (100B) operates in the second mode, the second pointer (400B) is not displayed and the first pointer (400A) may be displayed on the first screen (200A) or its extended screen (200B).

[0282] In FIG. 12, it was explained that when the second user terminal (100B) operates in the first mode, the first user terminal (100A) only transmits the user command input through the first input device (123A) to the second user terminal (100B) and does not execute the user command itself. However, even if the second user terminal (100B) operates in the first mode, in some cases, the first user terminal (100A) may need to directly execute the user command input through the first input device (123A) without transmitting it to the second user terminal (100B). This will be further described with reference to FIGS. 14 and 15. FIG. 14 is a control flowchart of a first user terminal and a second user terminal that are communicatively connected to each other according to one aspect of the present disclosure. FIG. 15 illustrates a first user terminal and a second user terminal that are communicatively connected to each other according to one aspect of the present disclosure.

[0283] As illustrated in FIG. 15, the first user terminal (100A) can display the first screen (200A) [S1201], and the second user terminal (100B) can operate in the first mode to display the second screen (200B) [S1217].

[0284] The first control unit (180A) of the first user terminal (100A) can control the first pointer (400A) to be displayed on the first screen (200A), as shown in (15-1) of FIG. 15, even if the second user terminal (100B) operates in the first mode [S1208].

[0285] The first user terminal (100A) can receive a first user command through the first input device (123A) of the first user terminal (100A) [S1209]. The first user command may be, for example, selecting (e.g., clicking) a desired graphic object (e.g., an icon, a menu, a link, a text input box, etc.) on the first screen (200A) by moving the first pointer (400A) using the first input device (123A) onto the desired clickable graphic object.

[0286] If the graphic object is a text input box, the first user command may include a typing command (e.g., entering a search word) entered by the user via a first input device (e.g., a keyboard) after the selection command.

[0287] The first control unit (180A) of the first user terminal (100A) can execute an operation according to the first user command [S1211]. For example, if the graphic object is a search word input box, the first control unit (180A) of the first user terminal (100A) can perform a search operation corresponding to the search word and control the display of the search result on at least one of the first screen (200A) and its extended screen (200B).

[0288] The first user terminal (100A) can receive a first control target change command through the first input device (123A) of the first user terminal (100A) [S1401]. The control target change command may be, for example, a third shortcut key input or a fourth shortcut key input received through the first input device (123). In the case of the third shortcut key input, the control target may be set to the first user terminal (100A), and in the case of the fourth shortcut key input, the control target may be set to the second user terminal (100B). Alternatively, each time the third shortcut key is input, the control target may be alternately selected as one of the first user terminal (100A) and the second user terminal (100B). The third shortcut key input and the fourth shortcut key input are merely examples, and it is to be understood that other inputs may also be used for the control target selection command.

[0289] Then, the first control unit (180A) of the first user terminal (100A) can control to transmit a control target selection notification to the second user terminal (100B) through the first communication connection or the second communication connection [S1403]. The control target selection notification is for the first user terminal (100A) to inform the second user terminal (100B) that the second user terminal (100B) has become the control target of the first input device (123A).

[0290] And, the first control unit (180A) of the first user terminal (100A) can control the first pointer (400A) to disappear from the first screen (200A) [S1405].

[0291] The second control unit (180B) of the second user terminal (100B) can control the second pointer (400B) to be displayed on the second screen (200B) in response to the control target selection notification [S1219].

[0292] In FIG. 14, steps S1405 and S1219 are illustrated as being performed after step S1403. However, steps S1403 and S1219 may be performed after step S1405. Alternatively, steps S1403 and S1405 may be performed substantially simultaneously.

[0293] The first user terminal (100A) can receive a second user command through the first input device (123A) of the first user terminal (100A) [S1221]. The second user command may include at least one of pointer display information (e.g., pointer position information, pointer movement information, and / or pointer virtual coordinate information), a selection command, a typing command, etc.

[0294] The second user command may be, for example, moving the second pointer (400B) using the first input device (123A) over a desired clickable graphical object (e.g., an icon, menu, link, text input box, etc.) on the second screen (200B) to select (e.g., click) the desired graphical object.

[0295] If the graphic object is a text input box, the second user command may include a typing command (e.g., entering a search word) entered by the user via the first input device (e.g., a keyboard) after the selection command.

[0296] Additionally, the second user command may further include pointer display information regarding the display position (or movement direction) of the second pointer (400B) on the second screen (200B).

[0297] The first user terminal (100A) can transmit the second user command to the second user terminal (100B) via the second communication connection [S1223]. That is, the first control unit (180A) of the first user terminal (100A) can control the second user terminal (100B) to only transmit the second user command to the second user terminal (100B) and not to perform an operation according to the second user command itself. For example, if the second user command is a command regarding a search, the first user terminal (100A) can only transmit the second user command to the second user terminal (100B) and not perform a search operation.

[0298] The second user terminal (100B) can receive a second user command and perform an operation according to the second user command [S1225].

[0299] The second control unit (180B) of the second user terminal (100B) can control the display of the second pointer (400B) on the second screen (200B) to correspond to the pointer display information of the second user command, as shown in (15-2) of FIG. 15.

[0300] For example, if the graphic object is a search word input box, the second control unit (180B) of the second user terminal (100B) can perform a search operation corresponding to the search word according to a second user command and control the search result to be displayed on the second screen (200B).

[0301] The first user terminal (100A) can receive a second control target change command through the first input device (123A) of the first user terminal (100A) [S1409].

[0302] Then, the first control unit (180A) of the first user terminal (100A) can control to transmit a control target release notification to the second user terminal (100B) through the first communication connection or the second communication connection [S1411]. The control target release notification is for the first user terminal (100A) to inform the second user terminal (100B) that the second user terminal (100B) is no longer a control target of the first input device (123A).

[0303] The second control unit (180B) of the second user terminal (100B) can control the second pointer (400B) to disappear from the second screen (200B) in response to a control target release notification, as shown in (15-1) of FIG. 15 [S1413].

[0304] And, the first control unit (180A) of the first user terminal (100A) can control the first pointer (400A) to be displayed on the first screen (200A), as shown in (15-1) of FIG. 15 [S1208].

[0305] In FIG. 14, steps S1413 and S1208 are depicted as being performed after step S1411. However, steps S1411 and S1413 may be performed after step S1208. Alternatively, steps S1411 and S1208 may be performed substantially simultaneously.

[0306] In the above, it has been described that the control target of the user command input through the first input device (123A) of the first user terminal (100A) is determined through the shortcut key operation of the first input device (123A). However, the control target of the user command input through the first input device (123A) of the first user terminal (100A) may also be determined through the movement of a pointer (e.g., the first pointer (400A) or the second pointer (400B)). This will be further described with reference to FIGS. 16 and 17. FIG. 16 is a control flowchart of a first user terminal and a second user terminal that are communicatively connected to each other according to one aspect of the present disclosure. FIG. 17 illustrates a first user terminal and a second user terminal that are communicatively connected to each other according to one aspect of the present disclosure.

[0307] As illustrated in FIG. 17, the first user terminal (100A) can display the first screen (200A) [S1201], and the second user terminal (100B) can operate in the first mode to display the second screen (200B) [S1217].

[0308] The first control unit (180A) of the first user terminal (100A) can control the first pointer (400A) to be displayed on the first screen (200A) even when the second user terminal (100B) operates in the first mode [S1208].

[0309] The first user terminal (100A) can receive a first user command through the first input device (123A) of the first user terminal (100A) [S1209]. The first user command may be, for example, selecting (e.g., clicking) a desired graphic object (e.g., an icon, a menu, a link, a text input box, etc.) on the first screen (200A) by moving the first pointer (400A) using the first input device (123A) onto the desired clickable graphic object.

[0310] If the graphic object is a text input box, the first user command may include a typing command (e.g., entering a search word) entered by the user via a first input device (e.g., a keyboard) after the selection command.

[0311] The first control unit (180A) of the first user terminal (100A) can execute an operation according to the first user command [S1211]. For example, if the graphic object is a search word input box, the first control unit (180A) of the first user terminal (100A) can perform a search operation corresponding to the search word and control the display of the search result on at least one of the first screen (200A) and its extended screen (200B).

[0312] As illustrated in (17-1) of FIG. 17, a user can control a first pointer (400A) to pass through an edge area (E1) where the first screen (400A) and the second screen (200B) are adjacent to each other from the first screen (200A) toward the second screen (200B) (i.e., in the first direction) by using a first input device (e.g., a mouse, a touch pad, etc.) of the first user terminal (100A) [S1601]. The passage of the first pointer (400A) through the edge area (E1) is as described above in FIG. 5.

[0313] Then, the first control unit (180A) of the first user terminal (100A) can control to transmit a control target selection notification to the second user terminal (100B) through the first communication connection or the second communication connection in response to the first pointer (400A) passing through the edge area (E1) in the first direction [S1403]. The control target selection notification is for the first user terminal (100A) to inform the second user terminal (100B) that the second user terminal (100B) has become the control target of the first input device (123A).

[0314] Then, the first control unit (180A) of the first user terminal (100A) can control to transmit a control target selection notification to the second user terminal (100B) through the first communication connection or the second communication connection [S1403].

[0315] And, the first control unit (180A) of the first user terminal (100A) can control the first pointer (400A) to disappear from the first screen (200A) [S1405].

[0316] The second control unit (180B) of the second user terminal (100B) can control the second pointer (400B) to be displayed on the second screen (200B) in response to the control target selection notification [S1219].

[0317] In FIG. 16, steps S1405 and S1219 are depicted as being performed after step S1403. However, steps S1403 and S1219 may also be performed after step S1405. Alternatively, steps S1403 and S1405 may be performed substantially simultaneously.

[0318] The first user terminal (100A) can receive a second user command through the first input device (123A) of the first user terminal (100A) [S1221]. The second user command may include at least one of pointer display information (e.g., pointer position information, pointer movement information, and / or pointer virtual coordinate information), a selection command, a typing command, etc.

[0319] The second user command may be, for example, moving the second pointer (400B) using the first input device (123A) over a desired clickable graphical object (e.g., an icon, menu, link, text input box, etc.) on the second screen (200B) to select (e.g., click) the desired graphical object.

[0320] If the graphic object is a text input box, the second user command may include a typing command (e.g., entering a search word) entered by the user via the first input device (e.g., a keyboard) after the selection command.

[0321] Additionally, the second user command may further include pointer display information regarding the display position (or movement direction) of the second pointer (400B) on the second screen (200B).

[0322] The first user terminal (100A) can transmit the second user command to the second user terminal (100B) via the second communication connection [S1223]. That is, the first control unit (180A) of the first user terminal (100A) can control the second user terminal (100B) to only transmit the second user command to the second user terminal (100B) and not to perform an operation according to the second user command itself. For example, if the second user command is a command regarding a search, the first user terminal (100A) can only transmit the second user command to the second user terminal (100B) and not perform a search operation.

[0323] The second user terminal (100B) can receive a second user command and perform an operation according to the second user command [S1225].

[0324] The second control unit (180B) of the second user terminal (100B) can control the second pointer (400B) to be displayed on the second screen (200B) in response to the pointer display information of the second user command.

[0325] For example, if the graphic object is a search word input box, the second control unit (180B) of the second user terminal (100B) can perform a search operation corresponding to the search word according to a second user command and control the search result to be displayed on the second screen (200B).

[0326] As illustrated in (17-2) of FIG. 17, the user can control the second pointer (400B) to pass through the edge area (E1) where the first screen (400A) and the second screen (200B) are adjacent to each other from the second screen (200B) toward the first screen (200A) (i.e., in the second direction) by using the first input device (e.g., mouse, touch pad, etc.) of the first user terminal (100A) [S1601]. The passage of the second pointer (400B) through the edge area (E1) is as described above in FIG. 5.

[0327] Then, the second control unit (180B) of the second user terminal (100B) can control to transmit a control target release notification to the first user terminal (100B) through the first communication connection or the second communication connection [S1605]. The control target release notification is for the second user terminal (100B) to inform the first user terminal (100A) that the second user terminal (100B) is no longer a control target of the first input device (123A).

[0328] If the first user terminal (100A) can determine from the user's operation input of the first input device (123A) that the second pointer (400B) passes through the edge area (E1) in the second direction, step S1605 may be omitted.

[0329] And, the second control unit (180B) of the second user terminal (100B) can control the second pointer (400B) to disappear from the second screen (200B) [S1413].

[0330] And, the first control unit (180A) of the first user terminal (100A) can control the first pointer (400A) to be displayed on the first screen (200A) [S1208].

[0331] In FIG. 14, steps S1413 and S1208 are illustrated as being performed after step S1605. However, steps S1605 and S1208 may be performed after step S1413. Alternatively, steps S1413 and S1605 may be performed substantially simultaneously.

[0332] The above-described present disclosure can be implemented as computer-readable code on a program-recorded medium. A computer-readable medium includes any type of recording device that stores data readable by a computer system. Examples of computer-readable media include hard disk drives (HDDs), solid-state disk drives (SSDs), silicon disk drives (SDDs), read-only memory (ROM), random-access memory (RAM), CD-ROMs, magnetic tapes, floppy disks, and optical data storage devices.

Claims

1. Input device; display; A communication unit for a first communication connection and a second communication connection with an external user terminal; and Create a first screen and an extended screen of the first screen, Display the first screen on the above display, Transmitting the extended screen to the external user terminal via the first communication connection, A user terminal including a control unit that controls the transmission of at least one of the pointer display information of the input device and the user command input through the input device to the external user terminal through a second communication connection.

2. In the first paragraph, the control unit, A user terminal characterized in that it receives display identification data from the external user terminal through a first communication connection.

3. In the second paragraph, the control unit, Receive the IP address and product unique identifier of the external user terminal from the external user terminal via a wireless network, If the received product unique identifier is identical to the product unique identifier of the display identification data, the IP address is matched to the external user terminal, Perform PIN code authentication with the external user terminal using the above IP address, Receive a client key code for automatic connection from the external user terminal, A user terminal characterized in that it controls to establish a second communication connection with the external user terminal using the client key code for the automatic connection.

4. In the first paragraph, the control unit, Receive mode status notification from the external user terminal, If the external user terminal is determined to be operating as a main device through the above mode status notification, request the external user terminal to create an input communication channel in the second communication connection, A user terminal characterized in that it controls to transmit at least one of the pointer display information of the input device and the user command input through the input device to the external user terminal through the input communication channel.

5. In the first paragraph, the control unit, A user terminal characterized in that, while a first communication connection and a second communication connection are established with a first external user terminal, a first communication connection and a second communication connection are established with a second external user terminal.

6. In paragraph 1, A user terminal characterized in that the first communication connection is an HDMI (High Definition Multimedia Interface) or DP (DisplayPort) connection, and the second communication connection is a Wi-Fi communication connection.

7. In the first paragraph, the control unit, In response to an SDP (Service Discovery Protocol) request signal received from the external user terminal, an SDP response signal is transmitted to the external user terminal, A user terminal characterized by controlling Bluetooth pairing with the external user terminal as a second communication connection.

8. In the first paragraph, the control unit, A user terminal characterized in that it is determined whether the external user terminal operates in the first mode or the second mode based on a mode status notification signal received from the external user terminal.

9. In paragraph 8, the control unit, When the external user terminal operates in the second mode, a pointer is displayed on the display, A user terminal characterized in that the pointer is controlled not to be displayed on the display when the external user terminal operates in the first mode.

10. In paragraph 8, the control unit, When a user command is input through the input device while the external user terminal is operating in the second mode, an operation corresponding to the user command is executed, A user terminal characterized in that, when a user command is input through the input device while the external user terminal is operating in the first mode, the user command is transmitted to the external user terminal through a second communication connection without executing an operation corresponding to the user command.

11. In paragraph 8, the control unit, Displaying a pointer on the display while the external user terminal operates in the first mode; A user terminal characterized in that, in response to the input device being controlled to cause the pointer to disappear by passing through an edge area of ​​the display, a control target selection notification is transmitted to the external user terminal.

12. In the 11th paragraph, the control unit, When a user command is input through the input device while the pointer is displayed on the display, an action corresponding to the user command is executed; A user terminal characterized in that, when the user command is input through the input device after the pointer passes through the edge area and disappears, the user command is transmitted to the external user terminal through a second communication connection without executing an operation corresponding to the user command.

13. Input devices; display; A communication unit for a first communication connection and a second communication connection with an external user terminal; and While operating in the second mode, an extended screen received from the external user terminal through the first communication connection is displayed on the display, wherein the extended screen may include a first pointer, A user terminal including a control unit that, while operating in a first mode, displays a second screen on the display, and generates a second pointer based on pointer display information received through a second communication connection and controls the display to display the second pointer on the second screen.

14. In the 13th paragraph, the control unit, A user terminal characterized in that it controls to create an input communication channel in response to an input communication channel creation request received from the external user terminal through a second communication connection and to transmit information of the created input communication channel to the external user terminal.

15. In the 13th paragraph, the control unit, A user terminal characterized in that it controls to perform an operation according to a user command received through a second communication connection while operating in the first mode.

16. In the 13th paragraph, the control unit, A user terminal characterized in that it controls to transmit a mode status notification to the external user terminal whenever a mode switch is performed between the first mode and the second mode.

17. In the 13th paragraph, the control unit, A user terminal characterized in that, when a control target selection notification is received from the external user terminal while operating in the first mode, a second pointer based on the pointer display information received through the second communication connection is controlled to be displayed on the second screen.

18. In paragraph 17, the control unit, A user terminal characterized in that, in response to the pointer passing through the edge area of ​​the display and disappearing, a control target release notification is transmitted to the external user terminal.

19. A step of establishing a first communication connection and a second communication connection with a first external user terminal; A step of creating a first screen and displaying it on a display; A step of creating an extended screen of the first screen and transmitting it to an external user terminal through the first communication connection; and A method for controlling a user terminal, comprising the step of transmitting at least one of pointer display information of an input device and a user command input through the input device to the external user terminal via a second communication connection.

20. A step of establishing a first communication connection and a second communication connection with an external user terminal; While operating in the second mode, a step of displaying an extended screen received from the external user terminal through the first communication connection on the display, wherein the extended screen can include a first pointer; and A method for controlling a user terminal, comprising: while operating in a first mode, displaying a second screen on the display; generating a second pointer based on pointer display information received through a second communication connection and displaying the second pointer on the second screen;

Images