Terminal device for controlling external device, and device control method therefor

Abstract

A terminal device for controlling an external device is disclosed. A processor of the present device controls a display to display a video call screen with a correspondent device performing a mirroring operation in which the screen of an external device is shared, generates a control signal for the external device when a user operation for controlling the external device is input, and transmits same to the correspondent device through a communication unit.

Description

Terminal device for controlling an external device and method for controlling the device

[0001] The present disclosure relates to a terminal device for controlling an external device and a method for controlling the device.

[0002] Recent advancements in electronic technology have led to the development of a variety of electronic devices. In particular, electronic devices supporting various services, such as video calling and mirroring, are being developed. Video calling allows multiple users to communicate while seeing each other's faces through their own devices. Mirroring refers to a service that mirrors the image displayed on one electronic device's display on another electronic device.

[0003] Also known as screen sharing, the mirroring service allows users to share the image displayed on an electronic device, such as a laptop, tablet, or mobile device, with another display device, such as a television or monitor.

[0004] With various services being provided as described above, the need for a more diverse and convenient way to utilize these services has arisen.

[0005] According to the present disclosure, a terminal device according to at least one embodiment includes a communication unit, a display, and a processor.

[0006] The processor controls the display to display a video call screen with a counterpart device that performs a mirroring operation for sharing the screen of an external device, and when a user operation for controlling the external device is input, generates a control signal for the external device, and transmits the control signal to the counterpart device through the communication unit so that the counterpart device transmits the control signal to the external device.

[0007] Meanwhile, a mobile device according to at least one embodiment of the present disclosure includes a communication unit, a display, and a processor for performing communication with an external device.

[0008] The processor, when a mirroring function is performed for a second external device that is making a video call with a first external device, provides the screen of the display to the second external device through the communication unit, and when control over the user terminal device is granted to the first external device, allows limited control by the first external device.

[0009] Meanwhile, a server device according to at least one embodiment of the present disclosure includes a communication unit, a memory, and a processor for performing communication with a plurality of terminal devices.

[0010] The processor, when a video call is performed between the plurality of terminal devices, generates a video call screen including images provided by each of the terminal devices and transmits the screen to each terminal device through the communication unit, and when a mirroring operation is performed to share the screen of an external device on at least one of the plurality of terminal devices, receives information about the screen of the external device from the at least one terminal device through the communication unit and stores it in the memory, reconfigures the video call screen to add the stored screen of the external device, and transmits the reconfigured video call screen to each terminal device through the communication unit, and when a user operation for controlling the external device is transmitted from one of the plurality of terminal devices, generates a control signal for causing the external device to perform an operation corresponding to the user operation, and transmits the control signal to a terminal device mirrored with the external device through the communication unit.

[0011] Meanwhile, a control method of a terminal device according to at least one embodiment of the present disclosure displays a video call screen with a counterpart device (Correspondent Device) that performs a mirroring operation for sharing the screen of an external device.

[0012] When a user operation for controlling the external device is input, a control signal corresponding to the user operation is generated, and the control signal is transmitted to the counterpart device so that the counterpart device transmits the control signal to the external device.

[0013] Meanwhile, according to one embodiment of the present disclosure, a non-transitory computer-readable recording medium storing a computer program for a terminal device to control an external device is provided. The computer program may sequentially perform the steps of displaying a video call screen with a counterpart device (Correspondent Device) performing a mirroring operation for sharing the screen of the external device, generating a control signal corresponding to a user operation when a user operation for controlling the external device is input, and transmitting the control signal to the counterpart device so that the counterpart device transmits the control signal to the external device.

[0014] FIG. 1 is a drawing for explaining a terminal device controlling an external device according to one embodiment of the present disclosure.

[0015] FIG. 2 is a block diagram showing the configuration of a terminal device according to an embodiment of the present disclosure.

[0016] FIG. 3 is a diagram for explaining a process of sharing a screen of a terminal device according to various embodiments of the present disclosure.

[0017] Figures 4 to 7 are drawings for explaining user operations of a terminal device according to various embodiments of the present disclosure.

[0018] Figure 8 is a flowchart illustrating an example of terminal device operation when multiple devices are connected.

[0019] FIG. 9 is a drawing for explaining a server device that controls an external device according to one embodiment of the present disclosure.

[0020] FIG. 10 is a block diagram showing the configuration of a server device according to one embodiment of the present disclosure.

[0021] FIG. 11 is a block diagram showing the configuration of a mobile device according to an embodiment of the present disclosure.

[0022] FIG. 12 is a flowchart for explaining a device control method of a terminal device according to an embodiment of the present disclosure.

[0023] Figure 13 is a flowchart illustrating an example of a device control method in a state where multiple devices are connected to each other.

[0024] The terms used in the various embodiments of this disclosure have been selected from widely used, current terms, taking into account the functions of this disclosure. However, these terms may vary depending on the intentions of those skilled in the art, precedents, the emergence of new technologies, etc. Furthermore, in certain cases, terms may be arbitrarily selected by the applicant, and in such cases, their meanings will be described in detail in the description of the relevant disclosure. Therefore, the terms used in this disclosure should be defined based on the meaning of the terms and the overall content of this disclosure, rather than simply their names.

[0025] In this disclosure, expressions such as “has,” “can have,” “includes,” or “may include” indicate the presence of a corresponding feature (e.g., a component such as a number, function, operation, or part), and do not exclude the presence of additional features.

[0026] The expression "at least one of A and / or B" should be understood to mean either "A" or "B" or "A and B".

[0027] The expressions “first,” “second,” “first,” or “second,” etc., used in this disclosure can describe various components, regardless of order and / or importance, and are only used to distinguish one component from another, but do not limit the components.

[0028] When it is said that a component (e.g., a first component) is “(operatively or communicatively) coupled with / to” or “connected to” another component (e.g., a second component), it should be understood that the component may be directly coupled to the other component, or may be connected through another component (e.g., a third component).

[0029] Singular expressions include plural expressions unless the context clearly dictates otherwise. In this disclosure, terms such as "comprise" or "consist of" are intended to indicate the presence of a feature, number, step, operation, component, part, or combination thereof described in the specification, but should be understood not to preclude the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

[0030] In the present disclosure, a "module" or "part" performs at least one function or operation and may be implemented as hardware or software, or as a combination of hardware and software. Furthermore, multiple "modules" or multiple "parts" may be integrated into at least one module and implemented as at least one processor (not shown), excluding any "modules" or "parts" that need to be implemented as specific hardware.

[0031] In this disclosure, the term user may refer to a person using an electronic device or a device used by the person.

[0032] An embodiment of the present disclosure will be described in more detail with reference to the attached drawings below.

[0033] FIG. 1 is a drawing for explaining a terminal device controlling an external device according to one embodiment of the present disclosure.

[0034] The terminal device (100) may be implemented as a TV including a display, but is not limited thereto, and is applicable to any device having a display function, such as a video wall, a large format display (LFD), a digital signage, a digital information display (DID), a projector display, etc. In addition, the terminal device (100) may include various types of displays, such as a liquid crystal display (LCD), an organic light-emitting diode (OLED), a liquid crystal on silicon (LCoS), a digital light processing (DLP), a quantum dot (QD) display panel, a quantum dot light-emitting diodes (QLED), etc. The terminal device (100) is not limited thereto, and the terminal device (100) may be implemented as various types of output devices.

[0035] Additionally, it goes without saying that the terminal device (100) can be implemented as a user terminal device or an audio output device (e.g., a sound bar, a home theater system, a room speaker, headphones, earphones, etc.).

[0036] As described above, the terminal device (100) according to one embodiment of the present disclosure can be implemented in various forms, and depending on the form, can independently provide various services, such as content display. Furthermore, the terminal device (100) can also provide services that link with other terminal devices. For example, the terminal device (100) can provide video call services, video conferencing services, general chat services, or video chat services.

[0037] Referring to FIG. 1, a state in which a user (1) performs a video call with another user (2) using his / her terminal device (100) is illustrated. In the present disclosure, for the convenience of explanation, the device used by the other user (2) is referred to as a counterpart device (200). For example, when a user (1) and the counterpart (2) make a video call, the terminal device (100) transmits an image captured of the user (1) using a built-in or connected camera (10) and an audio input through a microphone (not shown) to the counterpart device (200). The counterpart device (200) also provides an image captured of the counterpart (2) using a camera (20) and the audio thereof to the terminal device (100). This video call service may be performed through a separate relay server (not shown) or may be performed through a direct connection between terminal devices. The present disclosure is not limited thereto.

[0038] In addition to the videos each user (1) and the other party (2) have filmed, they can also share documents, images, and other files stored in their respective devices (100, 200), and by connecting another device in a mirroring manner, they can also view the screen of that device together.

[0039] In Fig. 1, a state is shown in which a counterpart (2) performs a mirroring operation on the screen of another terminal device (300) with the counterpart device (200). For convenience of explanation, the other terminal device is hereinafter referred to as an external device (300). In Fig. 1, the terminal device (100) and the counterpart device (200) are each depicted as a TV, and the external device (300) is depicted as a mobile phone, but each device may be implemented in various forms.

[0040] As illustrated in FIG. 1, when the other party (2) executes the mirroring function to allow the screen of the external device (300) to be shared with the other party's device (200), the shared screen contents can be displayed as is on the terminal device (100) of the user (1) making the video call.

[0041] The user (1) can view the screen of the external device (300) of the other party (2) together with the other party (2), and, if necessary, the user (1) can directly control it.

[0042] When a user (1) inputs a user operation to control an external device, the terminal device (100) generates a control signal corresponding to the user operation. For example, the terminal device (100) may generate a control signal to enlarge and display specific content displayed on the screen of the external device (300) or to select a specific application icon to execute an application.

[0043] The terminal device (100) transmits the generated control signal to the counterpart device (200), and the counterpart device (200) transmits the generated control signal to the control device (300). The control device (300) performs an operation corresponding to the received control signal. Fig. 2 is a block diagram illustrating the configuration of a terminal device according to an embodiment of the present disclosure.

[0044] Referring to FIG. 2, the terminal device (100) includes a communication unit (110), a processor (120), a display (130), and a memory (140).

[0045] The communication unit (110) may include wired or wireless input / output interfaces (or input / output terminals) according to various standards. For example, the communication unit (110) may include various interfaces such as HDMI (High Definition Multimedia Interface), MHL (Mobile High-Definition Link), USB (Universal Serial Bus), DP (Display Port), Thunderbolt, VGA (Video Graphics Array) port, RGB port, D-SUB (D-subminiature), DVI (Digital Visual Interface), AP-based Wi-Fi (Wireless LAN network), Bluetooth, Zigbee, wired / wireless LAN (Local Area Network), WAN (Wide Area Network), Ethernet, IEEE 1394, AES / EBU (Audio Engineering Society / European Broadcasting Union), optical, coaxial, etc.

[0046] When the processor (120) executes a video call function with the counterpart device (200), the communication unit (110) receives video, voice, and other data provided by the counterpart device (200), and can transmit video, voice, and other data of the user (1) generated through the terminal device (100) to the counterpart device (200).

[0047] Alternatively, when the processor (120) executes a mirroring function for a separate external device, the communication unit (110) may receive data for the screen of the external device or transmit data for the screen displayed on the display (130) of the terminal device (100) to the external device.

[0048] The display (130) is a configuration for displaying various screens. For example, when the processor (120) executes a video call function with the counterpart device (200), the display (130) can display the video call screen. The display (130) can be implemented as various types of displays such as an LCD (liquid crystal display), an OLED (organic light-emitting diode), an LCoS (Liquid Crystal on Silicon), a DLP (Digital Light Processing), a QD (quantum dot) display panel, a QLED (quantum dot light-emitting diodes), a μLED (Micro light-emitting diodes), a Mini LED, etc. Meanwhile, the display (130) can also be implemented as a touch screen combined with a touch sensor, a flexible display, a rollable display, a 3D display, a display in which a plurality of display modules are physically connected, etc.

[0049] The memory (140) is a configuration for storing various data and programs required for the operation of the terminal device. The memory (140) may be implemented as a volatile memory such as S-RAM (Static Random Access Memory) and D-RAM (Dynamic Random Access Memory), a non-volatile memory such as Flash Memory, ROM (Read Only Memory), EPROM (Erasable Programmable Read Only Memory), and EEPROM (Electrically Erasable Programmable Read Only Memory), a hard disk drive (HDD), or a solid state drive (SSD).

[0050] The memory (140) can be accessed by the processor (120). The processor (120) can perform operations such as reading / writing / modifying / deleting / updating data on the memory (140). In FIG. 2, the memory (140) is illustrated as being included as a separate unit from the processor (120), but depending on the embodiment, the memory (140) may be implemented in multiple units, and among them, a memory such as a ROM or RAM may be mounted within the processor (120).

[0051] The processor (120) is a component for controlling the operation of the terminal device (100). The processor (120) may be implemented as a digital signal processor (DSP), a microprocessor, or a timing controller (TCON) for processing digital signals. However, the processor (120) is not limited thereto, and may include one or more of a central processing unit (CPU), a micro controller unit (MCU), a micro processing unit (MPU), a controller, an application processor (AP), a communication processor (CP), an ARM processor, or an artificial intelligence (AI) processor, or may be defined by the relevant terminology. In addition, the processor (120) may be implemented as a system on chip (SoC) having a built-in processing algorithm, a large scale integration (LSI), or may be implemented in the form of a field programmable gate array (FPGA). The processor (120) may perform various functions by executing computer executable instructions stored in a memory.

[0052] When a method according to an embodiment of the present disclosure includes multiple operations, the multiple operations may be performed by one processor or by multiple processors. For example, when a first operation, a second operation, and a third operation are performed by a method according to an embodiment, the first operation, the second operation, and the third operation may all be performed by the first processor, or the first operation and the second operation may be performed by the first processor (e.g., a general-purpose processor) and the third operation may be performed by the second processor (e.g., an artificial intelligence-specific processor).

[0053] The processor (120) may be implemented as a single core processor including one core, or may be implemented as one or more multi-core processors including multiple cores (e.g., homogeneous multi-cores or heterogeneous multi-cores). When one or more processors (120) are implemented as multi-core processors, each of the multiple cores included in the multi-core processor may include internal processor memory such as cache memory or on-chip memory, and a common cache shared by the multiple cores may be included in the multi-core processor. In addition, each of the multiple cores (or some of the multiple cores) included in the multi-core processor may independently read and execute a program instruction for implementing a method according to an embodiment of the present disclosure, or all (or some) of the multiple cores may be linked to read and execute a program instruction for implementing a method according to an embodiment of the present disclosure.

[0054] The processor (120) performs an operation corresponding to a user operation input through the terminal device (100).

[0055] When a user selects an application for performing a video call function, the processor (120) executes the application and displays an execution screen on the display (130). The user can input various identification information, such as the name, phone number, ID, email address, and messenger ID of the other party to be called, on the execution screen. The processor (120) searches for a counterparty device based on the input identification information, and when a counterparty device (200) corresponding to the identification information is found, the processor transmits a request for a video call connection to the counterparty device (200). When the counterparty device (200) receives the request, it automatically executes the video call application and then waits for the user's response. When the user, i.e., the other party (2), responds to the video call, the counterparty device (200) transmits a response signal to the request. When the response signal is received through the communication unit (110), the processor (120) controls the communication unit (110) to connect the video call session of the counterparty device (200). The processor (120) activates the camera and microphone built into or connected to the terminal device (100) to configure a video call screen including the image of the user (1), and then displays the screen on the display (130), while controlling the communication unit (110) to transmit the image and voice of the user to the counterpart device (200). When the image and voice of the counterpart device (1) are transmitted from the counterpart device (200) through the video call session, the processor (120) additionally displays them on the video call screen. Depending on the embodiment, the video call function may be performed by a different mechanism, and various embodiments of the present disclosure are not limited to any one mechanism.

[0056] During a video call in this manner, when the counterpart device (200) executes a mirroring function that shares the screen of the external device (300), the screen of the external device (300) is shared on the display screen of the counterpart device (200). The counterpart device (200) transmits information about the screen of the external device shared by the mirroring function to the terminal device (100).

[0057] When the processor (120) receives information about the screen of an external device (300) through the communication unit (110), it stores the information in the memory (140).

[0058] The processor (120) controls the display (130) to add the screen of the received external device to the video call screen and display it.

[0059] In addition to the other party's (2) video on the video call screen, the user (1) can also check the screen of an external device (300) used separately by the other party (2). In this case, a situation may arise where the user (1) wishes to directly control the external device (300).

[0060] For example, if a user (1) wants to explain to the other party (2) how to use an external device (300), it may be more convenient to explain to the other party (2) while directly controlling the external device (300). In addition, if the other party (2) is a pet or an elderly person with weak mental abilities, the user (1) can control the operation of the external device (300) placed at the location of the other party (2) to execute various functions such as making an emergency call or playing content.

[0061] In this case, the user can input user operations to control the external device. The user operations can be input in various forms depending on the type and characteristics of the terminal device (100). This will be described in detail in the following section.

[0062] When such a user operation is input, the processor (120) generates a control signal that causes the external device (300) to perform an operation corresponding to the user operation. The processor (120) generates a control signal corresponding to the user operation and transmits the control signal to the counterpart device (200) via the communication unit (110) so that the counterpart device (200) transmits the control signal to the external device (300).

[0063] Specifically, when a user (1) inputs a user operation to control an external device (300) through a terminal device (100), the processor (120) transmits a message asking whether control of the external device (300) is permitted to the counterpart device (200) through the communication unit (110) so that the counterpart device (200) displays the message on its own display.

[0064] The counterpart device (200) displays the received message, and the counterpart (2) can select whether to allow control in response to the message. If the counterpart (2) selects to allow control of the external device (300), the counterpart device (200) transmits a response regarding whether to allow control.

[0065] When a response signal that does not allow control is received, the processor (120) controls the display (130) to display a message indicating that control is not permitted.

[0066] On the other hand, when the processor (120) receives a response signal allowing control, the processor (120) can secure control over the external device (300). When a user operation is input while the control is secured, the processor (120) can generate a control signal that causes the external device (300) to perform an operation corresponding to the user operation. The control signal can be generated in different formats depending on the type of the external device (300). For example, when the external device (300) has a touch screen, and a plurality of icons are displayed on the touch screen, and the processor of the external device (300) identifies that one of the icons has been selected, it can be implemented to perform a control operation such as executing an application corresponding to the icon or playing content. In this case, the processor (120) can generate a control signal that includes a control code or interaction that notifies that the user has selected a specific icon within the screen currently displayed on the external device (300). Alternatively, the processor (120) may generate a control signal that includes information about the coordinates of a touch or selected point within the currently displayed screen of the external device (300). Alternatively, if the external device (300) and the terminal device (100) are products of the same manufacturer or products that use the same signal system, the processor (120) may generate a control signal corresponding to an execution command for a specific application or a playback command for content, identically to a control command directly generated by the processor of the external device (300). The processor (120) may set a flag corresponding to the external device (300) or add an indicator to the header portion of the control signal to allow the counterpart device (200) to transmit the received control signal to the external device (300). Meanwhile, the method of generating and transmitting the control signal is merely an example and is not limited thereto.

[0067] Meanwhile, in the terminal device (100) controlling the external device (300), the processor (120) may transmit video and audio to the counterpart device (200) using an external server rather than directly through the communication unit (110). The communication unit (110) may receive data for a video call screen including the video captured by the camera (10) of the terminal device (100), the video captured by the counterpart device (200), and the screen of the external device (300) from the external server.

[0068] The processor (120) controls the display (130) to display a video call screen based on the received data. Thereafter, when a user operation for controlling an external device (300) is input on the video call screen, a control signal corresponding to the user operation is generated and the control signal is transmitted to the counterpart device (200) through the communication unit (110).

[0069] Below, the video call screen sharing process between the terminal device (100) and the counterpart device (200) will be described in detail.

[0070] FIG. 3 is a diagram for explaining a process of sharing a screen of a terminal device according to various embodiments of the present disclosure.

[0071] Referring to FIG. 3, when a terminal device (100) and a counterpart device (200) are making a video call and the counterpart device (200) and an external device (300) are performing mirroring communication, the display screen (131) of the terminal device (100) and the display screen (231) of the counterpart device (200) appear as shown.

[0072] The processor (120) controls the display (130) to display images captured by a camera built into or connected to the terminal device (100), images received from a counterpart device, and screens of external devices all within the video call screen (131). In the present disclosure, the areas where captured images, counterpart device images, and external device screens are displayed are referred to as first areas, second areas, and third areas, respectively. The processor (120) can display the aforementioned captured images, external device screens, and counterpart device images in each area within the video call screen (131) based on layout information stored in the memory (140).

[0073] Meanwhile, the size of the third area where the external device screen is displayed may be the same as the actual screen size of the external device (300), but if the display (130) of the terminal device (100) is much larger, the processor (120) may enlarge and scale the screen of the external device (300) and display it in the third area. In this case, the aspect ratio may be maintained as is, but is not necessarily limited thereto.

[0074] Referring to Fig. 3, on the display screen (131) of the terminal device, the image of the user (1) is placed in the first area (132) on the upper right, and the image of the other party (2) is placed in the second area (133) on the lower right. The screen of the external device is placed in the third area (134) on the left.

[0075] On the display screen (231) of the counterparty, the video call screen of the counterparty (2) is placed in the first area (232), the video call screen of the user (1) is placed in the second area (233) below it, and the display screen of the external device (300) that performs mirroring communication with the counterparty (200) is placed in the third area (234) on the left. The position of each screen to be placed is not limited thereto and may be placed in other positions. Meanwhile, among the screens to be placed, the video call image (132, 232) of the user or the counterparty may be selectively displayed on the display screen. If the user sets his / her image to be invisible or the counterparty sets his / her image to be invisible, the processor (120) may control the display (130) to display the video call screen excluding the user's image or the counterparty's image based on the setting status.

[0076] In addition, in Fig. 3, considering the aspect ratio of the screen of the mobile phone, the state in which the external device screen is displayed in the third area (134, 234) on the left is illustrated, but if the type of external device is changed and the aspect ratio is changed, the processor (120) may arrange each image in a corresponding layout.

[0077] Figures 4 to 7 are drawings for explaining user operations of a terminal device according to various embodiments of the present disclosure.

[0078] Referring to FIG. 4, the processor (120) controls the display to add highlight marks on the video call screen (131) where each image is arranged by region.

[0079] Specifically, the processor (120) may add a highlight mark to the edge portion of at least one of the first to third regions (131, 132, 133). The highlight mark includes graphic processing to display the region in a more distinctive manner than other regions. For example, various graphic processing may be used, such as displaying the edge portion of one region with a thick border, displaying the edge color differently from the edge colors of other regions, lighting up a brightly colored graphic object, changing the overall color of the region, lighting up the entire region, or displaying other regions in an opaque state. FIGS. 4 to 6 illustrate a case where a thick border mark is used. For convenience of explanation, the present disclosure describes it as a highlight mark, but it may also be referred to as a selected region mark, a highlight mark, etc.

[0080] The user can check the highlight display to know where the currently selected area is. Referring to the left drawing of Fig. 4, a state in which a highlight display is added to the first area (132) is shown. In this state, if the user inputs a movement operation using a device capable of left / right / up / down movement operations, the highlight display moves to an area other than the area in which the highlight display was previously displayed. Fig. 4 shows a state in which the highlight display has moved to the third area (134). If the user inputs a selection command in this state, the processor (120) can recognize that a user operation has been input to acquire control of the external device (300).

[0081] Meanwhile, if the external device (300) provides a screen containing multiple objects (e.g., icons, thumbnails, text, images, etc.), the user may directly select one of the objects within a third area on the video call screen. Fig. 5 is a drawing illustrating this process.

[0082] As described in FIG. 4, when the user inputs a selection command while the highlight has moved to the third area (134), the processor (120) indicates a state in which the highlight has moved to one (135-1) of the objects (135-1 to 135-n) displayed on the external device screen within the third area (134) and is displayed. In this state, when the user inputs a movement operation again using a device capable of moving left / right / up / down, the processor (120) controls the display (130) to move the highlight to the next object (135-2) and display it. In this state, when a selection command is input again, the processor (120) generates a control signal to perform an operation corresponding to the object (135-2) where the highlight is located, and transmits the control signal to the counterpart device (200) via the communication unit (110).

[0083] When the counterpart device (200) receives a control signal, it transmits it to the external device (300). The external device (300) performs an operation corresponding to the object selected by the user based on the control signal. Specifically, if the user selects a web browser icon, the external device (300) launches the web browser and displays a default web page. Alternatively, if the user selects an icon for playing music or checking photo content, the external device (300) may perform an operation such as playing music or displaying photo content.

[0084] The above-described movement operation and selection commands can be input through various input devices such as a remote control for controlling the terminal device, a mobile phone for communicating with the terminal device, buttons or a touch screen provided on the main body of the terminal device, etc.

[0085] In FIGS. 4 and 5, a method of selecting an object by moving an indicator such as a highlight display has been described. However, if the display (130) of the terminal device (100) itself is capable of receiving touch manipulation input, the user may attempt control by directly touching the third area (134). Alternatively, an input device such as a remote control may be used. FIG. 6 illustrates a case in which an operation cursor (136) that moves according to the movement of an input device such as a remote control is used.

[0086] Referring to FIG. 6, the processor (120) can display a cursor (136) within a video call screen. The user can move the cursor (136) using an input device to directly select the desired area (132, 133, 134) and objects (135-1 to 135-n).

[0087] For example, a user can move a cursor (136) using a remote control, which is one of the devices capable of performing movement operations, to move the cursor (136) to an object (e.g., 135-1) on the screen of an external device, and then select the object by pressing a button on the remote control. If the display (130) is implemented as a touch screen, the user can also directly touch the object.

[0088] When implemented in the same manner as in FIG. 6, the processor (120) can identify the coordinate values ​​within the actual screen corresponding to the coordinate values ​​of the point directly touched or selected by the user, taking into account the ratio between the size of the scaled third area (134) and the actual screen size of the external device (300). For example, assuming that the scale is enlarged three times, if the coordinate values ​​of the point directly touched or selected by the user are (3x, 3y), the processor (120) can recognize that the point (x, y) within the screen of the actual external device (300) has been touched or selected. Accordingly, the processor (120) can generate a control signal including information on the recognized coordinate values. If information on the arrangement positions and object types of each object is received from the external device (300) and stored in the memory (140), the processor (120) can directly generate and transmit a control signal to perform an operation corresponding to the object based on the information and the recognized coordinate values.

[0089] FIG. 7 is a drawing for explaining user operations of a terminal device according to an embodiment of the present disclosure.

[0090] Referring to Fig. 7, the external device (300) is not a mobile phone, but rather a device such as a desktop or laptop. Even in this case, all of the contents described above in Figs. 4 to 6 can be applied. For example, if the external device performing mirroring communication with the counterpart device (200) is a laptop, the user (1) can gain control of the laptop's display screen and control the display.

[0091] The above description assumes a one-on-one video call between a terminal device and a counterpart device. However, multi-party video call or video conference functions may also be implemented. In this case, based on the terminal device (100), there may be multiple counterpart devices.

[0092]

[0093] *Figure 8 is a flowchart illustrating an example of terminal device operation when multiple devices are connected.

[0094] When there are multiple counterpart devices, the terminal device (100) can perform mirroring with at least one of the multiple counterpart devices and an external device. For example, assuming that one counterpart device (200) is mirroring with an external device (300), the remaining counterpart devices and the terminal device (100) may attempt to secure control over each other.

[0095] Figure 8 illustrates the process for assigning priority to control rights in such cases.

[0096] When the processor (120) is performing a video call with a plurality of counterpart devices (200), and at least one of the plurality of counterpart devices is connected to an external device (300) to perform mirroring communication, the processor (120) controls the display (130) to display a video call screen that includes at least one of the screens of the users of each counterpart device and the screens of the external device (S810).

[0097] In this state, when a user inputs a user operation to control an external device, the processor (120) identifies whether there is a device that has control among the remaining counterpart devices (S820).

[0098] If there is no device having control authority, the processor (120) can control the external device by transmitting a control signal to the counterpart device according to user operation as described in the various embodiments described above (S840).

[0099] On the other hand, if there is a specific device that has control authority over the external device (300), the control authority priority is compared between the specific device and the terminal device (100) (S830). Information regarding the priority of the terminal device (100) may be stored in advance in the memory (140). The priority may be preset and stored based on various criteria such as the type or location of each terminal device, and the relationship between users. Alternatively, the priority may be adaptively determined for each call based on other criteria such as the order of participation in the video call or the number of participations. Alternatively, the priority may be determined based on agreement between the users of each terminal device.

[0100] As a result of comparing priorities, if the priority of the terminal device (100) is higher than the priority of a specific device, the user (1) has the control to control the display of the external device (300) (S840).

[0101] On the other hand, if the result of comparing the priorities shows that the control priority of the terminal device (100) is lower than that of a specific device, the processor (120) controls the display (140) to display a guidance message indicating that control of the external device (300) is not possible (S850).

[0102] FIG. 9 is a drawing for explaining a server device that controls an external device according to one embodiment of the present disclosure.

[0103] Although the various embodiments described above illustrate and describe direct connections between devices, communication between multiple devices may be relayed by a server device.

[0104] Below, we will explain how to control external devices through server devices.

[0105] The server device (900) may be implemented as a central server (or integrated server) that handles interactions between various operating systems and applications in all network systems, or as a cloud server utilizing cloud computing technology. Here, cloud computing refers to an Internet-based computing technology, and is a web-based software service in which programs are placed on a utility data server on the Internet and are called up and used on a computer or mobile phone at any time.

[0106] Additionally, the server device (900) can communicate with multiple terminal devices. The multiple terminal devices can transmit identification information of the multiple terminal devices to the server device (900). Alternatively, the multiple terminal devices can transmit identification information of devices other than the multiple terminal devices to the server device (900) or authentication information for accessing a user's account.

[0107] Referring to FIG. 9, a server device (900) is connected to a plurality of terminal devices, such as a terminal device (100) of a user (1), a counterpart device (200) of a counterpart (2), a desktop (910), and a laptop (930). The devices that can be connected are not limited thereto and can be connected to various terminal devices. When at least one of the plurality of terminal devices performs mirroring communication with an external device, each terminal device can obtain control rights to control the external device and control the display of the external device. For example, when there is an external device (920) that performs mirroring communication with at least one of the plurality of terminal devices, such as a desktop (910), each terminal device can obtain control rights to control the external device (920).

[0108] As illustrated in FIG. 9, when a server device (900) intervenes, each image included in a video call screen displayed on a terminal device (100) and a counterpart device (200) may be generated based on data provided by the server device (900). The server device (900) may transmit all images to be displayed in at least one of the first to third areas of the video call screen at once, or may transmit only some of the images. Alternatively, the server device (900) may configure the video call screen itself and provide data for the screen to the terminal device and the counterpart device (100, 200), respectively.

[0109] For example, if the counterpart device (200) is not performing mirroring communication with the external device (300), the server device (900) can receive the counterpart device image captured by the counterpart device (200) from the counterpart device (200) and transmit it to the terminal device (100). Alternatively, if the counterpart device (200) is performing mirroring communication with the external device (300), both the external device screen and the counterpart device image can be transmitted to the terminal device (100) at once or sequentially.

[0110] FIG. 10 is a block diagram showing the configuration of a server device according to one embodiment of the present disclosure.

[0111] Referring to FIG. 10, the server device includes a communication unit (940), a processor (950), and a memory (960) for performing communication with a plurality of terminal devices.

[0112] When a video call involving multiple terminal devices is performed, the processor (950) generates a video call screen including images provided by each terminal device and transmits the screen to each terminal device through the communication unit (940). Thereafter, when mirroring communication is performed on at least one terminal device among the multiple terminal devices to share the screen of an external device, information about the display of the external device is received from at least one terminal device through the communication unit (940) and stored in the memory (960). The processor (950) reconfigures the video call screen to add the screen of the stored external device and transmits the reconfigured video call screen to each terminal device through the communication unit (940).

[0113] When a user operation for controlling an external device is transmitted from one of a plurality of terminal devices, the processor (950) generates a control signal for causing the external device to perform an operation corresponding to the user operation, and transmits the control signal to the terminal device mirrored with the external device through the communication unit (940).

[0114] Accordingly, multiple users can control external devices that perform video calls and screen sharing with each other while performing mirroring communication with the terminal device.

[0115] If there are multiple terminal devices that have transmitted user operations to control an external device, the processor (950) compares the priorities of each terminal device based on information in the memory (960) that stores information on the priorities of each of the multiple terminal devices, and grants control authority over the external device to a terminal device with a higher priority.

[0116] There are various types of server devices, such as the MCU (Multipoint Control Unit) and SFU (Selective Forwarding Unit). The MCU (Multipoint Control Unit) uses a server to mix multiple users' videos into a single media stream and transmit it. This method mixes video calls between multiple users, reducing traffic when exchanging data or video. The SFU (Selective Forwarding Unit) similarly uses a server. Multiple terminal devices individually connect to the server, maintaining each connection while transmitting and receiving video calls or data through the server. This method is suitable for large-scale connections, and because each terminal device is connected to the server, individual layout adjustments are possible.

[0117] In terms of the connection method of a terminal device that controls an external device, there is a direct connection method such as P2P (Peer to Peer), and as described above, there are also MCU (Multipoint Control Unit) and SFU (Selective Forwarding Unit) methods, and various embodiments of the present disclosure can be applied to any of these methods.

[0118] Meanwhile, if the terminal device (100) obtains control over the external device (300) and can control the display of the external device (300), there is a risk that a security issue may arise with respect to the external device (300) of the other party (2). For example, if the external device (300) is a mobile device, various data and information such as photos, emails, and messenger contents within the mobile device may be leaked. Even if the other party (2) allows control over his / her mobile device, he / she may not have considered the risk of personal data leakage, and therefore, precautions against this may be necessary.

[0119] Therefore, in order to solve the security problem of a terminal device controlling an external device, the following description assumes that the external device (300) is a mobile device.

[0120] FIG. 11 is a block diagram showing the configuration of a mobile device according to an embodiment of the present disclosure.

[0121] Referring to FIG. 11, the mobile device includes a memory (310) in which data and applications are stored, a processor (320), a display (330), and a communication unit (340).

[0122] When mirroring communication is performed for a second external device that is in a video call with a first external device, the processor (320) provides the screen of the display (330) to the second external device. In this state, when control over the mobile device (300) is granted to the first external device, the processor (320) allows only limited control by the first external device. That is, the processor (320) performs control operations according to user operations inputted through a touch screen or other user interface provided in the mobile device without restriction after user authentication (fingerprint recognition, password input, pattern input, facial recognition, etc.) is performed. Therefore, the user can freely use various data or applications stored in the mobile device.

[0123] However, if the processor (320) is granted control from the first external device in a video call with the second external device performing mirroring communication, and if a control signal transmitted from the first external device is received through the second external device, the processor (320) may not allow control of some data or applications.

[0124] Specifically, the processor (320) may allow control by a first external device based on a memory in which information about the data and applications is stored, for data and applications whose security level is below a preset standard among the data and applications.

[0125] If a control signal from the first external device is received for data and applications whose security level exceeds the preset standard, the processor (320) provides a message to the first external device through the second external device notifying that the control is not possible.

[0126] The security level can be set as a default or set by the user depending on the type of data and application. For example, a high security level can be set for data such as photos or video content, contact information, sent and received message contents, SNS applications, stock trading applications, financial applications, and email sending and receiving programs. Meanwhile, regarding security issues regarding personal information and various data and applications, the user (1) or the other party (2) can allow access only to specific individuals or specific accounts mutually approved through the user registration process, or can allow registration and access through a key authentication method during user registration. When a control signal for executing data or an application with a security level to which access is permitted is input, the processor (320) can control the display (330) to display a screen for viewing the data or to execute the selected application and display the execution screen. On the other hand, when a control signal is input to execute data or an application with a security level that does not allow access, the processor (320) can transmit a message to the terminal device (100) through the counterpart device (200) notifying that the data or application is uncontrollable.

[0127] The methods for securing external devices are not limited to those described above, and security issues can be addressed in various ways.

[0128] FIG. 12 is a flowchart for explaining a device control method of a terminal device that controls an external device according to one embodiment of the present disclosure.

[0129] Referring to FIG. 12, when a video call is performed between a terminal device (100) and a counterpart device (200), a video call screen including an image received from the counterpart device (200) is displayed on the display (130) of the terminal device (S1210).

[0130] When the counterpart device (200) performs mirroring communication to share the screen of the external device (S1220), information about the screen of the external device is received by the terminal device (100) through the counterpart device (200) and additionally displayed on the video call screen of the display (130) of the terminal device (S1230).

[0131] Thereafter, when a user operation for controlling an external device is input by a user (1) through a terminal device (100), a control signal corresponding to the user operation is generated, and the control signal is transmitted to the counterpart device (200) so that the counterpart device (200) transmits the control signal to the external device, thereby controlling the external device (S1240). Accordingly, the user can explain to the counterpart how to use the external device, or can take care of the counterpart by directly controlling the external device on behalf of the counterpart and playing the contents of the external device, or performing emergency contact, etc. Since the method for the user to obtain control has been specifically described in the above-described section, a duplicate description will be omitted.

[0132] Meanwhile, when there are multiple counterpart devices, and at least one of the multiple counterpart devices is connected to an external device and performs a mirroring operation, when a user operation for controlling the external device is input, the presence or absence of a device having control over the external device among the multiple counterpart devices is identified. If there is no specific device having control over the external device or the priority of the terminal device (100) is higher than the priority of the specific device, a control signal is generated and transmitted to the counterpart device to which the external device is connected.

[0133] Thereafter, if the priority of a specific device having control is higher than the priority of the terminal device (100), a guidance message is displayed informing that control of the external device is not possible. Since the message notification display has been specifically described in the above-described section, a duplicate explanation is omitted.

[0134] Figure 13 is a flowchart illustrating an example of a device control method in a state where multiple devices are connected to each other.

[0135] According to FIG. 13, a terminal device can perform a video call with multiple counterpart devices, and obtain control according to the priority of each device.

[0136] When a video call is initiated with multiple counterpart devices, the terminal device (100) can share the screens of each device (S1310).

[0137] In this state, if at least one of the multiple counterpart devices is connected to an external device and performs a mirroring operation, the terminal device (100) may also display the screen of the mirrored external device in addition to the video call screen.

[0138] In this state, when the user of the terminal device (100) inputs a user operation to control an external device mirrored on the counterpart device (S1320), the terminal device (100) performs an operation to acquire control (S1340). The process of acquiring control has been described in the various embodiments described above, so a redundant description will be omitted.

[0139] If the terminal device (100) does not have control rights to control the external device, only the display is shared (S1330). On the other hand, if the terminal device (100) has control rights to control the external device, it acquires control rights to control the display of the external device (S1340).

[0140] However, if there are multiple counterpart devices including the terminal device (100), the highest priority for acquiring control is required (S1350). If the terminal device (100) among the multiple devices does not have the highest priority for acquiring control compared to other counterpart devices, the request for permission to acquire control to the external device is retried through the counterpart device that performs mirroring communication with the external device until the highest priority is reached (S1360). On the other hand, if the terminal device (100) has the highest priority for acquiring control, it can control the display of the external device and transmit control data (S1370). Even if the terminal device (100) does not have the highest priority, it may acquire control if it has a higher priority than other devices that have requested control.

[0141] Meanwhile, the step of additionally displaying on the video call screen of the display (130) of the terminal device displays an image captured by the camera (10) of the terminal device (100) in a first area within the video call screen, displays an image received from the counterpart device in a second area within the video call screen, and displays the screen of the external device in a third area within the video call screen.

[0142] Additionally, the step of transmitting the control signal to the counterpart device includes adding a highlight to one of the first to third areas and displaying it. When a movement operation for moving the highlight is input, the highlight is moved to another area and displayed according to the movement operation. Thereafter, when a user selection is input while the highlight is added to the third area, it is recognized that a user operation for controlling the external device has been input.

[0143] Meanwhile, in the step of transmitting a control signal to the counterpart device, when a user selection is input while a highlight is added to the third area, a highlight is added to one of the multiple objects included in the screen of the external device and displayed, and when a move operation is input, the highlight is moved to another object among the multiple objects and displayed.

[0144] Afterwards, when a user selection is input while a highlight is added to one of the plurality of objects, a control signal is generated to perform an action corresponding to the object to which the highlight is added, and the generated control signal is transmitted to the counterpart device.

[0145] Meanwhile, the various embodiments described above may be applied to a product as an embodiment alone, but at least some of the contents may be implemented in combination with other embodiments of the present disclosure.

[0146] The present disclosure relates to a terminal device controlling an external device and a device control method. When a terminal device and multiple counterpart devices are connected and engaged in video calls and screen sharing, the present disclosure is effective in that the terminal device can control the display of at least one of the counterpart devices, with respect to the external device that is engaged in mirroring communication with the counterpart device. In other words, the present disclosure not only enables control of a remote counterpart device, but also enables control of other external devices that are engaged in mirroring communication with the counterpart device.

[0147] Meanwhile, the methods according to the various embodiments described above may be implemented in the form of program code for performing each step, stored on a recording medium, and distributed. In this case, a device equipped with the recording medium can perform the method of controlling a terminal device for controlling an external device described above.

[0148] Such recording media may be various types of computer-readable media, such as ROM, RAM, memory chips, memory cards, external hard drives, hard drives, CDs, DVDs, magnetic disks or magnetic tapes.

[0149] Although the preferred embodiments of the present disclosure have been illustrated and described above, the present disclosure is not limited to the specific embodiments described above, and various modifications may be made by a person skilled in the art to which the present disclosure pertains without departing from the gist of the present disclosure as claimed in the claims, and such modifications should not be understood individually from the technical idea or prospect of the present disclosure.

Claims

1. In the terminal device, Department of Communications; display; and Processor; including; The above processor, Controls the display to display a video call screen with a counterpart device (Correspondent Device) that performs a mirroring operation to share the screen of an external device, A terminal device that, when a user operation for controlling the external device is input, generates a control signal for the external device and transmits the control signal to the counterpart device through the communication unit so that the counterpart device transmits the control signal to the external device.

2. In paragraph 1, It further includes a memory for storing data on the image of the counterpart device and the screen of the external device; The above processor, Configure the video call screen including the image of the counterpart device and the screen of the external device, and control the display to display the configured video call screen. The above processor, A terminal device that controls the counterpart device to display a message inquiring whether to allow control of the external device when a user operation for controlling the external device is input, and generates the control signal when a response allowing control of the external device is received from the counterpart device.

3. In paragraph 1, The above processor, When the above counterpart device is a plurality of devices, and at least one of the plurality of counterpart devices is connected to an external device and performs the mirroring operation, if a user operation for controlling the external device is input, Identifying whether there is a device having control over the external device, and if there is no device having control or the priority of the terminal device is higher than the priority of the device having control, generating the control signal and transmitting it to the counterpart device to which the external device is connected; A terminal device that controls the display to display a guidance message notifying that control of the external device is impossible when the priority of the device having the above control authority is higher than the priority of the terminal device.

4. In paragraph 1, Including more cameras; The above processor transmits the image captured by the camera to an external server through the communication unit, The above processor, Receive data on a video call screen including a video captured by the camera, a video captured by the counterpart device, and a screen of the external device from the external server, and control the display to display the video call screen based on the data. A terminal device that, when a user operation for controlling the external device is input on the video call screen, generates a control signal corresponding to the user operation and transmits the control signal to the counterpart device through the communication unit.

5. In paragraph 2, Including cameras; The above processor, Based on the layout information stored in the memory, the image captured by the camera is displayed in a first area within the video call screen, the image received from the counterpart device is displayed in a second area within the video call screen, and the screen of the external device is displayed in a third area within the video call screen. Add a highlight mark to one of the first to third areas and display it, and when a movement operation for moving the highlight mark is input, move the highlight mark to another area and display it according to the movement operation. A terminal device that recognizes that a user operation for controlling the external device has been input when a user selection is input while the highlight display is added to the third area.

6. In paragraph 5, If the screen of the above external device contains multiple objects, The above processor, When the user selection is input while the highlight is added to the third area, the highlight is added to one of the plurality of objects and displayed, and when the move operation is input, the highlight is moved to another object among the plurality of objects and displayed. A terminal device, which, when the user selection is input while the highlight is added to one of the plurality of objects, generates a control signal to perform an action corresponding to the object to which the highlight is added, and transmits the control signal to the counterpart device through the communication unit so as to transmit the control signal to the external device.

7. For mobile devices, A communication unit for communicating with an external device; display; and Includes a processor, The above processor, when a mirroring function is performed for a second external device that is making a video call with a first external device, provides the screen of the display to the second external device through the communication unit, A mobile device that allows limited control by the first external device when control over the user terminal device is granted to the first external device.

8. In paragraph 7, memory where data and applications are stored; The above processor, For data and applications among the above data and applications whose security level is below the preset standard, control by the first external device is permitted based on the stored memory. A mobile device, wherein when a control signal from the first external device for data and applications exceeding the above-mentioned standard is received, a message notifying that the control is unavailable is provided to the first external device through the second external device.

9. For server devices, A communication unit for performing communication with multiple terminal devices; memory; and Processor; including; The above processor, When a video call is performed between the above multiple terminal devices, a video call screen including the video provided by each terminal device is generated and transmitted to each terminal device through the communication unit. When a mirroring operation for sharing the screen of an external device is performed on at least one terminal device among the plurality of terminal devices, information about the screen of the external device is received from the at least one terminal device through the communication unit and stored in the memory, the video call screen is reconstructed to add the stored screen of the external device, and the reconstructed video call screen is transmitted to each terminal device through the communication unit. A server device that, when a user operation for controlling the external device is transmitted from one of the plurality of terminal devices, generates a control signal for causing the external device to perform an operation corresponding to the user operation, and transmits the control signal to a terminal device mirrored with the external device through the communication unit.

10. In paragraph 9, The above memory stores information about the priority of each of the plurality of terminal devices, The above processor, A server device that, when there are multiple terminal devices transmitting user operations for controlling the external device, compares the priorities of the terminal devices based on the information stored in the memory, and grants control authority over the external device to a terminal device with a higher priority.

11. In a device control method of a terminal device, A step of displaying a video call screen with a counterpart device (Correspondent Device) performing a mirroring operation for sharing the screen of an external device; and A device control method, comprising: when a user operation for controlling the external device is input, a step of generating a control signal corresponding to the user operation and transmitting the control signal to the counterpart device so that the counterpart device transmits the control signal to the external device.

12. In paragraph 11, When the above counterpart device is a plurality of devices, and at least one of the plurality of counterpart devices is connected to an external device and performs the mirroring operation, if a user operation for controlling the external device is input, A step of identifying whether there is a device among the plurality of counterpart devices that has control over the external device, and if there is no device having the control or the priority of the terminal device is higher than the priority of the device having the control, generating the control signal and transmitting it to the counterpart device to which the external device is connected; and A device control method further comprising: a step of displaying a guidance message notifying that control of the external device is impossible if the priority of the device having the control right is higher than the priority of the terminal device.

13. In paragraph 11, The steps to be additionally displayed on the above video call screen are: Displaying an image captured by a camera of the terminal device in a first area within the video call screen, displaying an image received from the counterpart device in a second area within the video call screen, and displaying the screen of the external device in a third area within the video call screen. The step of transmitting the above control signal to the counterpart device is: A step of displaying a highlight mark by adding it to one of the first to third areas; When a movement operation for moving the above highlight display is input, a step for moving the above highlight display to another area according to the movement operation; and A device control method, comprising: a step of recognizing that a user operation for controlling the external device has been input when a user selection is input while the highlight display is added to the third area; 14. In paragraph 13, The step of transmitting the above control signal to the counterpart device is: When the user selection is input while the highlight mark is added to the third area, a step of adding the highlight mark to one of the plurality of objects included in the screen of the external device and displaying it; A step of moving the highlight display to another object among the plurality of objects when the above movement operation is input; When the user selection is input while the highlight is added to one of the plurality of objects, a step for generating a control signal to perform an action corresponding to the object to which the highlight is added; and A device control method further comprising a step of transmitting the control signal to the counterpart device.

15. A non-transitory computer-readable recording medium storing computer instructions that, when executed by a processor of a terminal device, cause the terminal device to perform an operation, wherein the operation is: A step of displaying a video call screen with a counterpart device (Correspondent Device) performing a mirroring operation for sharing the screen of an external device; and A non-transitory computer-readable recording medium, comprising: a step of generating a control signal corresponding to the user operation when a user operation for controlling the external device is input, and transmitting the control signal to the counterpart device so that the counterpart device transmits the control signal to the external device;

Images