Information processing terminal and program

Abstract

The information processing terminal includes: a display unit; a receiving unit that receives image data of images displayed on screens of a plurality of other terminals; a display control unit that displays, on a display screen of the display unit, each of images based on image data received from each of the terminals, a plurality of divided screen regions associated with each of the plurality of other terminals; a reception unit that receives designation of a position on the display screen; a calculation unit that specifies another terminal associated with the divided screen area to which the received position belongs, and calculates position information indicating the position on the screen of the specified another terminal corresponding to the position on the image displayed in the divided screen area; and a transmission unit that transmits the location information to the specified other terminal.

Description

Technical Field

[0001] The present invention relates to an information processing terminal for performing remote display and remote operation. Background Art

[0002] For the purpose of remotely operating an information processing terminal (referred to as a source terminal) while confirming the processing content of the terminal in a physically separate location, a remote terminal has been proposed: the remote terminal is used to connect to the source terminal, display a screen that is the same as the screen displayed by the source terminal (so-called mirroring), and accept user operations and provide them to the source terminal (remote operation) (Patent Document 1).

[0003] Prior art literature

[0004] Patent Literature

[0005] Patent Document 1: International Publication No. 2018 / 159579 Summary of the invention

[0006] Problems to be solved by the invention

[0007] Patent Document 1 describes connecting a remote terminal having a touch panel to a plurality of source terminals, determining which source terminal an input operation is directed to based on a touch position on the touch panel, and determining a destination for sending position information based on the determination result.

[0008] Among them, in recent years, there are many types of information processing terminals, and it is conceivable that the screen size, aspect ratio, and other display device hardware performance of the source terminal and the remote terminal may be different.

[0009] Furthermore, even if the source terminal and the remote terminal have the same display-related hardware performance, when multiple source terminals are connected simultaneously as described in Patent Document 1, the screen area is divided in the display device of the remote terminal to display the images sent from each source terminal. In this case, the display form (image size, etc.) of the image displayed on the source terminal and the image displayed on the remote terminal is different. In such a situation, when a position on the screen is specified on the remote terminal by a touch operation, etc., it cannot be guaranteed to match the position on the screen of the remote terminal. That is, the content of the touch operation performed on the remote terminal may not be correctly reproduced on the source terminal.

[0010] The purpose of the present invention is to reproduce the screen displayed on the screen of the source terminal at the remote terminal and to reproduce the operation content inputted at the remote terminal on the screen of the source terminal without depending on the hardware performance of the display device and the number of connected source terminals.

[0011] Means for solving problems

[0012] The present invention provides, in one embodiment, an information processing terminal comprising: a display unit; a receiving unit that receives image data representing an image displayed on the screens of multiple other terminals; a display control unit that displays images based on the image data received from the multiple other terminals, respectively, on multiple divided screen areas of the display screen of the display unit that correspond to the multiple other terminals; an accepting unit that accepts the designation of a position on the display screen; a calculating unit that determines the other terminal that corresponds to the divided screen area to which the accepted position belongs, and calculates position information representing the position on the screen of the determined other terminal that corresponds to the position on the image displayed in the divided screen area; and a sending unit that sends the position information to the determined other terminal.

[0013] In a preferred embodiment, the display control unit determines the user's operation content based on the position accepted by the acceptance unit, and based on the determined operation content, enlarges, reduces, rotates or scrolls the image displayed in the divided screen area to which the accepted position belongs. The information processing terminal also has a switching unit, which switches between an asynchronous mode and a synchronous mode based on an instruction from the user. In the asynchronous mode, an image that directly reflects the determined operation content in an image based on image data received from other terminals is displayed on the display unit, and position information based on the accepted position is not sent to the other terminal. In the synchronous mode, an image that does not directly reflect the operation content is displayed on the display unit, and position information based on the accepted position is sent to the other terminal.

[0014] In a preferred embodiment, the information processing terminal further includes a setting unit configured to set, among the plurality of divided screen regions, a valid region in which calculation of the position information is executed when a position is specified and an invalid region in which the calculation is not executed.

[0015] In a preferred embodiment, when a position in the invalid region is specified, the setting unit changes the invalid region to the valid region.

[0016] In a preferred embodiment, the display control unit determines a display size and a display direction based on the size and shape of each divided screen area while maintaining an aspect ratio of an image based on the image data.

[0017] In a preferred embodiment, the information processing terminal also has a storage unit, which stores display function information indicating the correspondence between the position on the image displayed on the screen of the other terminal and the position on the screen for each of the multiple other terminals, and the display control unit calculates the position information based on the display function information stored in correspondence with each other terminal determined during the communication connection.

[0018] In a preferred embodiment, the display control unit determines a size or a resolution of an image displayed on the display unit according to a data amount of the image data received from the plurality of other terminals.

[0019] In another aspect, the present invention provides a program for causing a computer having a display unit to execute the following steps: receiving image data representing an image displayed on a screen of multiple other terminals; displaying images based on the image data received from the multiple other terminals, respectively, on a plurality of divided screen areas corresponding to the multiple other terminals, respectively, on the display screen of the display unit; accepting a designation of a position on the display screen; determining the other terminal corresponding to the divided screen area to which the accepted position belongs, calculating position information representing the position on the screen of the determined other terminal corresponding to the position on the image displayed in the divided screen area; and sending the position information to the determined other terminal.

[0020] Effects of the Invention

[0021] According to the present invention, an image displayed on a screen of a source terminal can be reproduced on a remote terminal, and operation contents inputted on a remote terminal can be reproduced on a source terminal. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 is a diagram showing an example of the overall configuration of the information processing system 100 .

[0023] Figure 2 2 is a diagram showing an example of the configuration of the source terminal 10 .

[0024] Figure 3 2 is a diagram showing an example of the configuration of the information processing terminal 20 .

[0025] Figure 4 2 is a diagram showing an example of the function information management table Tb1.

[0026] Figure 5 : is a diagram showing an example of the window definition table Tb2.

[0027] Figure 6 It is a diagram showing an example of the screen allocation table Tb3.

[0028] Figure 7 2 is a diagram showing an example of the display state management table Tb4.

[0029] Figure 8 202 is a diagram showing a configuration example of a display screen SC of the display unit 202 .

[0030] Fig. 9 1 is a diagram showing an example of allocating an image of the source terminal 10 to divided screen areas.

[0031] Fig.10 3 is a diagram showing an example of an image GR1 displayed in the divided screen region CR1.

[0032] Fig.11 2 is a sequence diagram showing an example of the operation of the information processing system 100 during connection.

[0033] Fig.12 This is a flowchart showing an example of the flow of image processing operations of the information processing terminal 20 .

[0034] Fig.13 This is a flowchart showing an example of the flow of operations of the position information conversion process.

[0035] Fig.14 FIG. 2 is a diagram showing another example of allocating the image of the source terminal 10 to the divided screen areas. DETAILED DESCRIPTION

[0036] <Implementation Method>

[0037] <Configuration of Information Processing System>

[0038] Figure 1 1 is a diagram showing an example of the overall configuration of the information processing system 100. Figure 1 As shown, the information processing system 100 includes source terminals 10A and 10B (hereinafter, referred to as "source terminals 10" when they are not distinguished) and an information processing terminal 20. The source terminal 10 is a terminal that displays an image as an information resource (source), and the image as the information resource (source) is an operation object. The source terminal 10 is, for example, a monitor for a personal computer, a smart phone, a television, etc.

[0039] For example, Figure 1 The source terminal 10A shown is a 27-inch monitor for a personal computer. The image resolution of the source terminal 10A is 1920×1080, and its aspect ratio is 16:9.

[0040] and, Figure 1 The source terminal 10B shown is a smartphone having a size of 6.1 inches. The image resolution of the source terminal 10B is 1284×2778, and its aspect ratio is about 9:19.5.

[0041] In the following, for the sake of convenience, only source terminal 10A and source terminal 10B are shown as source terminals, but Figure 1The number of source terminals 10 of the information processing system 100 is not limited to two, and may be one, or may be three or more. Furthermore, the source terminal 10 may be formed by a combination of multiple devices. For example, the source terminal 10 may be obtained by installing a dongle equipped with an image processor in the multimedia interface of an LCD TV.

[0042] The information processing terminal 20 is a terminal that is connected to one or more source terminals 10 by wireless communication and remotely controls the source terminals 10. For example, Figure 1 The information processing terminal 20 shown is a tablet PC with a monitor size of 12 inches. The image resolution of the information processing terminal 20 is 1920×1200, and its aspect ratio is 16:10.

[0043] exist Figure 1 In the example shown, the information processing terminal 20 is connected to source terminals 10A and 10B, which are two different source terminals 10, and the images displayed by the source terminals 10A and 10B are displayed on the divided screens of the terminal. The user can operate the source terminals 10A and 10B by operating the respective areas of the divided screens of the information processing terminal 20.

[0044] <Structure of source terminal>

[0045] Figure 2 2 is a diagram showing an example of the configuration of the source terminal 10 . Figure 2 The source terminal 10 shown includes a communication unit 101 , a display unit 102 , a sound emitting unit 103 , a storage unit 104 , a voice control unit 105 , a display control unit 106 , an input unit 107 , and a control unit 110 .

[0046] The control unit 110 includes a CPU (Central Processing Unit), a ROM (Read Only Memory), and a RAM (Random Access Memory), and the CPU reads out and executes a computer program (hereinafter referred to as a program) stored in the ROM and the storage unit 104 to control each unit of the source terminal 10. The control unit 110 controls the sound playback unit 103 via the voice control unit 105 and controls the display unit 102 via the display control unit 106.

[0047] The sound emitting unit 103 is a device that emits sound under the control of the audio control unit 105 , and is, for example, a speaker, a headphone, or the like.

[0048] The sound control unit 105 is a device that controls the sound emitting unit 103 according to the instruction of the control unit 110, and is, for example, a sound controller.

[0049] The display unit 102 has a display screen such as a liquid crystal display, and displays images under the control of the display control unit 106 .

[0050] The display control unit 106 is a device that controls the display unit 102 according to the instruction of the control unit 110, such as a display controller or a graphic controller. The display control unit 106 may include a GPU (Graphics Processing Unit) or may be implemented as one or more CPUs having a rendering processing function.

[0051] The storage unit 104 is a storage device such as a semiconductor memory (Solid State Drive) or a hard disk drive (Hard Disk Drive), and stores various programs and data to be read by the CPU of the control unit 110 .

[0052] The input unit 107 includes operation buttons, a keyboard, a touch panel, and other operation elements (not shown) for performing various instructions, receives user operations, and sends signals corresponding to the operation contents to the control unit 110. The input unit 107 also includes an operation content determination unit 108 and a position information detection unit 109.

[0053] The position information detection unit 109 included in the input unit 107 detects position information indicating a position on the screen or image indicated by the user's operation. The position information detection unit 109 is, for example, the above-mentioned touch panel. The touch panel is formed of a transparent material and is overlapped on the display screen of the display unit 102. The position information detection unit 109 sends the detected position information to the operation content determination unit 108.

[0054] The operation content determination unit 108 included in the input unit 107 determines the user's operation accepted by the above-mentioned operating element. The operation content determination unit 108 receives a signal issued by the operating element in response to the user's operation, and determines the content of the operation performed by the user based on the pattern of the received signal, for example, by referring to a correspondence table stored in the storage unit 104. In addition, the operation content determination unit 108 uses the position information obtained from the above-mentioned position information detection unit 109 as a signal indicating the user's operation.

[0055] In addition, part or all of the functions of the input unit 107 may be implemented by the control unit 110 .

[0056] The communication unit 101 is a communication circuit connected to a communication line (not shown) in a wireless manner. The source terminal 10 exchanges information with other devices (ie, external devices including the information processing terminal 20) connected to the communication line via the communication unit 101 .

[0057] The communication unit 101 provides a function of connecting the source terminal 10 to a communication line in accordance with the so-called wireless LAN standard, such as IEEE802.11. In addition, the source terminal 10 may be directly connected to the information processing terminal 20 without passing through a communication line. For example, each source terminal 10 and the information processing terminal 20 may also have a function of connecting to each other in accordance with the standards of Near Field Communication (NFC), such as ISO / IEC18092 (NFCIP-1), ISO / IEC14443, ISO / IEC15693, or IEEE802.15. For example, the source terminal 10 and the information processing terminal 20 may also be connected using a display transmission technology based on one-to-one wireless communication such as Miracast (registered trademark) and AirPlay (registered trademark).

[0058] Furthermore, the source terminal 10 and the information processing terminal 20 may be directly connected to each other so as to be communicable via a wired method in accordance with IEEE802.3 or the like. In this case, the communication unit 101 is an interface for wired communication.

[0059] <Structure of Information Processing Terminal>

[0060] Figure 3 2 is a diagram showing an example of the configuration of the information processing terminal 20 . Figure 3 The illustrated information processing terminal 20 includes a control unit 200 , a communication unit 201 , a display unit 202 , a sound emitting unit 203 , a storage unit 204 , a voice control unit 205 , a receiving unit 206 , and a display control unit 212 .

[0061] The communication unit 201 is a communication circuit connected to a communication line (not shown) by wireless means. The information processing terminal 20 exchanges information with other devices (i.e., external devices including the source terminal 10) connected to the communication line through the communication unit 201. In addition, the communication unit 201 may also be an interface for wired communication.

[0062] The communication unit 201 includes a receiving unit 213 and a transmitting unit 214. The receiving unit 213 of the communication unit 201 receives image data representing images displayed on screens of the plurality of source terminals 10, respectively, from the plurality of source terminals 10. That is, the receiving unit 213 is an example of a receiving unit that receives image data representing images displayed on screens of a plurality of other terminals.

[0063] The transmission unit 214 of the communication unit 201 transmits information from the information processing terminal 20 to the above-mentioned plurality of source terminals 10 .

[0064] The control unit 200 includes a CPU, a ROM, and a RAM, and the CPU reads out and executes programs stored in the ROM and the storage unit 204 to control each unit of the information processing terminal 20 .

[0065] Furthermore, the control unit 200 controls the sound emitting unit 203 via the audio control unit 205 , and controls the display unit 202 via the display control unit 212 .

[0066] The sound emitting unit 203 is a device that emits sound under the control of the audio control unit 205 , and is, for example, a speaker, a headphone, or the like.

[0067] The voice control unit 205 is a device, such as a voice controller, that controls the sound playback unit 203. Specifically, the voice control unit 205 performs the following processing: voice signals are separated from the image signals received from each source terminal 10, and these voice signals are mixed as needed, or only the voice signal of one source terminal selected by the user among the multiple source terminals 10 is selected by the user, and the voice control unit 205 provides the processed voice signal to the sound playback unit 203.

[0068] The display unit 202 has a display screen such as a liquid crystal display, and displays images under the control of the display control unit 212 .

[0069] The display control unit 212 is a device that controls the display unit 202, and is, for example, a display controller or a graphic controller. The display control unit 212 may include a GPU, or may be implemented as one or more CPUs having a rendering processing function.

[0070] When the receiving unit 213 receives image data from a plurality of source terminals 10, the display control unit 212 divides the screen of the display unit 202 according to the number of source terminals 10. The divided screen regions (hereinafter, also referred to as "divided screen regions") are associated with the source terminals 10, respectively. The display control unit 212 displays images based on the image data received from the plurality of source terminals 10, respectively, in the divided screen regions associated with the source terminals 10.

[0071] In addition, the image based on the image data received from the source terminal 10 includes an image adjusted by image size change, enlargement / reduction, rotation (display direction change) and the like in the information processing terminal 20. In this case, these adjustments are sent to the source terminal 10, the source terminal 10 displays the adjusted image, and sends the image data representing the image to the information processing terminal 20 again. The image adjustment performed in the information processing terminal 20 also includes cropping of a portion, adjustment of the display position of the entire divided screen area, adjustment of the resolution / bit rate / frame rate and the like. For example, when the size of the divided screen area is smaller than the size of the image of the source terminal 10, the information processing terminal 20 crops a portion of the image.

[0072] That is, the display control unit 212 is an example of a display control unit that displays images based on image data received from a plurality of other terminals on a plurality of divided screen areas associated with the plurality of other terminals, respectively, on the display screen of the display unit.

[0073] The receiving unit 206 has operating elements (not shown) such as operation buttons, keyboards, and touch panels for performing various instructions, receives user operations, and sends signals corresponding to the operation contents to the control unit 200. In addition, part or all of the functions of the receiving unit 206 may also be implemented by the control unit 200.

[0074] Furthermore, the receiving unit 206 includes an operation content specifying unit 207 and a position information detecting unit 208 .

[0075] The position information detection unit 208 of the receiving unit 206 detects position information indicating a position on the screen or image indicated by the user's operation. The position information detection unit 208 is, for example, the above-mentioned touch panel. The touch panel can be formed of a transparent material and overlapped on the display screen of the display unit 202.

[0076] Furthermore, the position information detection unit 208 is not limited to a touch panel as long as it can detect the position on the display screen intended by the user. The position information detection unit 208 may be, for example, a mouse, a joystick, a trackball, a pointing stick, etc. The position information detection unit 208 sends the detected position information to the operation content determination unit 207.

[0077] The operation content determination unit 207 included in the acceptance unit 206 determines the user's operation accepted by the above-mentioned operating element. The operation content determination unit 207 receives a signal sent by the operating element according to the user's operation, and determines the content of the user's operation according to the pattern of the received signal, for example, by referring to the correspondence table stored in the storage unit 204.

[0078] Furthermore, the operation content specifying unit 207 accepts designation of a position on the screen of the display unit 202 from the user using the position information obtained from the position information detecting unit 208. That is, the accepting unit 206 including the operation content specifying unit 207 and the position information detecting unit 208 is an example of an accepting unit that accepts designation of a position on the display screen.

[0079] Here, when the above-mentioned accepting unit 206 accepts the designation of a position on the screen of the display unit 202 from the user and determines the content of the user's operation, the display control unit 212 changes the image displayed in the divided screen area of ​​the display unit 202 according to the determined content of the operation. For example, when the user's operation is to enlarge, reduce, rotate or scroll the image, the display control unit 212 enlarges, reduces, rotates or scrolls the image displayed in the divided screen area according to the content.

[0080] Note that the user's operation content is specified by the operation content specifying unit 207 of the accepting unit 206 , but may be specified by the display control unit 212 .

[0081] That is, the display control unit 212 is an example of a display control unit that determines the user's operation content based on the position accepted by the acceptance unit, and based on the determined operation content, enlarges, reduces, rotates or scrolls the image displayed in the divided screen area to which the accepted position belongs.

[0082] Furthermore, when the accepting unit 206 accepts the designation of a position on the screen of the display unit 202 from the user and determines the content of the operation by the user, the transmitting unit 214 of the communication unit 201 transmits the position information indicating the designated position to the source terminal 10 associated with the divided screen area including the position. That is, the transmitting unit 214 is an example of a transmitting unit that transmits the position information to the determined other terminal.

[0083] The storage unit 204 is a storage unit such as a solid state drive or a hard disk drive, and stores various programs, data, and the like.

[0084] Furthermore, the storage unit 204 stores a table group. Figure 3 The table group stored in the storage unit 204 shown is a function information management table Tb1, a window definition table Tb2, a screen allocation table Tb3, and a display state management table Tb4.

[0085] <Function Information Management Table>

[0086] Figure 4 2 is a diagram showing an example of the function information management table Tb1. Figure 4The function information management table Tb1 shown stores a “source terminal ID” column in which identification information for identifying each source terminal 10 is described and a “function information” column in which the function of the source terminal 10 is described in association with each other.

[0087] In the “source terminal ID” column, for example, the MAC (Media Access Control) address of the source terminal 10 is described.

[0088] The "Function Information" column includes "Image Resolution" and "Coordinate Resolution". The "Image Resolution" item includes information related to the image, such as the difference between the scanning method of progressive scanning and interlaced scanning, and the effective vertical resolution.

[0089] The "coordinate resolution" item includes information on the number of pixels arranged vertically and horizontally in the screen, for example. In addition, it is not necessary to uniquely determine the position on the screen (touch panel) from the position on the image based only on the image resolution information. This functional information is also called display function information.

[0090] The function information management table Tb1 stores the image resolution and coordinate resolution of the source terminal 10 for each source terminal 10. Therefore, the information processing terminal 20 that stores the function information management table Tb1 in the storage unit 204 can determine the correspondence between the position on the image displayed on the screen of the display unit 102 of each source terminal 10 and the position on the screen. That is, the storage unit 204 that stores the function information management table Tb1 is an example of a storage unit that stores display function information indicating the correspondence between the position on the image displayed on the screen of the terminal and the position on the screen for each of a plurality of other terminals.

[0091] The function information management table Tb1 may be updated each time the information processing terminal 20 is connected to the source terminal 10. In this case, the control unit 200 of the information processing terminal 20 determines the display function information stored in association with the source terminal 10 when communicating with the source terminal 10.

[0092] <Window definition table>

[0093] Figure 5 : is a diagram showing an example of the window definition table Tb2. Figure 5 The window definition table Tb2 shown stores the "divided screen area" column, the "area position, shape, size" column, and the "valid / invalid designation" column in association with each other.

[0094] The “divided screen area” column stores identification information (here, a serial number) assigned to each of the divided screen areas of the display screen divided in the display unit 202 of the information processing terminal 20 .

[0095] The "region position, shape, and size" column stores information on the position, shape, and size of the divided screen region identified by the corresponding serial number.

[0096] The "valid / invalid designation" column stores whether the divided screen area identified by the corresponding serial number is valid or invalid.

[0097] <Screen Allocation Table>

[0098] Figure 6 It is a diagram showing an example of the screen allocation table Tb3. Figure 6 The screen allocation table Tb3 shown in the figure stores the "divided screen area" column and the "source terminal ID" column in correspondence. The "divided screen area" column of the screen allocation table Tb3 corresponds to the "divided screen area" column of the window definition table Tb2. In addition, the "source terminal ID" column of the screen allocation table Tb3 corresponds to the "source terminal ID" column of the function information management table Tb1. That is, the function information management table Tb1 and the window definition table Tb2 are associated with each other through the screen allocation table Tb3.

[0099] <Display Status Management Table>

[0100] Figure 7 2 is a diagram showing an example of the display state management table Tb4. Figure 7 The display state management table Tb4 shown stores the "divided screen area" column and the "display state" column in association with each other. Here, the "divided screen area" column of the display state management table Tb4 corresponds to the "divided screen area" column of the window definition table Tb2.

[0101] Figure 7 The “display state” column of the display state management table Tb4 shown has an “enlargement ratio / reduction ratio” item, a “center position of image” item, and a “direction” item.

[0102] The "enlargement ratio / reduction ratio" item indicates the enlargement ratio or reduction ratio of the image displayed in the divided screen area identified by the corresponding serial number. The "enlargement ratio / reduction ratio" item stores, for example, a numerical value representing the ratio of the size of the image of the information processing terminal 20 to the size of the image of the source terminal 10 as a percentage.

[0103] The item “center position of image” stores the center position of an image (also referred to as an original image) representing the display screen of the source terminal 10 displayed in the divided screen area identified by the corresponding serial number.

[0104] The "Orientation" item stores the orientation of the image displayed in the divided screen area identified by the corresponding serial number relative to the original image. For example, when the top, bottom, left, and right of the image displayed in the divided screen area are consistent with those of the original image, the item is "Original". On the other hand, when the image displayed in the divided screen area is an image obtained by rotating the original image 90 degrees to the left, the item is "Rotated 90 degrees to the left".

[0105] <Functional Structure of Control Unit>

[0106] Figure 3 The control unit 200 shown in the figure functions as a calculation unit 209 , a mode switching unit 210 , and a setting unit 211 by executing the above-mentioned program.

[0107] The setting unit 211 implemented by the control unit 200 updates the window definition table Tb2 contained in the storage unit 204 according to the instruction received from the user by the receiving unit 206. Thus, the setting unit 211 sets the designation of whether to enable or disable for each divided screen area.

[0108] The divided screen area designated as valid is a divided screen area for which calculation of position information indicating a position is performed when a position is designated. On the other hand, the divided screen area designated as invalid is a divided screen area for which calculation of position information indicating a position is not performed when a position is designated.

[0109] That is, the setting unit 211 is an example of a setting unit that sets a valid area in which calculation of position information is performed when a position is specified and an invalid area in which the calculation is not performed, among a plurality of divided screen areas.

[0110] Furthermore, as described above, the designation of validity or invalidity may be performed for each divided screen area, but validity / invalidity may be set individually within one divided area.

[0111] The mode switching unit 210 implemented by the control unit 200 switches the display mode when the reception unit 206 identifies the content of the user's operation and the identified content of the operation indicates a mode switch. The display modes are an asynchronous mode and a synchronous mode.

[0112] The asynchronous mode is a mode in which, according to an instruction from the user, processing is performed on an image based on image data received from the source terminal 10, and position information based on the received position is not sent to the source terminal 10. In addition, the display control unit 212 displays the image after the above processing, that is, the image inside the information processing terminal 20 directly reflecting the user's operation content, on the display unit 202.

[0113] The synchronous mode is a mode in which position information based on the received position is transmitted to the source terminal 10 without reflecting processing in accordance with instructions from the user inside the information processing terminal 20 for an image based on image data received from the source terminal 10 .

[0114] That is, the mode switching unit 210 is an example of a switching unit that switches between an asynchronous mode and a synchronous mode according to instructions from a user, in which an image that directly reflects the determined operation content in an image based on image data received from other terminals is displayed on the display unit, and position information based on the accepted position is not sent to other terminals, and in which a synchronous mode an image that does not directly reflect the operation content is displayed on the display unit, and position information based on the accepted position is sent to other terminals.

[0115] By switching the mode of the mode switching unit 210, the information processing terminal 20 can distinguish between "full mirroring" in which various operations such as touch, long press, pinch in, and pinch out performed on the display image of the terminal are reflected in the image displayed on the screen of the source terminal 10 and "incomplete mirroring" in which various operations such as touch, long press, pinch in, and pinch out performed on the display image of the terminal are not reflected in the image displayed on the screen of the source terminal 10. For example, the following usage method is considered: when the purpose is to monitor the processing result of the source terminal 10, the synchronous mode is selected to synchronize the content displayed on the screen of the source terminal 10 with the display content displayed on the screen of the information processing terminal 20 in real time. On the other hand, when it is desired to carefully confirm the content displayed on the screen of the source terminal 10, the asynchronous mode is selected, and after performing operations such as enlargement and scrolling and performing necessary confirmation, it is changed to the synchronous mode again. In the asynchronous mode, there is no need to communicate with the source terminal 10, so compared with the asynchronous mode, the display processing such as enlargement and scrolling is accelerated, so the operation response is improved. In this way, by providing a function for appropriately switching modes, user convenience is improved.

[0116] Figure 8 2 is a diagram showing a configuration example of a display screen SC of the display unit 202. The display unit 202 of the information processing terminal 20 has a display screen SC such as a liquid crystal display. The display screen SC has an area OR that receives instructions for the application itself executed in the information processing terminal 20, and an area CR that receives instructions for an image displayed on the source terminal 10. Figure 8 As shown, when two source terminals 10 are operated, the region CR is divided into a divided screen region CR1 and a divided screen region CR2.

[0117] The area OR has a mode switching button IC1. When the user presses the mode switching button IC1, the receiving unit 206 determines the content of the operation as an "operation to switch modes". As a result, the mode switching unit 210 switches the display mode of the display unit 202. The display to be switched can be all the divided screen areas or the divided screen area that has just been operated. In addition, the display to be switched can also be the divided screen area that has been "validly designated" as described above.

[0118] When the accepting unit 206 accepts the designation of a position on the screen of the display unit 202 from the user, the calculating unit 209 identifies the source terminal 10 associated with the divided screen area to which the accepted position belongs, by referring to the above-mentioned table group stored in the storage unit 204. Then, the calculating unit 209 calculates the position information indicating the position on the screen of the source terminal 10 corresponding to the above-mentioned position on the image displayed in the divided screen area, based on the above-mentioned table group.

[0119] That is, the calculation unit 209 is an example of a calculation unit that determines other terminals that have established a correspondence with the divided screen area to which the accepted position belongs, and calculates position information representing the position on the screen of the determined other terminal corresponding to the position on the image displayed in the divided screen area.

[0120] In addition, there is a case where any table in the table group stored in the storage unit 204 is updated every time the information processing terminal 20 is connected to the source terminal 10. In this case, the calculation unit 209 determines the display function information of the source terminal 10 when communicating with the source terminal 10, and calculates the position information based on the display function information. That is, the calculation unit 209 is an example of a calculation unit that calculates the position information indicating the position of the terminal on the screen based on the display function information that is determined at the time of communication connection and stored in correspondence with each other terminal.

[0121] Fig. 9 1 is a diagram showing an example of allocating an image of the source terminal 10 to divided screen areas. As described above, the area CR is divided into the divided screen area CR1 and the divided screen area CR2. The divided screen area CR1 has a width of W1 and a height of H1. Also, the divided screen area CR2 has a width of W2 and a height of H2.

[0122] The calculation unit 209 refers to the screen allocation table Tb3 to identify the identification information (source terminal ID) of the source terminal 10 associated with each divided screen area. The calculation unit 209 also refers to the function information management table Tb1 to identify the function information of each source terminal 10.

[0123] Then, the calculation unit 209 makes the size of the image displayed on each source terminal 10 correspond to the height and width of the corresponding divided screen area according to whether the aspect ratio is maintained and whether the display direction is maintained, and calculates the representative position (here, the center coordinates) and the enlargement / reduction ratio of the image. The calculated representative position and enlargement / reduction ratio are stored in the display state management table Tb4.

[0124] Fig. 9 The divided screen area CR1 is shown Figure 1 For example, when the entire image on the screen of the source terminal 10A is set to be displayed in the divided screen area CR1 without changing the aspect ratio or rotating (keeping the original orientation), first, the calculation unit 209 calculates the aspect ratio of the width W1 and the height H1 of the divided screen area CR1, and obtains the image resolution from the function information of the source terminal 10A.

[0125] Here, if the image resolution of the source terminal 10A is "1080p", the aspect ratio of the image is determined to be 16: 9. The calculation unit 209 compares the aspect ratio with the aspect ratio of the divided screen area CR1, and calculates the enlargement / reduction ratio and the representative position so that the image of the source terminal 10A fits exactly within the divided screen area CR1.

[0126] That is, the calculation unit 209 is an example of a calculation unit that determines the display size and display direction while maintaining the aspect ratio of the image based on the image data, based on the size and shape of each divided screen area.

[0127] When the divided screen area CR1 is made to display the image of the source terminal 10A, Fig. 9 As shown, the calculation unit 209 calculates the height CH1, width CW1, enlargement / reduction ratio, representative position, etc., and updates the display state management table Tb4 based on the calculation unit 209. In addition, the calculation unit 209 may store the position, shape, and size of the image displayed in the divided screen area CR1 in the window definition table Tb2.

[0128] Thus, when the accepting unit 206 accepts designation of a position on the screen of the display unit 202 from the user, the calculation unit 209 can calculate position information indicating the position on the screen of the source terminal 10 corresponding to the position.

[0129] In the above example, the calculation unit 209 calculates the maximum enlargement / reduction ratio for the entire display in the divided screen area when setting the image on the screen of the source terminal 10 to be displayed in the corresponding divided screen area as a whole, but different values ​​may be calculated in other settings. For example, when the enlargement / reduction ratio is determined, the calculation unit 209 may calculate the display position corresponding thereto. Furthermore, when setting is made to directly use the size of the original image of the source terminal 10, the calculation unit 209 may calculate the display position corresponding thereto. Furthermore, when setting is made to prioritize either the vertical or horizontal direction, the calculation unit 209 may calculate the maximum enlargement / reduction ratio that matches the length of the prioritized direction with the length of the divided screen area.

[0130] Furthermore, the calculation unit 209 calculates the enlargement / reduction ratio, the representative position, and the like of the image displayed in the divided screen area based on the operation content specified by the operation content specifying unit 207 of the receiving unit 206 .

[0131] Fig.10 2 is a diagram showing an example of an image GR1 displayed in the divided screen area CR1. The calculation unit 209 calculates the enlargement / reduction ratio, representative position, etc. according to the instruction, while displaying the image GR1 of the source terminal 10A in the divided screen area CR1 without changing the aspect ratio or rotating it. As a result, the information processing terminal 20 displays the image GR1 in the divided screen area CR1. Fig.10 As shown in (a) of FIG. 8 , the image GR1 is displayed in the divided screen area CR1 .

[0132] For example, when the receiving unit 206 receives a pinch operation from the user, the calculation unit 209 calculates the representative position according to the enlargement / reduction ratio corresponding to the operation. The pinch operation is an operation of bringing multiple fingers close together in order to capture the screen. As a result, the information processing terminal 20 Fig.10 As shown in (b) of FIG. 8 , the image GR1 is displayed in the divided screen area CR1 .

[0133] Furthermore, for example, when the receiving unit 206 receives a downward flick operation from the user, the calculation unit 209 calculates the representative position based on the operation. Fig.10 As shown in (c) of FIG. 8 , the image GR1 is moved to the lower end of the divided screen area CR1 and displayed.

[0134] Furthermore, for example, when the receiving unit 206 receives a gesture operation of rotating 90 degrees to the left from the user, the calculation unit 209 calculates the enlargement / reduction ratio, the representative position, etc. according to the operation. As a result, the information processing terminal 20 Fig.10As shown in (d) of FIG. 1 , the image GR1 is rotated 90 degrees to the left relative to the original orientation, and is enlarged or reduced to match the size of the divided screen area CR1 and displayed.

[0135] Furthermore, for example, when the receiving unit 206 receives a pinch-out operation from the user, the calculating unit 209 calculates the representative position according to the zoom-in / zoom-out ratio corresponding to the operation. The pinch-out operation is an operation of enlarging the screen by spreading a plurality of fingers apart so as to enlarge the operation object on the screen. As a result, the information processing terminal 200 Fig.10 As shown in (e), the image GR1 is compared Fig.10 The image GR1 shown in (a) is displayed in an enlarged manner in the divided screen area CR1.

[0136] Here, for example, when the receiving unit 206 receives a flick operation to the left from the user, the calculation unit 209 calculates the representative position based on the operation. Fig.10 The image GR1 is scrolled to the left as shown in (f) of FIG. The scrolling of the image GR1 stops when the right end thereof appears at the right end of the divided screen area CR1.

[0137] <Operation when connecting to information processing system>

[0138] Fig.11 1 is a sequence diagram showing an example of the operation during connection of the information processing system 100. A plurality of source terminals 10 each transmit a wireless signal including identification information of the terminal (step S601). The information processing terminal 20 as a remote operation terminal receives the wireless signal and requests connection to the source terminal 10 that transmitted the wireless signal (step S602).

[0139] The information processing terminal 20 confirms the source terminal ID which is identification information of the source terminal 10 (step S603), and reads out the function information (display function information) from the function information management table Tb1 (step S604).

[0140] For example, when the function information management table Tb1 does not store the function information of the source terminal 10, the information processing terminal 20 requests the function information from the source terminal 10 (step S605). The source terminal 10 sends the function information to the information processing terminal 20 in response to the request (step S606). Thus, the information processing terminal 20 and the source terminal 10 establish a connection for sharing images, voices, dynamic images, etc. with each other (step S607).

[0141] <Image Processing Operations of Information Processing Terminal>

[0142] Fig.122 is a flowchart showing an example of the flow of image processing operations of the information processing terminal 20. The control unit 200 of the information processing terminal 20 monitors the receiving unit 206 and determines whether a touch operation of the user is detected (step S502). The control unit 200 continues the determination while determining that no touch operation is detected (step S502; No).

[0143] On the other hand, when it is determined that a touch operation is detected (step S502; yes), the control unit 200 determines whether the position touched by the user is the position of a designated operation icon (mode switching button IC1, etc.) (step S504). When it is determined that the touched position is the position of a designated operation icon (step S504; yes), the control unit 200 performs processing corresponding to each touched operation icon (step S506), and returns the processing to step S502.

[0144] On the other hand, if it is determined that the touched position is not the position of the designated operation icon (step S504; No), the control unit 200 determines whether the touched position is within the effective area (step S508). If it is determined that the touched position is not within the effective area (step S508; No), the control unit 200 ignores the user's touch operation (step S510) and returns the process to step S502.

[0145] In step S510, the control unit 200 may also designate the touched area as valid. In this case, the information processing terminal 20 changes the invalid area whose position is designated by touch or the like to the valid area. That is, the information processing terminal 20 is an example of a setting unit that changes the invalid area to the valid area when the position in the invalid area is designated. Thus, the information processing terminal 20 can switch the area from invalid to valid through a simple operation such as touch.

[0146] In this case, before the invalid area whose position is designated is changed to the valid area, another divided screen area which is the valid area may be changed to the invalid area.

[0147] When it is determined that the touched position is within the valid area (step S508 ; Yes), the control unit 200 determines whether the synchronization mode is set for the divisional screen area including the touched position (step S512 ).

[0148] When it is determined that the divisional screen area including the touched position is set to the synchronization mode (step S512; yes), the control unit 200 converts the position information indicating the touched position (step S514). That is, the control unit 200 refers to the table group of the storage unit 204 and converts the position information of the display unit 202 of the position touched by the user into the position information of the display unit 102 of the source terminal 10. Then, the control unit 200 transmits the converted position information from the transmission unit 214 of the communication unit 201 to the source terminal 10 (step S516).

[0149] When it is determined that the divisional screen area including the touched position is not set in synchronization mode (step S512 ; No), the control unit 200 specifies the content of the accepted operation (step S518 ), and performs drawing processing according to the content of the operation (step S520 ).

[0150] Fig.13 2 is a flowchart showing an example of the flow of the operation of the position information conversion process. The control unit 200 of the information processing terminal 20 refers to the table group stored in the storage unit 204, and determines the position on the display screen of the display unit 202 of the information processing terminal 20, for example, designated by an operation received from the user (step S702). Next, the control unit 200 converts the determined position into coordinates defined in the divided screen area to which the position belongs (step S704).

[0151] Then, the control unit 200 converts the coordinates obtained by converting the above position into a position on the image (i.e., content) displayed in the divided screen area (step S706). Then, the control unit 200 converts the position on the image into a position on the screen of the source terminal 10 (step S708).

[0152] Thus, the information processing terminal 20 converts the position indicated by the operation received from the user to the position on the screen of the source terminal 10. Therefore, the source terminal 10 receiving the converted position information from the information processing terminal 20 can process the position designated by the user of the information processing terminal 20 as if it were designated at the source terminal 10.

[0153] With the above-described structure, the information processing system 100 can reproduce the contents (still images, moving images) displayed on the respective screens of the source terminal 10 on the information processing terminal 20 (remote operation terminal), and can reproduce the operation contents inputted into the information processing terminal 20 on the source terminal 10.

[0154] In particular, even if the screens of the plurality of source terminals 10 differ from the screen of the information processing terminal 20 in terms of display function such as size or shape (aspect ratio), the information processing terminal 20 can appropriately display the image of the source terminal 10. Furthermore, operations input to the information processing terminal 20 can be appropriately reflected in the source terminal 10.

[0155] Furthermore, as long as the source terminal 10 has the function of outputting content and receiving converted location information, it can equally process the location information received from the information processing terminal 20 and the location information indicated by touch operations and mouse operations performed on the screen of the terminal itself (when the source terminal 10 is a dongle, etc., the screen of a display device directly connected to the terminal itself).

[0156] <Modification>

[0157] The control unit 110 and the control unit 200 have a CPU as a general-purpose processor, but may also have other processors. The other processors used here may be, for example, FPGA (Field Programmable Gate Array), ASIC (Application Specific Integrated Circuit) or other programmable logic devices.

[0158] In the above-described embodiment, the control unit 200 functions as the calculation unit 209, but the display control unit 212 may function as the calculation unit 209. In this case, the display control unit 212 is an example of a display control unit that determines the display size and display direction while maintaining the aspect ratio of the image based on the image data, based on the size and shape of each divided screen area.

[0159] In summary, in the information processing system of the present invention, it is sufficient to perform the following steps: connect a computer having a display unit and one or more other terminals, receive image data representing an image displayed on a screen of the aforementioned one or more other terminals; display images based on the image data received from the aforementioned one or more other terminals respectively on the display screen of the aforementioned display unit in divided screen areas corresponding to the aforementioned one or more other terminals; accept the designation of a position on the aforementioned display screen; determine the other terminal corresponding to the divided screen area to which the accepted position belongs, calculate position information representing the position on the screen of the determined other terminal corresponding to the position on the image displayed in the divided screen area; and send the aforementioned position information to the determined other terminal.

[0160] Furthermore, when functioning as the calculation unit 209 described above, the display control unit 212 is an example of a display control unit that calculates position information based on display function information stored in association with each other terminal identified at the time of communication connection.

[0161] In the above-mentioned embodiment or modification, the control unit 200, for example, refers to the function information management table Tb1 when communicating with the source terminal 10 to determine the display function information of the source terminal 10. However, the control unit 200 may request the display function information from the source terminal 10 when communicating with the source terminal 10. Furthermore, the control unit 200 may determine the display function information of the source terminal 10 based on the data volume of the image data received from the source terminal 10.

[0162] This determination may be made by the display control unit 212. That is, in this case, the display control unit 212 is an example of a display control unit that determines the size or resolution of an image displayed on the display unit according to the data amount of image data received from another terminal.

[0163] For example, when performing streaming playback of a moving image, the information processing terminal 20 may reduce the resolution of the image displayed in the divided screen area corresponding to the source terminal 10 after the total amount of image data received per unit time from the source terminal 10 reaches a predetermined value. In this case, the information processing terminal 20 can reduce the load of image processing.

[0164] In the above embodiment, when two source terminals 10 are operated, region CR is divided into divided screen region CR1 and divided screen region CR2 arranged on the left and right. However, the number of source terminals 10 to which region CR set on display unit 202 of information processing terminal 20 is allocated is not limited to two.

[0165] The information processing terminal 20 can allocate one source terminal 10 to one region CR. In this case, the region CR is not divided (partitioned) but directly allocated to one source terminal 10. Furthermore, the information processing terminal 20 can divide the region CR according to the number of source terminals 10 and allocate three or more source terminals 10.

[0166] Furthermore, when the region CR is divided into a plurality of divided screen regions, each divided screen region can be arranged in various ways.

[0167] Fig.14 FIG. 2 is a diagram showing another configuration example in which the image of the source terminal 10 is allocated to the divided screen areas. The information processing terminal 20 may also allocate the plurality of divided screen areas in a horizontal manner, as shown in FIG. Fig.14 As shown in (a) of FIG. 1 , a plurality of divided screen areas are arranged so that one is included in another. Fig.14In the example shown in (a), the divided screen area CR2 is included in the divided screen area CR1. In this example, the divided screen area CR1 and the divided screen area CR2 are in a so-called "parent-child" configuration relationship. In addition, the divided screen areas are not limited to completely separated configurations or included configurations, but may also be partially repeated configurations (so-called "overlapping" configurations). When a certain divided screen area is in an overlapping or included relationship with other divided screen areas, valid or invalid designations can be set in specific areas within the divided screen areas.

[0168] Furthermore, the information processing terminal 20 may divide the area CR into areas equal to or greater than the number of the source terminals 10, and assign the source terminals 10 to the divided screen areas. In other words, an area in which the screen of any connected source terminal is not displayed may be set within the area CR. For example, Fig.14 As shown in (b), the information processing terminal 20 divides the area CR into three divided screen areas CR1, CR2, and CR3, and assigns the source terminal 10 to two of the divided screen areas. For example, when the source terminal 10A is assigned to the divided screen area CR1 and the source terminal 10B is assigned to the divided screen area CR2, the divided screen area CR3 becomes an area that is not assigned to any source terminal 10. Since the divided screen area CR3 is not assigned to the source terminal 10, even if the user performs an operation, there is no source terminal 10 that reflects the operation.

[0169] Furthermore, the divided screen area to which the source terminal 10 is not assigned may be arranged between the divided screen areas to which the source terminal 10 is assigned. Fig.14 As shown in (d), the information processing terminal 20 arranges the divided screen area CR3 not allocated to any source terminal 10 between the divided screen area CR1 and the divided screen area CR2. This reduces the possibility that a swipe operation on one divided screen area may intrude into the other divided screen area and be misrecognized.

[0170] For example, suppose in Fig.14In the screen shown in (d), the information processing terminal 20 detects the user's finger touching the divided screen area CR1. When the control unit 200 of the information processing terminal 20 detects that the finger has moved from the divided screen area CR1 to the divided screen area CR2 via the divided screen area CR3, the control unit 200 of the information processing terminal 20 (1) detects the start of the contact between the finger and the divided screen area CR1, (2) detects the finger crossing the divided screen area CR3 while touching the divided screen area CR3, and (3) detects the finger entering the divided screen area CR2 from the boundary with the divided screen area CR3. The pattern consisting of the combination of the order of the detection and the respective times can be compared with a predetermined condition, and the control unit 200 invalidates the operation on the divided screen area CR2 and judges only the operation on the divided screen area CR1 as valid.

[0171] That is, by providing a divided screen area that is not allocated to any source terminal 10 and limiting the screen area that accepts operations, the information processing terminal 20 can prevent erroneous operations such as those during slide operations.

[0172] The program executed by the control unit 200 of the information processing terminal 20 can be provided in a state stored in a recording medium readable by a computer device such as a magnetic recording medium such as a magnetic tape and a disk, an optical recording medium such as an optical disk, an optical magnetic recording medium, or a semiconductor memory. In addition, the program can also be downloaded via a communication line such as the Internet.

[0173] The program executed by the control unit 200 is an example of a program for causing a computer having a display unit to perform the following steps: receiving image data representing an image displayed on a screen of one or more other terminals; displaying images based on the image data received from one or more other terminals, respectively, on a plurality of divided screen areas corresponding to the display screen of the display unit and corresponding to the one or more other terminals; accepting designation of a position on the display screen; determining the other terminal corresponding to the divided screen area to which the accepted position belongs, calculating position information representing the position on the screen of the determined other terminal corresponding to the position on the image displayed in the divided screen area; and sending the position information to the determined other terminal.

[0174] Description of symbols

[0175] 10 (10A, 10B): source terminal; 100: information processing system; 101: communication unit; 102: display unit; 103: sound playback unit; 104: storage unit; 105: voice control unit; 106: display control unit; 107: input unit; 108: operation content determination unit; 109: position information detection unit; 110: control unit; 20: information processing terminal; 200: control unit; 201: communication unit; 202: display unit; 203: sound playback unit; 204: storage unit; 205: voice control unit; 206: receiving unit; 207: operation content determination unit; 208: position information detection unit; 209: calculation unit; 210: mode switching unit; 211: setting unit; 212: display control unit; 213: receiving unit; 214: sending unit; Tb1: function information management table; Tb2: window definition table; Tb3: screen allocation table; Tb4: display status management table.

Claims

1. An information processing terminal, comprising: Display unit; a receiving unit that receives image data representing images displayed on screens of a plurality of other terminals; a display control unit for displaying images based on the image data received from the plurality of other terminals, respectively, on a plurality of divided screen areas respectively corresponding to the plurality of other terminals, on the display screen of the display unit; an accepting unit that accepts a designation of a position on the display screen; a calculation unit that determines another terminal that has established a correspondence with the divided screen area to which the received position belongs, and calculates position information indicating a position on the screen of the determined other terminal that corresponds to a position on the image displayed in the divided screen area; and A sending unit sends the location information to the determined other terminal.

2. The information processing terminal according to claim 1, wherein: The display control unit determines the operation content of the user according to the position accepted by the acceptance unit, and according to the determined operation content, enlarges, reduces, rotates or scrolls the image displayed in the divided screen area to which the accepted position belongs, The information processing terminal also has a switching unit, which switches between an asynchronous mode and a synchronous mode according to an instruction from a user. In the asynchronous mode, an image that directly reflects the determined operation content in an image based on image data received from other terminals is displayed on the display unit, and position information based on the accepted position is not sent to the other terminal. In the synchronous mode, an image that does not directly reflect the operation content is displayed on the display unit, and position information based on the accepted position is sent to the other terminal.

3. The information processing terminal according to claim 1, wherein: The information processing terminal further includes a setting unit that sets, among the plurality of divided screen regions, a valid region in which calculation of the position information is executed when a position is specified and an invalid region in which the calculation is not executed.

4. The information processing terminal according to claim 3, wherein: The setting unit changes the invalid region to the valid region when a position within the invalid region is specified.

5. The information processing terminal according to claim 1, wherein: The display control unit determines a display size and a display direction while maintaining an aspect ratio of an image based on the image data, based on the size and shape of each divided screen area.

6. The information processing terminal according to claim 1, wherein: The information processing terminal further includes a storage unit that stores, for each of the plurality of other terminals, display function information indicating a correspondence between a position on an image displayed on a screen of the other terminal and a position on the screen. The display control unit calculates the position information based on the display function information stored in association with each other terminal identified during communication connection.

7. The information processing terminal according to claim 1, wherein: The display control unit determines a size or a resolution of an image displayed on the display unit according to a data amount of the image data received from the plurality of other terminals.

8. A program for causing a computer having a display unit to execute the following steps: receiving image data representing images displayed on screens of a plurality of other terminals; displaying images based on the image data received from the plurality of other terminals on a plurality of divided screen areas respectively corresponding to the plurality of other terminals on the display screen of the display unit; accepting designation of a position on the display screen; determining another terminal corresponding to the divided screen area to which the received position belongs, and calculating position information indicating the position on the screen of the determined other terminal corresponding to the position on the image displayed in the divided screen area; as well as The location information is sent to the determined other terminal.

Images