How to remotely control your device

Abstract

To enable remote control of a remote terminal that accepts only relative position information to be performed using an external input device that outputs absolute positions. [Solution] A first terminal (10) that communicates with a second terminal (20) that displays a mouse pointer at a position on a display screen based on the amount of change in the position of an input control and that displays the display screen of the second terminal (20) on its own terminal transmits first operation information to the second terminal (20) for moving the pointer from the initial position of the pointer on the display screen of the other terminal to a position on the screen of the other terminal that corresponds to the position on the display screen of its own terminal specified by the user. In addition, the first terminal (10) transmits second operation information to the second terminal (20) for moving the pointer from a position on the display screen of its own terminal that has been released from the user's specification to the initial position.

Description

[Technical Field]

[0001] The present invention relates to the remote control of a terminal. [Background technology]

[0002] There are devices that have a mirroring function that displays the screen displayed on the terminal to be remotely controlled (hereinafter referred to as the remote terminal) on the display screen of the own terminal, and a function that allows the user to remotely control the remote terminal by sending to the remote terminal position information indicating a specified position on the display screen of the own terminal by touching a touch panel, etc. (Patent Document 1, etc.). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2023-177072 Summary of the Invention [Problem to be solved by the invention]

[0004] Depending on the type of remote terminal, the specifications of the remote terminal's OS (Operating System) may limit the information that can be received as information representing the user's operation to relative position information (in other words, the amount of movement of the operator). In other words, to remotely control this type of remote terminal, the only option is to connect a mouse to the terminal itself. In other words, even if the user's terminal is equipped with an input device that outputs absolute positions, such as a touch panel that is widely used in mobile terminals, it is virtually impossible to connect and remotely control the remote terminal using this input device, which is inconvenient.

[0005] An object of the present invention is to enable remote control of a remote terminal that accepts only relative position information to be performed using an external input device that outputs absolute positions. [Means for solving the problem]

[0006] A program according to a first aspect of the present invention connects to a remote terminal that displays an operation object at a position on a display screen based on an input amount of change in the position of a control, and causes a computer of a terminal that displays image data received from the remote terminal on its display screen to execute the following steps: a first acquisition step of acquiring first coordinate information indicating a position on the display screen of the remote terminal that is specified by a user of the remote terminal; a first generation step of generating first operation information for moving the operation object from an initial position on the display screen of the remote terminal to a position on the screen of the remote terminal that corresponds to the specified position; a first transmission step of transmitting the first operation information to the remote terminal; a second acquisition step of acquiring second coordinate information indicating a position on the display screen of the remote terminal that has been released from the user's specification; a second generation step of generating second operation information for moving the operation object to the initial position; and a second transmission step of transmitting the second operation information to the remote terminal. According to this aspect, in a device that has the function of remotely controlling a device by mirroring the display screen of the other terminal and transmitting touch position information of its own terminal to the other terminal, even if the other terminal is configured not to accept absolute coordinates, touch operations that specify an absolute position (as well as other drag operations, etc.) can be reflected in the movement of the pointer.

[0007] In a preferred embodiment, the program further causes the computer to execute a determination step of determining reference position coordinates on a display screen of the terminal itself, the reference position coordinates corresponding to the initial position, and in the first generation step, generates the first operation information based on relative coordinates of the first coordinate information with respect to the reference position coordinates. According to this embodiment, the first operation information can be generated based on the relative coordinates of the reference position coordinates and the first coordinate information.

[0008] In a preferred aspect, the program further causes the computer to execute a determination step of determining an orientation of the display screen of the counterpart terminal, and in the determination step, determines the reference position coordinates based on the orientation determined in the determination step. According to this aspect, even if the orientation of the screen of at least one of the own terminal and the counterpart terminal changes, the correspondence between the screen positions of both terminals is maintained.

[0009] In a preferred embodiment, the program further causes the computer to execute a step of setting a partitioned screen area on the display screen of the terminal that corresponds to the display screen area of the other terminal when the terminal is simultaneously connected to multiple other terminals including the other terminal, and in the first generation step, determines a reference position within the partitioned screen area that corresponds to the initial position.

[0010] In a preferred embodiment, the terminal is connected to a second remote terminal that displays an operation object at a position on the display screen based on the absolute position of an input operator, and is capable of displaying image data received from the second remote terminal on the display screen of the terminal itself, and the program further causes the computer to execute a step of displaying a predetermined object at a position on the display screen of the terminal itself indicated by the first coordinate information, and the predetermined object is displayed in a first manner if the position on the display screen of the terminal itself indicated by the first coordinate information is within a display area corresponding to the remote terminal, and is displayed in a second manner different from the first manner if the position on the display screen of the terminal itself indicated by the first coordinate information is within a display area corresponding to the second remote terminal.

[0011] In another aspect, the present invention provides a terminal having: means for connecting to a remote terminal that displays an operation object at a position on a display screen based on an amount of change in the position of an input control; means for displaying image data received from the remote terminal on the display screen of the terminal; first acquisition means for acquiring first coordinate information that indicates a position on the display screen of the terminal designated by a user; first generation means for generating, based on the first coordinate information, first operation information for moving the operation object from an initial position on the display screen of the remote terminal to a position on the screen of the remote terminal that corresponds to the designated position; first transmission means for transmitting the first operation information to the remote terminal; second acquisition means for acquiring second coordinate information that indicates a position on the display screen of the terminal where the designation by the user has been canceled; second generation means for generating second operation information for moving the operation object to the initial position; and second transmission means for transmitting the second operation information to the remote terminal.

[0012] In yet another aspect, the present invention provides a computer system including a terminal and a counterpart terminal that displays an operation object at a position on a display screen based on a position change amount of an input control, wherein the terminal includes: means for displaying image data received from the counterpart terminal on its own display screen; first acquisition means for acquiring first coordinate information indicating a position on the display screen of the terminal designated by a user; first generation means for generating, based on the first coordinate information, first operation information for moving the operation object from an initial position on the display screen of the counterpart terminal to a position on the screen of the counterpart terminal corresponding to the designated position; first transmission means for transmitting the first operation information to the counterpart terminal; second acquisition means for acquiring second coordinate information indicating a position on the display screen of the terminal where the designation by the user has been canceled; second generation means for generating second operation information for moving the operation object to the initial position; and second transmission means for transmitting the second operation information to the counterpart terminal. [Brief explanation of the drawings]

[0013] [Figure 1]1A and 1B are diagrams showing an example of the configuration of a computer system 1 according to an embodiment of the present invention. [Figure 2A] 2 is a block diagram showing an example of the functional configuration of a second terminal 20. FIG. [Figure 2B] 2 is a block diagram showing an example of the functional configuration of a first terminal 10. FIG. [Figure 3A] 2 is a diagram for explaining the correspondence between positions on the display screen of a first terminal 10 and positions on the display screen of a second terminal 20. FIG. [Figure 3B] 2 is a diagram for explaining the correspondence between positions on the display screen of a first terminal 10 and positions on the display screen of a second terminal 20. FIG. [Figure 3C] 2 is a diagram for explaining the correspondence between positions on the display screen of a first terminal 10 and positions on the display screen of a second terminal 20. FIG. [Figure 3D] 2 is a diagram for explaining the correspondence between positions on the display screen of a first terminal 10 and positions on the display screen of a second terminal 20. FIG. [Figure 4] 10A and 10B are diagrams illustrating a process of generating first operation information during non-continuous touch operations. [Figure 5] 10A and 10B are diagrams for explaining a process of generating first operation information during a successive touch operation. [Figure 6] 10A and 10B are diagrams for explaining a process of generating second operation information. [Figure 7] FIG. 10 is a diagram illustrating an example of operation in the first terminal 10. [Figure 8] 10 is a diagram for explaining the movement of a pointer MP on the second terminal 20 during a tap operation. FIG. [Figure 9] FIG. 10 is a diagram for explaining the flow of processing in the first terminal 10 and the second terminal 20 during a drag operation. [Figure 10] 10 is a diagram for explaining the movement of a pointer MP on the second terminal 20 during a drag operation. FIG. [Figure 11] FIG. 10 is a diagram illustrating the movements of pointers C1 and C2 on second terminal 20 in another embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0014] The following embodiments are subject to various technically preferable limitations, but the present invention is not limited to the following embodiments.

[0015] FIGS. 1A and 1B are diagrams showing an example of the configuration of a computer system 1 according to an embodiment of the present invention. 1A, the computer system 1 includes a first terminal 10 and a second terminal 20. The first terminal 10 and the second terminal 20 communicate with each other in accordance with a wireless communication standard such as WiFi or Bluetooth.

[0016] The first terminal 10 and the second terminal 20 are information processing terminals such as smartphones, tablet terminals, notebook PCs, and desktop PCs. The first terminal 10 has a function to remotely control the second terminal and a function to reproduce the contents of the screen displayed on the second terminal on the screen of the first terminal 10 in real time (mirroring display). Specifically, operation information generated based on an operation input at the first terminal 10 is transmitted from the first terminal 10 to the second terminal, and the second terminal 20 executes image processing and other controls that reflect the operation contents. Meanwhile, image data representing the screen displayed on the second terminal 20 is transmitted to the first terminal 10 in real time, and the first terminal 10 performs display processing based on this image data. Note that audio data may be transmitted from the second terminal 20 to the first terminal 10 in addition to the image data, and the first terminal 10 may perform sound emission processing based on this audio data (audio synchronization).

[0017] 1(B), in the computer system 1, a plurality of second terminals 20 may be simultaneously connected wirelessly or by wire to one first terminal 10. That is, the first terminal 10 may be used to remotely control a plurality of second terminals 20 (second terminal 20-1, second terminal 20-2, second terminal 20-3, second terminal 20-4), and the screen displayed on each second terminal 20 may be reproduced in real time on the screen of the first terminal 10. For the sake of convenience, unless otherwise specified, it is assumed below that the one-to-one direct connection shown in Fig. 1(A) is used. Also, it is assumed that at least the first terminal 10 has a touch panel and has a function to detect the touch position by touching the screen with a finger.

[0018] 2A shows the functions of second terminal 20. Note that only functions specific to when second terminal 20 is connected to and remotely controlled by first terminal 10 will be described here, and descriptions of various functions that are implemented when second terminal 20 is used independently will be omitted.

[0019] The second terminal 20 has a communication unit 210, a display control unit 220, a display unit 230, and an attitude sensor 250. The communication unit 210 is composed of a communication module and the like, and is connected to the first terminal 10 via a wired or wireless connection to send and receive various information.

[0020] The display control unit 220 outputs image data indicating the image displayed on the display unit 230 in real time and transmits it to the first terminal 10 via the communication unit 210. The attitude sensor 250 is composed of a three-axis acceleration sensor, a gyro sensor, etc., and detects information indicating the attitude of the second terminal 20 (for example, whether the rectangular screen is being used in portrait or landscape orientation), and transmits this information to the first terminal 10 via the communication unit 210 as necessary. Note that the generation of information indicating the attitude of the second terminal 20 and its transmission to the first terminal 10 may be omitted. This is because, when image data is displayed over the entire display screen of the second terminal, the first terminal 10 can determine the orientation of the second terminal based on the received image data. When the display control unit 220 receives operation information that reflects the user's operation content input at the first terminal 10 via the communication unit 210, it draws an operation object (called a mouse cursor, a pointer, etc.; hereinafter simply referred to as a "pointer") at a position on the screen that corresponds to the operation information.

[0021] Here, the information that the communication unit 210 can receive as information representing the content of a user's operation is limited to relative position information (i.e., information representing the amount of movement (more precisely, the direction and amount of movement) of an operator provided on the second terminal 20 for receiving a user's operation). Specifically, while it is possible to connect and use a mouse, which is an operation device that outputs the amount of movement (relative position), to the second terminal 20, it is virtually impossible to connect and use an operation device that outputs the coordinates of a point on a two-dimensional area, such as a touch pen or tablet device. For this reason, as will be described in detail below, in this embodiment, the first terminal 10 connected to the second terminal 20 generates operation information that can be handled by the second terminal 20 based on information indicating the absolute position and supplies the operation information to the second terminal 20.

[0022] As mentioned above, the second terminal 20 cannot process information indicating an absolute position supplied from other devices, but like a typical smartphone, it may be equipped with an input device such as a touch panel and have the function of processing information indicating an absolute position supplied from within.

[0023] 2B shows an example configuration of the first terminal 10. As shown in the figure, the first terminal 10 includes a communication unit 110, an input / output unit 120, a storage unit 130, a posture detection unit 160, and a processing unit 140.

[0024] The communication unit 110 includes a communication module, a connection interface, and the like, and communicates with the second terminal 20 via a communication network or directly, either wired or wirelessly. Specifically, the communication unit 110 supplies image data received from the second terminal 20 to the display control means 141, and supplies information regarding the attitude of the second terminal 20 to the determination means 143. In addition, the communication unit 110 transmits operation information generated by the processing unit 140 to the second terminal 20.

[0025] The input / output unit 120 includes a display device (e.g., a liquid crystal display) that displays an image represented by image data provided from the processing unit 140, and a transparent touch sensor that is provided to cover the display surface of the display device. In this embodiment, the display device and the touch sensor form a touchpad. The touchpad outputs operation content data that indicates the user's touch position on the touch sensor to the processing unit 140. As a result, information that indicates the content of the user's operation (designation of a position on the screen) is supplied to the processing unit 140.

[0026] In this embodiment, the touch position represented by the operation content data indicates, for example, a position in a two-dimensional coordinate system with the upper left corner of the display surface as the origin (i.e., absolute position). Note that the first terminal 10 of this embodiment includes an input / output unit 120 that functions as a touchpad. However, if the first terminal 10 includes an external device interface for connecting other external devices, such as a USB interface, an input / output device such as a touchpad may be connected to the external device interface. In short, it is sufficient that the first terminal 10 has a two-dimensional area in which the user can specify a position.

[0027] The orientation detection unit 160 includes an acceleration sensor, a gyro sensor, etc., and outputs to the processing unit 140 information indicating the orientation of the first terminal 10 (for example, whether it is being used in portrait orientation or landscape orientation).

[0028] The storage unit 130 is a memory made up of storage elements, etc. This memory stores in advance a program PR1 for causing the processing unit 140 to implement the control method of the present invention. The volatile memory is used by the processing unit 140 as a work area when executing the program PR1.

[0029] Additionally, storage unit 130 stores display area mapping information MAP that describes the correspondence between positions on the touch panel screen (in other words, a two-dimensional input field) of second terminal 20 currently connected and positions on the screen of second terminal 20. More specifically, display area mapping information MAP describes the correspondence between position coordinates on the screen of first terminal 10, including a reference position (origin of the coordinate system) on the screen of first terminal 10, and position coordinates of second terminal 20 on the screen of first terminal 10, including the reference position (origin of the coordinate system) on the screen of second terminal 20. Note that the reference position on the screen of second terminal 20 may or may not be the same as the position (initial position) to which a pointer that performs drawing processing on second terminal 20 should return each time one tap operation on first terminal 10 is completed, as will be described later.

[0030] This display area mapping information MAP is generated by display control means 141, which will be described later. In a preferred embodiment, the display area mapping information MAP is determined based on the respective orientations of first terminal 10 and second terminal 20 (whether they are being used with the screen in portrait or landscape orientation, or in any other orientation (rotation angle)), the physical size of the respective screens of first terminal 10 and second terminal 20, the respective screen resolutions of first terminal 10 and second terminal 20, and an allocation algorithm that specifies to which area of the display screen of first terminal 10 the screen of second terminal 20 is to be allocated. Note that information about the screen orientation of the first terminal 10 and / or the second terminal 20 does not need to be reflected in the display area mapping information MAP if there is effectively no choice regarding the screen orientation of the first terminal 10 and / or the second terminal 20, or if the above algorithm specifies screen allocation without depending on the posture, such as the screen orientation, of the first terminal 10 and / or the second terminal 20. In this case, the above reference position is constant for the images displayed on the first terminal 10 and the second terminal 20.

[0031] The display area mapping information MAP may be acquired by the first terminal 10 from the second terminal 20 when the second terminal 20 starts connecting with the first terminal 10, or may be stored in advance in storage unit 130 and read out by the first terminal 10 from storage unit 130 when connection of the corresponding second terminal 20 is detected. Alternatively, it is preferable to update the display area mapping information MAP based on the content of the change each time the orientation of the second terminal 20 or the resolution setting of the second terminal 20 is changed during connection. This maintains a correct correspondence between the position on the screen of the first terminal 10 and the position on the screen of the second terminal 20, and the user can specify the intended position on the second terminal 20 by specifying a position on the screen while viewing the screen of the second terminal 20 displayed on the first terminal 10.

[0032] 3A to 3D, an example of the correspondence between positions on the screen (input field) of the first terminal 10 and positions on the screen (input field) of the second terminal 20 will be described below. Hereinafter, positions on the screen of the first terminal 10 will be represented in an XY (uppercase) coordinate system, and positions on the screen of the second terminal 20 will be represented in an xy (lowercase) coordinate system. In other words, the display area mapping information MAP represents the correspondence between (X, Y) and (x, y) or the conversion rule from one to the other.

[0033] In the example of FIG. 3A, the first terminal 10 and the second terminal 20 have the same screen size and resolution, and both are used in portrait orientation. Because the first terminal 10 and the second terminal 20 are currently connected, an image identical to the image displayed on the screen of the second terminal 20 (a map image in this example) is displayed across the entire screen (display area R1) of the first terminal 10. In this example, positions on the screen of the first terminal 10 are represented by a coordinate system with the origin Po(Xo,Yo) at the left corner of the screen, and positions on the screen of the second terminal 20 are represented by a coordinate system with the origin po(xo,yo) at the left corner of the screen. P1 and p1 are corresponding positions. The correspondence between the two positions can be expressed by a function F that converts the XY coordinate system into an xy coordinate system. In this example, F(X,Y)=(x,y), i.e., X=x, Y=y. In both the XY coordinate system and the xy coordinate system, the origin of the coordinate system is not limited to the left corner of the display area. In short, it is sufficient that there is a one-to-one correspondence between any position on the screen of the second terminal 20 (which may also be referred to as any position on the image displayed on the second terminal 20) and any position within the display area of the first terminal 10 where the screen of the second terminal 20 is displayed.

[0034] 3B, the first terminal 10 is used in landscape orientation, and a portion of the display screen of the first terminal 10 is allocated as region R2, while region RF is set as an area not subject to mirroring display, and the screen displayed on the second terminal 20 is reproduced only in region 2. If the X'Y' coordinate system in region R2 is defined as having an origin Po(Xb, Yb) as shown in the figure, the components of the X'Y' coordinate system can be expressed as a linear combination of the components of the xy coordinate system (i.e., a combination of translation of the origin and rotation of the axes and change in magnification of the axes), and the correspondence between positions in the XY coordinate system and positions in the xy coordinate system in the display area mapping information MAP can be expressed as a transformation function G.

[0035] In the example of Figure 3C, the first terminal 10 is used in portrait orientation and the second terminal 20 is used in landscape orientation, with area R3 allocated as the area for mirroring display. As shown in the figure, if the X"Y" coordinate system is defined with the origin Po(Xc, Yc), the position of X" can be expressed as a linear combination of the XY coordinate components, and the correspondence with the position in the xy coordinate system can be expressed in the form of a conversion function H.

[0036] The example in Fig. 3D shows a case where four second terminals 20 are simultaneously connected to one first terminal 10. In this example, the display screen of the first terminal 10 is divided into four, and each divided screen displays the same screen as that displayed on the corresponding second terminal 20. As shown in the figure, the αβ coordinate system defined by limiting Po(Xd, Yd) corresponds to the xy coordinate system, and a position in the αβ coordinate system can be expressed as a linear combination of the coordinate components of the XY coordinate system, and the correspondence relationship between a position in the XY coordinate system and a position in the xy coordinate system can be expressed using a conversion function J.

[0037] Returning to FIG. 2B, the details of the processing unit 140 will be described. The processing unit 140 is composed of one or more processors such as CPUs. The processing unit 140 is an example of a computer according to the present invention. When the power (not shown) of the first terminal 10 is turned on, the processing unit 140 reads the program PR1 from the non-volatile memory to the volatile memory and starts executing the read program PR1. The processing unit 140 operating in accordance with the program PR1 serves as the control center of the first terminal 10. More specifically, the processing unit 140 operating in accordance with the program PR1 functions as the acquisition means 142, the display control means 141, the determination means 143, the generation means, and the transmission means 145 shown in FIG. 2.

[0038] Display control means 141 displays image data received from second terminal 20 via communication unit 110 in a display area within a display screen determined according to a predetermined allocation algorithm. This realizes mirroring of the other party's screen, and the user of first terminal 10 can perform operations on input / output unit 120 to remotely control second terminal 20 (operations on the display screen of second terminal 20) while viewing the mirrored other party's screen.

[0039] In the following description, the display control means 141 displays the entire screen of the other party on the second terminal 20 as is on the display screen of the first terminal 10, as shown in Figures 3A to 3D, regardless of whether the first terminal 10 is held in a portrait or landscape orientation. However, only a partial area of the screen of the other party may be displayed on the display screen of the first terminal 10. In this case, only a position within that partial area can be specified on the first terminal 10.

[0040] Here, when the screen size of the display unit of first terminal 10 and the screen size of the display unit of second terminal 20 differ, a predetermined algorithm can be used to determine which area of the screen area of first terminal 10 to use (in other words, which partial area to allocate as the area for mirroring display). Information about the allocated partial area is updated when the allocation is changed or when the orientation of the display screen of one or both of first terminal 10 and second terminal 20 is changed, and this information is successively reflected in display area mapping information MAP, so that the correspondence between the positions on the screen of first terminal 10 and the positions on the screen of second terminal 20 is always maintained.

[0041] The determination means 143 determines the coordinates of a reference position (hereinafter referred to as reference position coordinates) on the display screen of the first terminal 10, which corresponds to the initial position on the other party's screen of the pointer displayed on the second terminal 20, based on the display area mapping information MAP. For example, as shown in Fig. 3A, if the orientation of the screen of the other party is portrait and the orientation of the first terminal 10 is portrait, the determination means 143 determines the position Po of the upper left corner in the display area of the display unit of the first terminal 10 as the reference position corresponding to the initial position po on the screen of the other party. As shown in Fig. 3B, if the orientation of the screen of the other party is portrait and the orientation of the first terminal 10 is landscape, the determination means 143 determines the position of the upper left corner of the screen of the other party displayed in the display area of the display unit of the first terminal 10 as the reference position.

[0042] Acquisition means 142 acquires, from input / output unit 120, first coordinate information indicating a position on the display screen designated by an operation performed on input / output unit 120 by the user of first terminal 10 (hereinafter, designated position) in order to remotely operate second terminal 20. In addition, processing unit 140 acquires second coordinate information indicating a position on the display screen of the first terminal where the designation by the user has been cancelled. Specifically, processing unit 140 detects that the designation of the position has been cancelled (for example, when a finger that had been touching the screen is removed from the screen). When the designated position indicated by the first coordinate information and the second coordinate information are the same, the position corresponds to the position of a touch operation on the display screen of second terminal 20 when the user remotely operating second terminal 20 were to directly operate the second terminal 20.

[0043] The processing unit 140 can also accept a position designation made by the user of the first terminal 10 continuously moving a finger or an operating device while touching the screen (i.e., continuously changing the designated position). In such a case, the designated position indicated by the first coordinate information and the second coordinate information will differ. That is, the processing unit 140 can also accept a drag operation, a flick operation, or other position designation that changes continuously over time made by the user of the first terminal 10.

[0044] The generating means 144 includes a first generating means 1441 and a second generating means 1442, and generates information required for remote operation to be supplied to the second terminal 20 in response to an operation performed by the first terminal 10.

[0045] Specifically, when the pointer is at the initial position (in other words, when it is determined that a new or independent touch operation has been performed without specifying a position in a series of operations such as a drag operation or a flick operation), the first generation means 1441 generates first operation information for moving the pointer on the second terminal 20 to a position on the screen of the other party corresponding to the specified position, based on the first coordinate information and reference position information indicating reference position coordinates. Specifically, the first coordinate information is information indicating relative coordinates of the position indicated by the first coordinate information with respect to the reference position indicated by the reference position information, and is expressed as an amount obtained by converting a vector amount in the coordinate system of the screen of the first terminal 10, with the position indicated by the reference position information as the start point and the position indicated by the first coordinate information as the end point, into a vector amount in the coordinate system of the screen of the second terminal 20 using the display area mapping information MAP. In other words, the first coordinate information corresponds to position information regarding the touch position on the screen of the other party relative to the initial position. For example, if the configuration and usage of the first terminal 10 and the second terminal 20 are as shown in FIG. 3A, first operation information indicating the amount of movement F(X1-Xo, Y1-Yo) from the reference position is generated based on the acquired first coordinate information (X1, Y1), as shown in FIG. 4.

[0046] On the other hand, when the pointer is not at the initial position (in other words, when a position is specified in the middle of a series of operations such as a drag operation or a flick operation), the first coordinate information indicates the difference (amount of position change) between the coordinates of the position previously specified on the screen of the first terminal 10 and the coordinates of the currently specified position, converted into a difference on the screen of the second terminal 20 using the display area mapping information MAP. For example, as shown in Fig. 5, when position P2 is specified following position P1 in the state shown in Fig. 4, first operation information indicating a movement amount F(ΔX, ΔY) on the screen of second terminal 20 corresponding to the movement amount (amount of change in position) on the screen of first terminal 10 is generated based on the acquired first coordinate information (X2, Y2), where ΔX = X2 - X1 and ΔY = Y2 - Y1.

[0047] The second generating means 1442 generates second operation information for moving the pointer to the initial position. As an example, the second operation information indicates the maximum negative value of the movement amount that the second terminal 20 can receive. Generally, when the value (movement amount) input from the mouse is an amount that moves to a position off the screen, the pointer display position is controlled to be displayed at a predetermined position such as a corner of the screen. If the second terminal 20 employs such control, the pointer can be moved to the left corner of the screen by supplying (-x_max, -y_max) to the second terminal 20 as the movement amount. Here, -x_max and -y_max indicate the maximum movement amounts in the x and y directions, respectively. For example, as shown in FIG. 6, if the designation is released at position P3, second operation information indicating the movement amount (-x_max, -y_max) is generated based on the second coordinate information (X3, Y3).

[0048] As another example, the second operation information is information that indicates relative coordinates with respect to a reference position indicated by the reference position information for the position indicated by the second coordinate information, and as a specific example, the second operation information may be an amount obtained by converting a vector quantity in the coordinate system of the screen of first terminal 10, which is made up of two coordinate components with the position indicated by the second coordinate information as the start point and the position indicated by the reference position information as the end point, into the coordinate system of the screen of second terminal 20. In this case, the second operation information is relative position information, similar to the first operation information. Alternatively, the amount indicated by the second operation information does not have to be the maximum negative value or an amount that returns exactly to the initial position, as described above, but rather it may be an amount that moves the pointer from the position where the current pointer is displayed to at least a position that is off the screen.

[0049] The transmitting means 145 transmits the first operation information generated by the first generating means 1441 or the second operation information generated by the second generating means 1442 to the second terminal 20 using the communication unit 110, depending on the operation content. Specifically, when a position is specified (such as when the finger of the user of the first terminal 10 is touching the screen), first operation information is transmitted, whereas when the position specification is released (such as when the finger of the user of the first terminal 10 is removed from the screen), second operation information is transmitted.

[0050] When second terminal 20 receives the first operation information, it moves pointer MP on the display screen in accordance with the received first operation information.

[0051] For example, as shown in Fig. 5(A), assume that the position indicated by a star in Fig. 5(A) is specified as the specified position. In this case, first operation information representing a vector having the position indicated by the reference position information as the start point and the position of the star as the end point is transmitted from the first terminal 10 to the second terminal 20. Upon receiving the first operation information, the second terminal 20 moves the pointer MP on the display screen to the position corresponding to the star, as shown in Fig. 5(B), in accordance with the received first operation information. Note that the first operation information corresponds to relative position information regarding the touch position on the screen of the other party, as viewed from the initial position. Therefore, even in the second terminal 20, in which information received as information representing the user's operation is limited to relative position information, the pointer can be moved without any problems.

[0052] An example of the operation of the first terminal 10 will be described with reference to FIG. Furthermore, the processing unit 140 operating in accordance with the program PR1 performs the processing exemplified in FIG. 7 until it is instructed to end the execution of the program PR1. When the processing unit 140 detects that a new position has been specified, such as when a finger touches the screen (S202; YES), it generates and transmits first operation information for moving the pointer to a position on the screen of the other party that corresponds to the specified position on the screen of the first terminal 10 (S204). When the specification is released at that position (S206; YES), in other words, when a tap operation (click operation) is performed at that position (S206; YES), second operation information is generated and transmitted, and the pointer is returned to its initial position (S210). As a result, a screen reflecting the tap operation performed by the user is displayed on the display unit of the second terminal 20 (and the first terminal 10 that mirrors and displays it). On the other hand, if the touch position has been updated (S206; NO and S211; YES), first operation information is generated and transmitted (S212). As a result, a screen reflecting the drag operation performed by the user is displayed on the display unit of the second terminal 20 (and the first terminal 10 that mirrors it). Thereafter, if touch release is detected (S206: YES), because the touch position and the release position are different (S208: NO), second operation information is generated and transmitted, and the pointer is returned to the initial position (S210). As a result, the series of drag operations is reflected on the display unit of the second terminal 20 (and the first terminal 10) (S214).

[0053] The following describes the movement of the pointer on the second terminal 20 that has received the first operation information and the second operation information. Note that the following description is of a screen displayed on the second terminal 20, but because mirroring is performed, the display content of the same screen is reproduced on the display unit of the first terminal 10 substantially in real time.

[0054] Figure 8 shows the movement of the pointer when a tap operation is performed. As shown in FIG. 11(a), when the second terminal 20 receives the first operation information generated in S204 of FIG. 7, it performs a drawing process to move the pointer MP from its initial position (which is set to the left corner of the screen in this case) to a specified position. 8(b), when the second operation information generated in S208 is received at this position, a drawing process is first performed to change the appearance of the point, such as by making the pointer MP emit light, in order to inform the user of the second terminal 20 that a tap (click operation) has been performed. The drawing process performed when the second operation information is received may be omitted. When the second operation information generated in S208 of FIG. 7 is received, a drawing process is performed to move the pointer to the initial position.

[0055] 9 and 10, a pointer drawing process in the second terminal 20 when a drag operation is performed will be described. Fig. 9 shows the time series of the processes in the first terminal 10 and the second terminal 20. Fig. 10 shows the screen of the second terminal 20 in each process. First, at time t1, when a user touches a certain position on the screen, first operation information corresponding to that position is transmitted (t2). The second terminal 20, upon receiving this first information, begins a drawing process to move the pointer (t3). At this time, the pointer is displayed as shown in (a) and (b) of FIG. 10. As the user continues to move their finger while still touching the screen, new first operation information is generated at predetermined times (t4), and the second terminal 20 updates the display position of the pointer according to each piece of first operation information (t5 and FIG. 10(c)). When the user of the first terminal 10 removes their finger from the screen at t5 (t6), second operation information is generated (t7), and drawing processing based on the second operation information is performed (t8). Specifically, as shown in (d) of FIG. 9, the pointer is displayed at that position and then moved to its initial position. Unlike a tap operation, it is preferable not to perform any special drawing processing at the position where the finger is removed. This clearly distinguishes it from a tap operation.

[0056] As described above, according to this embodiment, the screen of second terminal 20 is mirrored, and operation information reflecting a specified position on the screen of the first terminal is transmitted to second terminal 20. As a result, even if second terminal 20 does not accept absolute coordinates, first terminal 10, which remotely controls second terminal 20, can reflect a touch operation that specifies an absolute position in the movement of a pointer on second terminal 20. In other words, according to this embodiment, it becomes possible to remotely control a partner terminal that only accepts relative position information, using an external input device that outputs absolute positions.

[0057] <Other Examples> Although one embodiment of the present invention has been described above, this embodiment can be modified as follows.

[0058] In the above embodiment, one second terminal 20 is connected to the first terminal 10, but multiple other terminals including the second terminal 20 may be connected to the first terminal 10 at the same time. When multiple other terminals including the second terminal 20 are connected to the first terminal 10 at the same time, the processing unit 140 sets multiple divided screen display areas in a display area on the display device included in the input / output unit 120, which correspond one-to-one to the display screen areas of the multiple other terminals connected to the first terminal. Then, the processing unit 140 determines, for each of the multiple divided screen areas, a reference position corresponding to the initial position of the corresponding other terminal.

[0059] In addition to the first terminal 10, a second partner terminal is connected to the first terminal 10, which displays an operation object at a position on the display screen based on the absolute position of the input operator, and the first terminal 10 may display image data received from the second partner terminal on the display screen of its own terminal. For example, as shown in FIG. 11, the first terminal 10 is connected to a second terminal 20 as well as a device 90 that accepts input of an absolute position. Then, in accordance with the display area mapping information MAP, the first terminal 10 calculates and transmits to the device 90 the position on the screen of the device 90 that corresponds to the position specified on the screen of the first terminal 10 as the first operation information and second information described above. In other words, the process of converting the absolute position into a relative position (amount of movement) is not executed. For example, when connecting with the second terminal 20 or 90, the first terminal 10 receives identification information of the second terminal 20 or 90 from the other terminal, thereby determining whether the first operation information or second operation information is to be specified as an absolute position or a relative position.

[0060] In this case, the processing unit 140 may display a predetermined object at a position on the display screen of the own terminal indicated by the first coordinate information. If the position on the display screen of the own terminal indicated by the first coordinate information is within a display area corresponding to the other terminal, the processing unit 140 may display the object in a first manner. If the position on the display screen of the own terminal indicated by the first coordinate information is within a display area corresponding to the second other terminal, the processing unit 140 may display the object in a second manner different from the first manner. A specific example of the second manner is a manner in which a circle of a predetermined size glows for a predetermined period of time at the touch position (hereinafter referred to as a normal display manner). A specific example of the first manner is a manner in which a circle of a predetermined size glows for a longer period of time at the touch position. According to this manner, it is possible to determine whether the operating terminal is a terminal that only accepts relative positions (mouse input) or a terminal that can accept absolute positions based on the display manner of the object.

[0061] In particular, when the pointer returns to its initial position after the above-described operation is completed, the user of the first terminal 10 or the second terminal 20 may experience a sense of afterimage. In particular, when the second terminal 20 and the third terminal 90 are simultaneously connected, the user's sense of feedback of the operation performed by the user in the drawing content differs between remotely operating the second terminal 20 and remotely operating the third terminal 90. This may confuse the user of the first terminal 10, for example, as to whether they actually operated the pointer to return to its initial position or simply performed a tap operation. From this perspective, it is expected that the user's sense of discomfort will be alleviated by changing the appearance and presentation (e.g., shape, size, color, movement, or changes over time) of the pointer displayed on the screen corresponding to the second terminal 20 and the pointer displayed on the screen corresponding to the third terminal 90, among the screens displayed on the first terminal 10. In the example shown in FIG. 11, the shape of the pointer C1 displayed in region R5 is round, while the shape of the pointer C2 displayed in region R5 is star-shaped.

[0062] In this way, a pointer drawn independently by the first terminal 10 may be superimposed on the screen displayed on a terminal connected to the first terminal 10. Alternatively, the first terminal 10 may not display any pointer based on a user operation, but may display a screen including a pointer image displayed on a terminal connected to the first terminal 10. In this case, the pointer drawn independently by the first terminal 10 and the pointer image included in the image data received from the other terminal will not be displayed overlapping each other.

[0063] In short, the present invention is a terminal that connects to a remote terminal that displays an operation object at a position on a display screen based on a position change amount of an input operator, and displays image data received from the remote terminal on its own display screen, and that executes the following steps: a first acquisition step of acquiring first coordinate information indicating a position on the display screen of the remote terminal that is specified by a user of the remote terminal; a first generation step of generating first operation information for moving the operation object from an initial position on the display screen of the remote terminal to a position on the screen of the remote terminal that corresponds to the specified position; a first transmission step of transmitting the first operation information to the remote terminal; a second acquisition step of acquiring second coordinate information indicating a position on the display screen of the remote terminal that has been released from the user's specification; a second generation step of generating second operation information for moving the operation object to the initial position; and a second transmission step of transmitting the second operation information to the remote terminal. [Explanation of symbols]

[0064] 1... Computer system, 10... First terminal, 20... Second terminal, 110... Communication unit, 120... Input / output unit, 140... Processing unit, 141... Display control means, 142... Acquisition means, 143... Determination means, 144... Generation means, 1441... First generation means, 1442... Second generation means, 145... Transmission means, 160... Attitude detection unit, 130... Storage unit, 150... Bus, MAP... Display area mapping information

Claims

1. a computer of a terminal that is connected to a remote terminal that displays an operation object at a position on a display screen based on the input amount of change in the position of the operation button, and displays image data received from the remote terminal on a display screen of the terminal; a first acquisition step of acquiring first coordinate information indicating a position on a display screen of the terminal, the position being designated by a user of the terminal; a first generation step of generating first operation information for moving the operation object from an initial position on the display screen of the partner terminal to a position on the screen of the partner terminal corresponding to the specified position, based on the first coordinate information acquired in the first acquisition step; a first transmission step of transmitting the first operation information to the other terminal; a second acquisition step of acquiring second coordinate information indicating a position on the display screen of the terminal where the designation by the user has been cancelled; a second generation step of generating second operation information for moving the operation object from a position displayed on the screen of the counterpart terminal to the initial position based on the first operation information in response to acquiring the second coordinate information; a second transmission step of transmitting the second operation information to the other terminal; A program to execute.

2. The computer, a determination step of determining a reference position coordinate on a display screen of the terminal itself, the reference position corresponding to the initial position; In the first generating step, the first operation information is generated based on the relative coordinates of the first coordinate information with respect to the reference position coordinates.

2. The program according to claim 1 .

3. The computer, When the terminal is simultaneously connected to a plurality of other terminals including the other terminal, setting a divided screen area on the display screen of the terminal itself corresponding to the display screen area of the other terminal; determining a reference position within the divided screen area corresponding to the initial position in the first generating step; The program according to claim 1 or 2.

4. the terminal is capable of connecting to a second remote terminal that displays an operation object at a position on a display screen based on the absolute position of an input operator, and displaying image data received from the second remote terminal on the display screen of the terminal itself; The program further causing the computer to execute a step of displaying a predetermined object at a position on a display screen of the terminal indicated by the first coordinate information; the predetermined object is displayed in a first manner when the position on the display screen of the own terminal indicated by the first coordinate information is within a display area corresponding to the other terminal, and is displayed in a second manner different from the first manner when the position on the display screen of the own terminal indicated by the first coordinate information is within a display area corresponding to the second other terminal; The program according to claim 1.

5. a means for connecting to a remote terminal that displays an operation object at a position on a display screen based on the input amount of change in the position of the operator; means for displaying the image data received from the other terminal on a display screen of the own terminal; a first acquiring means for acquiring first coordinate information indicating a position on a display screen of the terminal designated by a user; a first generating means for generating, based on the first coordinate information acquired by the first acquiring means, first operation information for moving the operation object from an initial position on the display screen of the counterpart terminal to a position on the screen of the counterpart terminal corresponding to the specified position; a first transmitting means for transmitting the first operation information to the other terminal; a second acquiring means for acquiring second coordinate information indicating a position on the display screen of the terminal where the designation by the user has been cancelled; a second generation means for generating second operation information for moving the operation object from a position displayed on the screen of the counterpart terminal to the initial position based on the first operation information in response to acquiring the second coordinate information; second transmission means for transmitting the second operation information to the other terminal; A terminal having:

6. a remote terminal that displays an operation object at a position on a display screen based on the input position change amount of the operator, The terminal means for displaying the image data received from the other terminal on a display screen of the own terminal; a first acquiring means for acquiring first coordinate information indicating a position on a display screen of the terminal designated by a user; a first generating means for generating, based on the first coordinate information acquired by the first acquiring means, first operation information for moving the operation object from an initial position on the display screen of the counterpart terminal to a position on the screen of the counterpart terminal corresponding to the specified position; a first transmitting means for transmitting the first operation information to the other terminal; a second acquiring means for acquiring second coordinate information indicating a position on the display screen of the terminal where the designation by the user has been cancelled; a second generation means for generating second operation information for moving the operation object from a position displayed on the screen of the counterpart terminal to the initial position based on the first operation information in response to acquiring the second coordinate information; second transmission means for transmitting the second operation information to the other terminal; having Computer system.

Images