Remote control method for terminals

Abstract

This enables remote control of a target terminal that only accepts relative location information, using an external input device that outputs absolute location. [Solution] The first terminal 10 communicates with the second terminal 20, which displays a mouse pointer at a position on its display screen based on the amount of position change of the input operator, and displays the display screen of the second terminal 20 on its own terminal. The first terminal 10 sends first operation information to the second terminal 20 to move the pointer from the initial position of the pointer on the other terminal's display screen to a position on the other terminal's screen corresponding to a position on the first terminal's display screen specified by the user. The first terminal 10 also sends second operation information to the second terminal 20 to move the pointer from a position on the first terminal's display screen that has been unspecified by the user back to its initial position.

Description

Technical Field

[0001] The present invention relates to remote control of a terminal.

Background Art

[0002] There exists a device that has a mirroring function for displaying, on the display screen of the own terminal, the screen displayed on a terminal to be remotely operated (hereinafter referred to as the target terminal), and a function for remotely operating the target terminal by transmitting, to the target terminal, position information indicating a specified position on the display screen of the own terminal when the user touches the touch panel or the like (see Patent Document 1, etc.).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] [[ID=3 4]] Depending on the type of the target terminal, due to specifications such as the OS (Operating System) of the target terminal, the information that can be received as information representing the user's operation content may be limited to relative position information (in other words, the amount of movement of the operator). That is, when remotely operating this type of target terminal, one has no choice but to connect a mouse to the terminal as well. That is, even if the own terminal has an input device that outputs an absolute position, such as a touch panel, which is widely used in mobile terminals and the like, it is virtually impossible to remotely operate the target terminal using this input device, which is inconvenient.

[0005] An object of the present invention is to enable remote operation of a target terminal that accepts only relative position information using an external input device that outputs an absolute position.

Means for Solving the Problems

[0006] A program according to a first aspect of the present invention connects to a partner terminal that displays an operation object at a position on a display screen based on the amount of position change of an input operator, and causes the computer of a terminal that displays image data received from the partner terminal on the display screen of the terminal to execute: a first acquisition step of acquiring first coordinate information indicating a position on the display screen of the terminal specified by the user of the terminal; a first generation step of generating first operation information for moving the operation object from an initial position on the display screen of the partner terminal to a position on the screen of the partner terminal corresponding to the specified position; a first transmission step of transmitting the first operation information to the partner terminal; a second acquisition step of acquiring second coordinate information indicating a position on the display screen of the terminal after the user's specification has been released; a second generation step of generating second operation information for moving the operation object to the initial position; and a second transmission step of transmitting the second operation information to the partner terminal. According to this embodiment, in a device that has the function of performing remote control by mirroring the display screen of the other terminal and transmitting touch position information of the own terminal to the other terminal, even if the other terminal does not accept absolute coordinates, touch operations that specify an absolute position (in addition to other drag operations, etc.) can be reflected in the movement of the pointer.

[0007] In a preferred embodiment, the program causes the computer to further perform a determination step to determine a reference position coordinate on the display screen of the terminal corresponding to the initial position, and in the first generation step, generates the first operation information based on the relative coordinates of the first coordinate information with respect to the reference position coordinate. According to this embodiment, the first operation information can be generated based on the relative coordinates of the reference position coordinate and the first coordinate information.

[0008] In a preferred embodiment, the program causes the computer to further perform a determination step to determine the orientation of the display screen of the other terminal, and in the determination step, the reference position coordinates are determined based on the orientation determined in the determination step. According to this embodiment, even if the orientation of at least one of the screens of the local terminal and the other terminal changes, the correspondence between the screen positions of both is maintained.

[0009] In a preferred embodiment, the program further causes the computer to perform the step of setting a partitioned screen area on the display screen of the local terminal that corresponds to the display screen area of ​​the other terminal when the local terminal is connected simultaneously with a plurality of other terminals, including the other terminal, and in the first generation step, a reference position within the partitioned screen area that corresponds to the initial position is determined.

[0010] In a preferred embodiment, the terminal is connected to a second partner terminal that displays an operation object at a position on a display screen based on the absolute position of an input operator, and is capable of displaying image data received from the second partner terminal on the display screen of the terminal itself. The program further causes the computer to display a predetermined object at the position on the display screen of the terminal itself indicated by the first coordinate information, and the predetermined object is displayed in a first manner if the position on the display screen of the terminal itself indicated by the first coordinate information is within a display area corresponding to the partner terminal, and is displayed in a second manner different from the first manner if the position on the display screen of the terminal itself indicated by the first coordinate information is within a display area corresponding to the second partner terminal.

[0011] In another aspect, the present invention provides a terminal comprising: means for connecting to a partner terminal that displays an operation object at a position on a display screen based on the amount of position change of an input operator; means for displaying image data received from the partner terminal on the display screen of the own terminal; first acquisition means for acquiring first coordinate information indicating a position on the display screen of the own terminal as specified by the user; first generation means for generating first operation information for moving the operation object from an initial position on the display screen of the partner terminal to a position on the screen of the partner terminal corresponding to the specified position, based on the first coordinate information; first transmission means for transmitting the first operation information to the partner terminal; second acquisition means for acquiring second coordinate information indicating a position on the display screen of the own terminal as specified by the user; second generation means for generating second operation information for moving the operation object to the initial position; and second transmission means for transmitting the second operation information to the partner terminal.

[0012] In yet another aspect, the present invention provides a computer system comprising: a terminal and a partner terminal that displays an operation object at a position on its display screen based on the amount of position change of an input operator. The terminal includes means for displaying image data received from the partner terminal on its own display screen; first acquisition means for acquiring first coordinate information indicating a position on the display screen of the terminal specified by the user; first generation means for generating first operation information for moving the operation object from an initial position on the display screen of the partner terminal to a position on the screen of the partner terminal corresponding to the specified position, based on the first coordinate information; first transmission means for transmitting the first operation information to the partner terminal; second acquisition means for acquiring second coordinate information indicating a position on the display screen of the terminal after the user's specification has been released; second generation means for generating second operation information for moving the operation object to the initial position; and second transmission means for transmitting the second operation information to the partner terminal. [Brief explanation of the drawing]

[0013] [Figure 1](A) and (B) are diagrams showing examples of the configuration of a computer system 1 according to an embodiment of the present invention. [Figure 2A] This is a block diagram showing an example of the functional configuration of the second terminal 20. [Figure 2B] This is a block diagram showing an example of the functional configuration of the first terminal 10. [Figure 3A] This diagram illustrates the correspondence between the position on the display screen of the first terminal 10 and the position on the display screen of the second terminal 20. [Figure 3B] This diagram illustrates the correspondence between the position on the display screen of the first terminal 10 and the position on the display screen of the second terminal 20. [Figure 3C] This diagram illustrates the correspondence between the position on the display screen of the first terminal 10 and the position on the display screen of the second terminal 20. [Figure 3D] This diagram illustrates the correspondence between the position on the display screen of the first terminal 10 and the position on the display screen of the second terminal 20. [Figure 4] This diagram illustrates the process of generating the first operation information during non-continuous touch operations. [Figure 5] This diagram illustrates the process of generating the first operation information during continuous touch operations. [Figure 6] This is a diagram illustrating the process of generating second operation information. [Figure 7] This figure shows an example of operation at the first terminal 10. [Figure 8] This diagram illustrates the movement of the pointer MP on the second terminal 20 during a tap operation. [Figure 9] This diagram illustrates the processing flow at the first terminal 10 and the second terminal 20 during a drag operation. [Figure 10] This diagram illustrates the movement of the pointer MP on the second terminal 20 during a drag operation. [Figure 11] This figure illustrates the movement of pointers C1 and C2 in the second terminal 20 in another embodiment of the present invention. [Modes for carrying out the invention]

[0014] The embodiments described below are subject to various technically preferable limitations. However, the embodiments of the present invention are not limited to the forms described below.

[0015] FIGS. 1(A) and 1(B) are diagrams showing a configuration example of a computer system 1 according to an embodiment of the present invention. The computer system 1 includes, for example, as shown in FIG. 1(A), a first terminal 10 and a second terminal 20. The first terminal 10 and the second terminal 20 communicate according to wireless communication standards such as WiFi and Bluetooth.

[0016] The first terminal 10 and the second terminal 20 are information processing terminals such as smartphones, tablet terminals, notebook PCs, and desktop PCs. The first terminal 10 has a function of remotely controlling the second terminal and a function of reproducing (mirroring display) the content of the screen displayed on the second terminal in real time on the screen of the first terminal 10. Specifically, operation information generated based on an operation input at the first terminal 10 is transmitted from the first terminal 10 to the second terminal, and image processing and other controls reflecting the operation content are executed at the second terminal 20. On the other hand, image data representing the screen displayed on the second terminal 20 is transmitted to the first terminal 10 in real time, and display processing is performed at the first terminal 10 based on this image data. In addition to the image data, audio data may be transmitted from the second terminal 20 to the first terminal 10, and sound playback processing (audio synchronization) based on this audio data may be performed at the first terminal 10.

[0017] Also, as shown in FIG. 1(B), in the computer system 1, a plurality of second terminals 20 may be simultaneously connected to one first terminal 10 wirelessly or by wire. That is, remote control may be performed on a plurality of second terminals 20 (second terminal 20-1, second terminal 20-2, second terminal 20-3, second terminal 20-4) using the first terminal 10, and the screens displayed on each second terminal 20 may be reproduced in real time on the screen of the first terminal 10. For the sake of explanation, unless otherwise specified, the one-to-one direct connection configuration shown in Figure (A) will be assumed below. Furthermore, it will be assumed that at least the first terminal 10 is equipped with a touch panel and has a function to detect the touch position by touching the screen with a finger.

[0018] Figure 2A shows the functions of the second terminal 20. Here, only the functions specific to when the second terminal 20 is connected and receives remote control from the first terminal 10 are described, and the various functions that are exhibited when the second terminal 20 is used independently are omitted from the explanation.

[0019] The second terminal 20 includes a communication unit 210, a display control unit 220, a display unit 230, and a posture sensor 250. The communication unit 210 consists of a communication module and the like, and connects to the first terminal 10 by wire or wireless connection to send and receive various information.

[0020] The display control unit 220 outputs image data representing the image displayed on the display unit 230 in real time and transmits it to the first terminal 10 via the communication unit 210. The attitude sensor 250 consists of a 3-axis accelerometer and a gyroscope, and detects information indicating the attitude of the second terminal 20 (for example, whether the rectangular screen is being used in portrait or landscape orientation), and transmits it to the first terminal 10 via the communication unit 210 as needed. Note that the generation of information indicating the attitude of the second terminal 20 and its transmission to the first terminal 10 may be omitted. This is because, if image data is displayed across the entire display screen of the second terminal, the first terminal 10 can determine the orientation of the second terminal based on the received image data. When the display control unit 220 receives operation information that reflects the user's operations entered at the first terminal 10 via the communication unit 210, it draws an operation object (also called a mouse cursor, pointer, etc.; hereinafter simply referred to as "pointer") at the position on the screen corresponding to this operation information.

[0021] Here, the information that the communication unit 210 can receive as information representing the user's operation is limited to relative position information (i.e., information representing the amount of movement (more precisely, the direction and amount of movement) of the control element provided on the second terminal 20 for receiving user operations). Specifically, while it is possible to connect and use a mouse, which is an operation device that outputs the amount of movement (relative position), to the second terminal 20, it is practically impossible to connect and use an operation device that outputs the coordinates of a single point in a two-dimensional area, such as a stylus or tablet device. Therefore, as will be detailed below, in this embodiment, the first terminal 10 connected to the second terminal 20 generates operation information that the second terminal 20 can handle based on information indicating the absolute position, and supplies it to the second terminal 20.

[0022] As mentioned above, the second terminal 20 cannot process absolute position information supplied from other devices, but it may have an input device such as a touch panel, like a typical smartphone, and have a function to process absolute position information supplied internally.

[0023] Figure 2B shows an example configuration of the first terminal 10. As shown in the figure, the first terminal 10 comprises a communication unit 110, an input / output unit 120, a storage unit 130, an attitude detection unit 160, and a processing unit 140.

[0024] The communication unit 110 includes a communication module and connection interface, and communicates with the second terminal 20 via a wired or wireless connection, either through a communication network or directly. Specifically, the communication unit 110 supplies image data received from the second terminal 20 to the display control means 141, and supplies information regarding the attitude of the second terminal 20 to the determination means 143. It also transmits operation information generated by the processing unit 140 to the second terminal 20.

[0025] The input / output unit 120 includes a display device (e.g., a liquid crystal display) that displays an image represented by image data provided by the processing unit 140, and a transparent touch sensor provided to cover the display surface of the display device. In this embodiment, the display device and the touch sensor form a touch pad. The touch pad outputs operation data indicating the user's touch position on the touch sensor to the processing unit 140. This supplies the processing unit 140 with information indicating the content of the user's operation (specification of a position on the screen).

[0026] In this embodiment, the touch position represented by the operation data indicates a position (i.e., absolute position) in a two-dimensional coordinate system with the upper left corner of the display surface as the origin. While the first terminal 10 in this embodiment includes an input / output unit 120 that functions as a touchpad, if the first terminal 10 includes an external device interface for connecting other external devices such as a USB interface, an input / output device such as a touchpad may be connected to the external device interface. In short, it is sufficient that the first terminal 10 provides a two-dimensional area where the user can specify a position.

[0027] The attitude detection unit 160 includes an acceleration sensor, a gyroscope, etc., and outputs information indicating the attitude of the first terminal 10 (for example, whether it is being used vertically or horizontally) to the processing unit 140.

[0028] The memory unit 130 is a memory consisting of memory elements and the like. This memory pre-stores program PR1, which enables the processing unit 140 to implement the control method of the present invention. The volatile memory is used by the processing unit 140 as a work area when executing program PR1.

[0029] In addition, the memory unit 130 stores a display area mapping information MAP that describes the correspondence between the position on the touch panel screen (in other words, the two-dimensional input field) of the currently connected second terminal 20 and the position on the screen of the second terminal 20. More specifically, the display area mapping information MAP describes the correspondence between the position coordinates on the screen of the first terminal 10, including the reference position (origin of the coordinate system) on the screen of the first terminal 10, and the position coordinates on the screen of the second terminal 20, including the reference position (origin of the coordinate system) on the screen of the second terminal 20. The reference position on the screen of the second terminal 20 may or may not be the same as the position (initial position) to which the pointer to be drawn on the second terminal 20 should return after each tap operation on the first terminal 10 is completed, as will be described later.

[0030] This display area mapping information MAP is generated by the display control means 141, which will be described later. In a preferred embodiment, the display area mapping information MAP is determined based on the orientation of the first terminal 10 and the second terminal 20 (whether the screen is used in portrait or landscape orientation, or in any other orientation (rotation angle), etc.), the physical size of the screens of the first terminal 10 and the second terminal 20, the resolution of the screens of the first terminal 10 and the second terminal 20, and an allocation algorithm that defines which area of ​​the display screen of the first terminal 10 the screen of the second terminal 20 is allocated to. Furthermore, if there is virtually no choice regarding the screen orientation in the first terminal 10 and the second terminal 20, or if the above algorithm defines screen allocation without depending on the orientation or other orientation of the screens of the first terminal 10 and / or the second terminal 20, then information regarding the screen orientation of the first terminal 10 and / or the second terminal 20 does not need to be reflected in the display area mapping information MAP. In this case, the above reference position will be constant for the images displayed in the first terminal 10 and the second terminal 20.

[0031] The display area mapping information MAP may be obtained by the first terminal 10 when the second terminal 20 starts connecting with the first terminal 10, or it may be stored in the storage unit 130 in advance, and the first terminal 10 may read the display area mapping information MAP from the storage unit 130 when it detects the connection of the corresponding second terminal 20. Alternatively, it is preferable to update the display area mapping information MAP based on the changes whenever the orientation of the second terminal 20 or the resolution settings on the second terminal 20 are changed during the connection. This ensures that the correspondence between the position on the screen of the first terminal 10 and the position on the screen of the second terminal 20 is correctly maintained, and the user can specify the intended position on the second terminal 20 by viewing the screen of the second terminal 20 displayed on the first terminal 10 and specifying the position on that screen.

[0032] The following explains an example of the correspondence between the position on the screen (input field) of the first terminal 10 and the position on the screen (input field) of the second terminal 20, using Figures 3A to 3D. Hereafter, the position on the screen of the first terminal 10 will be represented in the XY (uppercase) coordinate system, and the position on the screen of the second terminal 20 will be represented in the xy (lowercase) coordinate system. In other words, the display area mapping information MAP represents the correspondence between (X,Y) and (x,y), or the conversion rule from one to the other.

[0033] In the example shown in Figure 3A, the first terminal 10 and the second terminal 20 have the same screen size and resolution, and both are used in portrait orientation. Since the first terminal 10 and the second terminal 20 are connected, the same image displayed on the screen of the second terminal 20 (a map image in this example) is displayed on the entire screen (display area R1) of the first terminal 10. In this example, the position on the screen of the first terminal 10 is represented in a coordinate system with the left corner of the screen as the origin Po(Xo,Yo), and the position on the screen of the second terminal 20 is represented in a coordinate system with the left corner of the screen as the origin po(xo,yo). P1 and p1 are corresponding positions. The correspondence between the two positions can be represented by a function F that converts the XY coordinate system to the xy coordinate system. In this example, the correspondence is F(X,Y)=(x,y), that is, X=x and Y=y. Note that in both the XY coordinate system and the xy coordinate system, the origin of the coordinate system is not limited to the left corner of the display area. In short, it is sufficient that there is a one-to-one correspondence between any position on the screen of the second terminal 20 (or any position on the image displayed on the second terminal 20) and any position within the display area of ​​the first terminal 10 on which the screen of the second terminal 20 is displayed.

[0034] In the example shown in Figure 3B, the first terminal 10 is used horizontally and is allocated as a portion of its display screen, region R2, while region RF is set as an area not subject to mirroring display, and only region 2 reproduces the screen displayed on the second terminal 20. If the X'Y' coordinate system in region R2 is defined as the origin Po(Xb,Yb) as shown in the figure, the components of the X'Y' coordinate system can be represented by a linear combination of the components of the xy coordinate system (i.e., a combination of translation of the origin and rotation or scaling of the axes), and the correspondence between the position in the XY coordinate system and the position in the xy coordinate system in the display area mapping information MAP can be represented by a transformation function G.

[0035] In the example shown in Figure 3C, the first terminal 10 is used in portrait orientation and the second terminal 20 is used in landscape orientation, with region R3 allocated as the region for mirroring display. As shown in the figure, if the X''Y'' coordinate system is defined with the origin Po(Xc,Yc), the position of X'' can be expressed as a linear combination of the coordinate components of the XY system, and the correspondence with the position in the xy coordinate system can be expressed in the form of a transformation function H.

[0036] In the example shown in Figure 3D, four second terminals 20 are simultaneously connected to one first terminal 10. In this example, the display screen of the first terminal 10 is divided into four sections, and each section displays the same screen as the corresponding second terminal 20. As shown in the figure, the αβ coordinate system, defined by limiting Po(Xd,Yd), corresponds to the xy coordinate system, and the position in the αβ coordinate system can be expressed as a linear combination of the coordinate components of the XY coordinate system. The correspondence between the position in the XY coordinate system and the position in the xy coordinate system can be expressed using the transformation function J.

[0037] Returning to Figure 2B, we will now explain the details of the processing unit 140. The processing unit 140 is composed of one or more processors such as CPUs. The processing unit 140 is an example of a computer in the present invention. When the power (not shown) of the first terminal 10 is turned on, the processing unit 140 reads program PR1 from non-volatile memory to volatile memory and starts executing the read program PR1. The processing unit 140, operating according to program PR1, plays the role of the control center of the first terminal 10. More specifically, the processing unit 140, operating according to program PR1, functions as the acquisition means 142, display control means 141, determination means 143, generation means and transmission means 145 shown in Figure 2.

[0038] The display control means 141 displays image data received from the second terminal 20 via the communication unit 110 in a display area within the display screen determined according to a predetermined allocation algorithm. This enables mirroring of the other party's screen, allowing the user of the first terminal 10 to perform operations (operations on the display screen of the second terminal 20) via the input / output unit 120 while referring to the mirrored other party's screen.

[0039] In the following, the display control means 141 displays the entire screen of the other party on the second terminal 20 on the display screen of the first terminal 10, as shown in Figures 3A to 3D, regardless of whether the first terminal 10 is held in a vertical or horizontal orientation. However, only a portion of the other party's screen may be displayed on the display screen of the first terminal 10. In that case, the first terminal 10 can only specify a position within that portion of the screen.

[0040] Here, if the screen size of the display unit of the first terminal 10 and the screen size of the display unit of the second terminal 20 are different, the part of the screen area of ​​the first terminal 10 to use (in other words, which part of the area to allocate as the area for mirroring display) can be determined using a predetermined algorithm. When the allocation of the allocated part of the area is changed, or when the orientation of the display screens of one or both of the first terminal 10 and the second terminal 20 changes, this information is sequentially reflected in the display area mapping information MAP, and as a result, the correspondence between the position on the screen of the first terminal 10 and the position on the screen of the second terminal 20 is always maintained.

[0041] The determination means 143 determines the coordinates of the reference position on the display screen of the first terminal 10 (hereinafter referred to as the reference position coordinates) that correspond to the initial position on the other party's screen of the pointer displayed on the second terminal 20, based on the display area mapping information MAP. For example, as shown in Figure 3A, if the orientation of the other party's screen is vertical and the orientation of the first terminal 10 is vertical, the determination means 143 determines the position Po of the upper left corner in the display area of ​​the display unit of the first terminal 10 as the reference position corresponding to the initial position po on the other party's screen. As shown in Figure 3B, if the orientation of the other party's screen is vertical and the orientation of the first terminal 10 is horizontal, the determination means 143 determines the position of the upper left corner of the other party's screen displayed in the display area of ​​the display unit of the first terminal 10 as the reference position.

[0042] The acquisition means 142 acquires from the input / output unit 120 first coordinate information indicating a position on the display screen (hereinafter referred to as the specified position) that was specified by an operation performed on the input / output unit 120 by the user of the first terminal 10, in order to remotely control the second terminal 20. In addition, the processing unit 140 acquires second coordinate information indicating a position on the display screen of its own terminal after the user's specification has been released. Specifically, it detects that the position specification has been released (for example, when a finger that was touching the screen leaves the screen). If the specified position indicated by the first coordinate information and the second coordinate information are the same, it corresponds to the position of the touch operation on the display screen of the second terminal 20 if the user remotely controlling the second terminal 20 were to directly operate the second terminal 20.

[0043] Furthermore, the processing unit 140 can also accept position specifications in which the user of the first terminal 10 continuously moves their finger or control device while touching the screen (i.e., continuously changes the specified position). In such cases, the specified position indicated by the first coordinate information will be different from the second coordinate information. That is, the processing unit 140 can also accept drag operations, flick operations, and other position specifications that change continuously over time performed by the user of the first terminal 10.

[0044] The generation means 144 includes a first generation means 1441 and a second generation means 1442, and generates information necessary for remote operation to be supplied to the second terminal 20 in response to an operation performed by the first terminal 10.

[0045] Specifically, the first generation means 1441 generates first operation information for moving the pointer on the second terminal 20 to a position on the other party's screen corresponding to a specified position, based on first coordinate information and reference position information indicating a reference position coordinate, when the pointer is at its initial position (in other words, when it is determined that a new or independent touch operation has been performed, without specifying a position in a series of operations such as dragging or flicking). Specifically, the first coordinate information represents the relative coordinates of the position indicated by the first coordinate information with respect to the reference position indicated by the reference position information. For example, it is expressed as a quantity obtained by converting a vector quantity in the coordinate system of the first terminal 10's screen, with the position indicated by the reference position information as the starting point and the position indicated by the first coordinate information as the ending point, into a vector quantity in the coordinate system of the second terminal 20's screen using display area mapping information MAP. That is, the first coordinate information corresponds to the relative position information of the touch position on the other party's screen, as seen from the initial position. For example, if the configuration and usage of the first terminal 10 and the second terminal 20 are as shown in Figure 3A, then, as shown in Figure 4, first operation information indicating the amount of movement from the reference position, F(X1-Xo,Y1-Yo), is generated based on the acquired first coordinate information (X1,Y1).

[0046] On the other hand, if the pointer is not in its initial position (in other words, if the position is specified in the middle of a series of operations such as dragging or flicking), the first coordinate information indicates the difference (amount of position change) between the coordinates of the previously specified position on the screen of the first terminal 10 and the coordinates of the currently specified position, converted into a difference on the screen of the second terminal 20 using the display area mapping information MAP. For example, as shown in Figure 5, if position P1 is specified followed by position P2 from the state shown in Figure 4, first operation information is generated based on the acquired first coordinate information (X2, Y2) that indicates the amount of movement F(ΔX, ΔY) on the screen of the second terminal 20, which corresponds to the amount of movement (amount of change in position) on the screen of the first terminal 10. Here, ΔX = X2 - X1 and ΔY = Y2 - Y1.

[0047] The second generation means 1442 generates second operation information for moving the pointer to the initial position. As an example, the second operation information indicates the maximum negative value of the movement amount that the second terminal 20 can receive. Generally, if the value (movement amount) input from the mouse is an amount that moves the pointer to a position outside the screen, control is performed to display the pointer at a predetermined position such as the corner of the screen. If the second terminal 20 employs such control, supplying (-x_max, -y_max) as the movement amount to the second terminal 20 will move the pointer to the left corner of the screen. Here, -x_max and -y_max indicate the maximum values ​​of the movement amount in the x and y directions, respectively. For example, as shown in Figure 6, if the specification is released at position P3, second operation information indicating the movement amount (-x_max, -y_max) is generated based on the second coordinate information (X3, Y3).

[0048] As another example, the second operation information represents information that shows the relative coordinates with respect to the reference position indicated by the reference position information for the position indicated by the second coordinate information. Specifically, it may be a quantity obtained by converting a vector quantity in the coordinate system of the screen of the first terminal 10, which consists of two coordinate components, with the position indicated by the second coordinate information as the starting point and the position indicated by the reference position information as the ending point, to the coordinate system of the screen of the second terminal 20. In this case, the second operation information will be relative position information, similar to the first operation information. Alternatively, the amount indicated by the second operation information does not have to be the maximum negative value or the amount needed to return exactly to the initial position, as described above; it does not need to be the amount needed to move the pointer from its current displayed position to a position at least off-screen.

[0049] The transmission means 145 transmits either the first operation information generated by the first generation means 1441 or the second operation information generated by the second generation means 1442 to the second terminal 20 using the communication unit 110, depending on the operation content. Specifically, when a location is specified (for example, when the user's finger on the first terminal 10 is touching the screen), the first operation information is sent, while when the location specification is removed (for example, when the user's finger on the first terminal 10 leaves the screen), the second operation information is sent.

[0050] When the second terminal 20 receives the first operation information, it moves the pointer MP on the display screen according to the received first operation information.

[0051] For example, suppose the designated location is the one indicated by the star in Figure 5(A), as shown in Figure 5(A). In this case, first operation information representing a vector starting from the position indicated by the reference position information and ending at the position of the star is transmitted from the first terminal 10 to the second terminal 20. Upon receiving this first operation information, the second terminal 20 moves the pointer MP on the display screen to the position corresponding to the star, as shown in Figure 5(B), according to the received first operation information. Since the first operation information corresponds to relative position information from the initial position regarding the touch position on the other party's screen, the pointer can be moved without any problems even if the second terminal 20 is limited to receiving relative position information as information representing the user's operation.

[0052] Figure 7 illustrates an example of the operation of the first terminal 10. Furthermore, the processing unit 140, which operates according to program PR1, performs the processing illustrated in Figure 7 until it is instructed to terminate the execution of program PR1. When the processing unit 140 detects that a new position has been specified, such as when a finger touches the screen (S202; YES), it generates first operation information and sends the first operation information to move the pointer to the position on the other device's screen relative to the specified position on the first device's screen (S204). If the specification is released at that position (S206; YES), in other words, if a tap operation (click operation) is performed at that position (S206; YES), second operation information is generated and sent, and the pointer returns to its initial position (S210). As a result, the screen reflecting the tap operation performed by the user is displayed on the display unit of the second device 20 (and the first device 10 which is mirroring it). On the other hand, if the touch position is updated (S206: NO and S211: YES), the first operation information is generated and transmitted (S212). As a result, the screen reflecting the drag operation performed by the user is displayed on the display unit of the second terminal 20 (and the first terminal 10 which is mirroring it). Subsequently, if a touch release is detected (S206: YES), the touch position and the release position are different (S208: NO), so the second operation information is generated and transmitted, and the pointer is returned to its initial position (S210). As a result, the series of drag operations are reflected on the display unit of the second terminal 20 (and the first terminal 10) (S214).

[0053] The following describes the movement of the pointer in the second terminal 20, which has received the first and second operation information. Although the following description refers to the screen displayed on the second terminal 20, due to mirroring, the same screen content is reproduced on the display unit of the first terminal 10 in virtually real time.

[0054] Figure 8 shows the movement of the pointer when a tap operation is performed. As shown in Figure (a), upon receiving the first operation information generated in S204 of Figure 7, the second terminal 20 performs drawing processing to move the pointer MP from its initial position (in this case, set to the left corner of the screen) to the specified position. Next, as shown in Figure 8(b), when the second operation information generated in S208 is received at this position, drawing processing is performed to change the appearance of the point, such as making the pointer MP light up, in order to inform the user of the second terminal 20 that it is a tap (click operation). The drawing processing when the second operation information is received may be omitted. When the second operation information generated in S208 in Figure 7 is received, drawing processing is performed to move the pointer back to its initial position.

[0055] Using Figures 9 and 10, the pointer drawing process on the second terminal 20 when a drag operation is performed will be explained. Figure 9 shows the processing in the first terminal 10 and the second terminal 20 in chronological order. Figure 10 shows the screen of the second terminal 20 during each process. First, at time t1, when the user touches a certain position on the screen, first operation information corresponding to that position is transmitted (t2). Upon receiving this first information, the second terminal 20 starts drawing processing to move the pointer (t3). At this time, the pointer is displayed as shown in Figures 10(a) and (b). In this state, as the user moves their finger while keeping it on the screen, new first operation information is generated at predetermined timings (t4), and the second terminal 20 updates the display position of the pointer according to each piece of first operation information (t5 and Figure 10(c)). When the user of the first terminal 10 lifts their finger from the screen at t5 (t6), second operation information is generated (t7), and drawing processing based on the second operation information is performed (t8). Specifically, as shown in Figure 9(d), the pointer is displayed at that position, and then moved back to its initial position. It is preferable that, unlike a tap operation, no special drawing processing is performed at the position where the finger is lifted. This makes the difference from a tap operation clear.

[0056] As described above, according to this embodiment, the screen of the second terminal 20 is mirrored, and operation information reflecting the specified position on the screen of the first terminal is transmitted to the second terminal 20. As a result, even if the second terminal 20 does not accept absolute coordinates, the first terminal 10 that remotely controls the second terminal 20 can reflect touch operations that specify absolute positions in the movement of the pointer on the second terminal 20. In other words, according to this embodiment, it becomes possible to remotely control a target terminal that only accepts relative position information using an external input device that outputs absolute positions.

[0057] <Other examples> Although one embodiment of the present invention has been described above, this embodiment can be modified as follows.

[0058] In the above embodiment, there was only one second terminal 20 connected to the first terminal 10, but multiple other terminals, including the second terminal 20, may be connected to the first terminal 10 simultaneously. When multiple other terminals, including the second terminal 20, are connected to the first terminal 10 simultaneously, the processing unit 140 sets up multiple partitioned screen display areas in the display area of ​​the display device included in the input / output unit 120 that correspond one-to-one with the display screen areas of each of the multiple other terminals connected to its own terminal. Then, for each of the multiple partitioned screen areas, the processing unit 140 determines a reference position that corresponds to the initial position on the corresponding other terminal.

[0059] In addition to the first terminal 10, a second partner terminal is connected to the first terminal 10, which displays an operation object on the display screen at a position based on the absolute position of the input operator. The first terminal 10 may also display image data received from the second partner terminal on its own display screen. For example, as shown in Figure 11, the first terminal 10 connects to the second terminal 20 as well as a device 90 that accepts absolute position input. Then, according to the display area mapping information MAP, the first terminal 10 calculates and transmits to 90 the position on the screen of 90 corresponding to the position specified on the screen of the first terminal 10 as the first operation information and second information described above. In other words, it does not perform the process of converting absolute position to relative position (movement amount). For example, when connecting to the second terminal 20 or 90, the identification information of the second terminal 20 or 90 is received from the other terminal, and it is determined whether the first operation information and second operation information are defined by absolute position or relative position.

[0060] In this case, the processing unit 140 may display a predetermined object at the position on the display screen of its own terminal indicated by the first coordinate information. If the position on the display screen of its own terminal indicated by the first coordinate information is within the display area corresponding to the other terminal, the object is displayed in the first mode. If the position on the display screen of its own terminal indicated by the first coordinate information is within the display area corresponding to the second other terminal, the object is displayed in the second mode, which is different from the first mode. A specific example of the second mode is a mode in which a circle of a predetermined size lights up for a predetermined time at the touch position (hereinafter referred to as the normal display mode). A specific example of the first mode is a mode in which a circle of a predetermined size lights up for a longer time than the predetermined time at the touch position. According to this mode, it is possible to determine whether the operating terminal is a terminal that only accepts relative positions (mouse input) or a terminal that can accept absolute positions through the display mode of the object.

[0061] In particular, the process by which the pointer returns to its initial position after the completion of the above-mentioned operation may give users of the first terminal 10 and the second terminal 20 a feeling of afterimage. Especially when the second terminal 20 and the third terminal 90 are connected simultaneously, the feeling of feedback from the operations performed by the user to the drawing content differs depending on whether the second terminal 20 or the 90 is remotely controlled. Therefore, the user of the first terminal 10 may become confused as to whether they actually manipulated the pointer to return to its initial position or simply performed a tap operation. From this perspective, it is expected that the user's discomfort can be reduced by changing the appearance and presentation (e.g., shape, size, color, movement, or how these change over time) of the pointer depending on whether it is displayed on the screen corresponding to the display screen of the second terminal 20 or on the display screen of the 90. In the example shown in Figure 11, the shape of pointer C1 displayed in area R5 is round, while the shape of pointer C2 displayed in area R5 is star-shaped.

[0062] Thus, the pointer drawn independently by the first terminal 10 may be superimposed on the screen displayed on the terminal connected to the first terminal 10. Alternatively, the first terminal 10 may not display any pointers based on user operations and may instead display the screen, including the pointer image, that is displayed on the terminal connected to the first terminal 10. In this case, the pointer drawn independently by the first terminal 10 and the pointer image included in the image data received from the other terminal will not overlap in their display.

[0063] In short, the present invention relates to a terminal that connects to a remote terminal that displays an operation object at a position on its display screen based on the amount of position change of an input operator, and displays image data received from the remote terminal on the display screen of the local terminal, and the terminal is configured to perform the following steps: a first acquisition step of acquiring first coordinate information indicating a position on the local terminal's display screen, which has been specified by the user of the local terminal; a first generation step of generating first operation information for moving the operation object from an initial position on the remote terminal's display screen to a position on the remote terminal's screen corresponding to the specified position; a first transmission step of transmitting the first operation information to the remote terminal; a second acquisition step of acquiring second coordinate information indicating a position on the local terminal's display screen, which has been released from the user's specification; a second generation step of generating second operation information for moving the operation object to the initial position; and a second transmission step of transmitting the second operation information to the remote terminal. [Explanation of Symbols]

[0064] 1...Computer system, 10...First terminal, 20...Second terminal, 110...Communication unit, 120...Input / output unit, 140...Processing unit, 141...Display control means, 142...Acquisition means, 143...Decision means, 144...Generation means, 1441...First generation means, 1442...Second generation means, 145...Transmission means, 160...Attitude detection unit, 130...Storage unit, 150...Bus, MAP...Display area mapping information

Claims

1. The terminal connects to a remote terminal that displays an operational object on its display screen based on the amount of position change of the input operator, and the computer of the terminal that displays the image data received from the remote terminal on the terminal's display screen, A first acquisition step involves obtaining first coordinate information that indicates the position on the display screen of the device, as specified by the user of the device. A first generation step of generating first operation information for moving the operation object from its initial position on the display screen of the other terminal to a position on the screen of the other terminal corresponding to the specified position, A first transmission step of transmitting the first operation information to the other terminal, A second acquisition step involves obtaining second coordinate information indicating the position on the display screen of the user's terminal after the user's designation has been canceled, A second generation step of generating second operation information for moving the operation object to the initial position, A second transmission step of transmitting the second operation information to the other terminal, A program to execute.

2. To the aforementioned computer, Further, a determination step is performed to determine the reference position coordinates on the display screen of the terminal corresponding to the initial position. In the first generation step, the first operation information is generated based on the relative coordinates of the first coordinate information with respect to the reference position coordinates. The program according to feature 1.

3. To the aforementioned computer, When the aforementioned local terminal connects to multiple other terminals, including the aforementioned other terminal, The steps include setting a partitioned screen area on the display screen of the own terminal that corresponds to the display screen area of ​​the aforementioned other terminal, In the first generation step, a reference position within the divided screen area corresponding to the initial position is determined. The program according to claim 1 or 2.

4. The aforementioned terminal is connected to a second partner terminal that displays an operation object on its display screen at a position based on the absolute position of the input operator, and is capable of displaying image data received from the second partner terminal on its own terminal's display screen. The aforementioned program, The computer is further instructed to display a predetermined object at the position on the display screen of its terminal indicated by the first coordinate information. The predetermined object is displayed in a first manner if the position on the display screen of the local terminal indicated by the first coordinate information is within the display area corresponding to the other terminal, and is displayed in a second manner different from the first manner if the position on the display screen of the local terminal indicated by the first coordinate information is within the display area corresponding to the second other terminal. The program according to claim 1.

5. A means for connecting to a partner terminal that displays an operational object on a display screen based on the amount of position change of the input operator, A means for displaying image data received from the other party's terminal on the display screen of the own terminal, A first acquisition means for acquiring first coordinate information that indicates a position on the display screen of the user's terminal, A first generation means generates first operation information based on the first coordinate information for moving the operation object from its initial position on the display screen of the other terminal to a position on the screen of the other terminal corresponding to the specified position, A first transmission means for transmitting the first operation information to the other terminal, A second acquisition means for acquiring second coordinate information indicating the position on the display screen of the user's terminal after the user's designation has been canceled, A second generation means for generating second operation information for moving the operation object to the initial position, A second transmission means for transmitting the second operation information to the other terminal, A terminal.

6. The system comprises a terminal and a partner terminal that displays an operation object on its display screen based on the amount of position change of the input operator. The aforementioned terminal is A means for displaying image data received from the other party's terminal on the display screen of the own terminal, A first acquisition means for acquiring first coordinate information that indicates a position on the display screen of the user's terminal, A first generation means generates first operation information based on the first coordinate information for moving the operation object from its initial position on the display screen of the other terminal to a position on the screen of the other terminal corresponding to the specified position, A first transmission means for transmitting the first operation information to the other terminal, A second acquisition means for acquiring second coordinate information indicating the position on the display screen of the user's terminal after the user's designation has been canceled, A second generation means for generating second operation information for moving the operation object to the initial position, A second transmission means for transmitting the second operation information to the other terminal, Having, Computer system.

Images