Computer and program

Abstract

To prevent excessive image deformation during gesture operations due to processing delays.SOLUTION: A computer according to the present invention is configured to communicate with a remote computer equipped with an input sensor and a display, and generates an image to be displayed on the remote computer's display. The computer sequentially receives, from the remote computer, a series of report data, each of which includes the position of an indicator on a sensor surface of the input sensor detected by the input sensor, detects gestures based on the series of report data received from the remote computer, generates a deformed image by applying the content and amount of deformation indicated by the detected gesture to the image or a previously transmitted deformed image, and transmits the deformed image to the remote computer. If the report data received from the remote computer is an end report indicating that the indicator has left the sensor surface, the computer performs processing for cancelling the deformation.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an image drawing method, and more particularly to an image drawing method for transforming an image in accordance with a gesture input by a user. [Background technology]

[0002] Remote desktop systems that display images on a remote computer are becoming popular. Patent Document 1 discloses an example of a remote desktop system. This type of remote desktop system includes a host computer that generates images to be displayed on the display of the remote computer, and a remote computer that displays images provided by the host computer. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2016-189127 Summary of the Invention [Problem to be solved by the invention]

[0004] Incidentally, the remote computer may have an input sensor for input using a finger or a pen. When the remote computer having the input sensor detects that a position in the image has been designated with a finger or a pen, it transmits various data including the designated position in the form of a report to the host computer. The host computer detects a gesture (e.g., pinch out, pinch in, drag, rotation, etc.) based on the series of reports thus received, and applies a deformation (e.g., enlargement, reduction, movement, rotation, etc.) to the image according to the detected gesture. When the deformed image is supplied from the host computer to the remote computer, the deformed image can also be viewed on the remote computer.

[0005] However, with the above process, after a user performs a gesture on a remote computer, communication between the remote computer and the host computer is required before the deformation caused by the gesture is reflected in the image displayed on the remote computer, resulting in a considerable processing delay. Due to this processing delay, the user may mistakenly believe that they have not yet performed enough gestures, even though they have actually performed a sufficient number of gestures, and may perform additional gestures. This results in excessive deformation of the image, as if inertia were at work, and therefore improvement was needed.

[0006] Therefore, one object of the present invention is to provide an image drawing method that can prevent excessive deformation of an image due to a gesture caused by a processing delay. [Means for solving the problem]

[0007] An image processing method according to a first aspect of the present invention is an image drawing method executed in a system including a host computer that runs an operating system and generates an image to be displayed on a display of a remote computer, and a remote computer that is equipped with an input sensor having a sensor surface and the display, and displays the image supplied from the host computer on the display, wherein the remote computer sequentially detects the position of an indicator on the sensor surface at a predetermined sampling rate using the input sensor, and each time the position of the indicator is detected, sequentially transmits report data including the detected position to the host computer, the host computer detects a gesture based on the series of report data received from the remote computer, generates a deformed image by applying a deformation of content and amount indicated by the detected gesture to the image or a previously transmitted deformed image, and transmits the deformed image to the remote computer, and the host computer executes a process to cancel the deformation in response to the report data received from the remote computer being an end report indicating that the indicator has left the sensor surface.

[0008] An image processing method according to a second aspect of the present invention is an image drawing method executed in a system including a host computer that runs an operating system and generates an image to be displayed on a display of a remote computer, and a remote computer that is equipped with an input sensor having a sensor surface and the display and displays the image supplied from the host computer on the display, wherein the remote computer sequentially detects the position of an indicator on the sensor surface at a predetermined sampling rate using the input sensor, determines whether or not the movement of the indicator indicated by the detected position corresponds to a predefined gesture, and if it is determined that the movement of the indicator does not correspond to a predefined gesture, generates report data for the input sensor and transmits it to the host computer, and if it is determined that the movement of the indicator corresponds to a predefined gesture, generates report data for a device of a different type from the input sensor and transmits it to the host computer. [Effects of the Invention]

[0009] According to the first aspect of the present invention, image deformation is cancelled in response to an end report, so that it is possible to prevent image deformation due to a gesture from becoming excessive due to processing delays.

[0010] According to the second aspect of the present invention, a gesture is converted into data with a small data size and transmitted, so that it is possible to prevent excessive deformation of an image due to a gesture caused by a processing delay. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a diagram showing a system configuration of computers 1 and 2 that execute an image processing method according to a first embodiment of the present invention. [Figure 2] 1A is a diagram showing the format of report data R generated by the operating system 30 of the computer 1 regarding the pen P, and FIG. 1B is a diagram showing the format of report data R generated by the processor 10 regarding the user's finger. [Figure 3]2 is a schematic block diagram showing the functional blocks of an operating system 30 of a computer 2. FIG. [Figure 4] FIG. 10 is a diagram showing a processing flow of image transformation processing executed by a drawing application 31. [Figure 5] FIG. 10 is a diagram showing a processing flow of a cancellation image number setting process performed by the drawing application 31 in order to set the above-mentioned cancellation image number n in the memory 11. [Figure 6] FIG. 10 is a diagram illustrating the relationship between time T_delay and the number n of cancellation images. [Figure 7] FIG. 10 is a diagram showing a processing flow of another example of the cancellation image number setting processing performed by the drawing application 31 to set the cancellation image number n in the memory 11. [Figure 8] FIG. 8 is a diagram showing a window image 40, which is an example of a test interface transmitted in step S30 of FIG. 7. [Figure 9] FIG. 2 is a schematic block diagram showing functional blocks of an operating system 30 of a computer 2 according to a modified example of the first embodiment of the present invention. [Figure 10] FIG. 10 is a diagram showing a process flow of report data transmission processing executed by the operating system 30 of the computer 1 according to the second embodiment of the present invention. [Figure 11] 10(a) and 10(b) are diagrams showing examples of correspondence tables that associate types of gestures with commands registered in the drawing application 31. FIG. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0013] Fig. 1 is a diagram showing the system configuration of computers 1 and 2 that execute an image processing method according to a first embodiment of the present invention. Each of computers 1 and 2 is an information processing device for personal use, such as a personal computer, a tablet terminal, or a smartphone. Although computers 1 and 2 may have the same configuration, in this embodiment, computer 1 is used as a remote computer and computer 2 is used as a host computer, and Fig. 1 shows only the configuration necessary for each to function as a remote computer or a host computer.

[0014] The computer 1, which is a remote computer, is configured to have a processor 10, a memory 11, a communication device 12, an input device 13, a display 14, and an input sensor 15. On the other hand, the computer 2, which is a host computer, is configured to have the processor 10, the memory 11, the communication device 12, and the display 14.

[0015] The processor 10 is the central processing unit of the computers 1 and 2, and is configured to read and execute from the memory 11 programs constituting the illustrated operating system 30, as well as various device drivers and various applications. The applications executed by the processor 10 of the computer 2 include a drawing application 31 having a function for generating images. The images generated by the drawing application 31 include images that are subject to deformation operations by the user. The operating system 30 also includes a desktop window manager 30a, which is a program for managing screen drawing. The desktop window manager 30a generates a video signal based on the image generated by the drawing application 31 and supplies it to the display 14 of the computer itself and other computers, and also supplies video signals supplied from other computers to the display 14 of the computer itself.

[0016] Memory 11 is a storage device that stores programs executed by processor 10 as well as various data referenced by processor 10, and specifically includes a main storage device such as a DRAM (Dynamic Random Access Memory) and an auxiliary storage device such as a hard disk.

[0017] The communication device 12 is a device that has the function of communicating with other computers via the Internet or an ad hoc network. The operating system 30 transmits and receives various data to and from the operating systems 30 of other computers via the communication device 12.

[0018] The input device 13 is a device for accepting user input to the operating system 30, and is configured to include, for example, a keyboard and a mouse. When a user operates the input device 13, the input device 13 generates data indicating the operation content and supplies it to the operating system 30. The operating system 30 accepts the user input based on the data thus supplied.

[0019] The display 14 is a device that visually outputs a video signal supplied from the operating system 30 (more specifically, the desktop window manager 30a). By outputting the video signal in this way, the display 14 allows the user to view an image generated by the drawing application 31.

[0020] The input sensor 15 is a device having a sensor surface, and has the function of repeatedly and sequentially detecting the position of a pointer such as the illustrated pen P or a user's finger on the sensor surface at a predetermined sampling rate, and receiving various data from the pen P. The data received from the pen P includes writing pressure detected by a pressure sensor built into the pen P. There are no particular limitations on the specific method of detecting the pen P and finger by the input sensor 15, but for example, an active electrostatic method or an electromagnetic induction method can be suitably used to detect the pen P, and a capacitance method can be suitably used to detect the finger. The following description will be continued assuming that the pen P is detected by the active electrostatic method and the finger is detected by the capacitance method.

[0021] The sensor surface of the input sensor 15 typically doubles as the display surface of the display 14. In this case, the user can use a pointer to operate an image displayed on the display 14 as if they were operating it by directly touching it. However, the sensor surface of the input sensor 15 may be provided separately from the display surface of the display 14, as in the case of a touchpad on a laptop computer or an external digitizer, for example.

[0022] Directly below the sensor surface are arranged a plurality of x-electrodes, each extending in the y-direction and arranged at equal intervals in the x-direction, and a plurality of y-electrodes, each extending in the x-direction and arranged at equal intervals in the y-direction. Input sensor 15 uses these electrodes to detect the position of the pointer and receive data from pen P. Below, the processing of input sensor 15 will be described in detail, separately for when the detection target is pen P and when it is a finger.

[0023] First, in the case where the detection target is a pen P, the input sensor 15 is configured to periodically transmit a predetermined uplink signal using either one of the multiple x-electrodes or the multiple y-electrodes as a transmitting electrode. This uplink signal includes a local ID to be assigned to an undetected pen P.

[0024] When the pen P, which has not yet been paired with the input sensor 15, receives an uplink signal through electrostatic coupling with the x-electrode or y-electrode, it extracts and acquires the local ID from the signal and transmits a downlink signal at a timing corresponding to the timing of receiving the uplink signal. At this stage, the downlink signal transmitted by the pen P consists solely of a position signal, which is an unmodulated carrier signal. The input sensor 15 receives this position signal at each of the multiple x-electrodes and multiple y-electrodes within the input sensor 15, detects the position of the pen P based on the reception strength at each electrode (global scan), and performs pairing with the pen P using the next or subsequent uplink signal.

[0025] The pen P that has established pairing with the input sensor 15 performs a process of transmitting a downlink signal including the position signal and a data signal, which is a carrier wave signal modulated by transmission data such as writing pressure, in response to an uplink signal periodically received from the input sensor 15. The input sensor 15 receives the transmitted position signal at a predetermined number of x-electrodes and y-electrodes near the previously detected position, thereby updating the position of the pen P (local scan), and also acquires data transmitted by the pen P by receiving the data signal. The updated position and acquired data are then supplied to the processor 10 along with the local ID assigned to the pen P. This enables the operating system 30 to sequentially acquire data such as the position and writing pressure for each pen P.

[0026] Next, we will explain the case where the detection target is a user's finger. The input sensor 15 is configured to periodically execute a process of supplying a predetermined finger detection signal to either one of the multiple x-electrodes or the multiple y-electrodes and receiving the signal via the other of the multiple x-electrodes or the multiple y-electrodes. When a finger approaches an intersection of the x-electrodes and the y-electrodes, the reception strength of the finger detection signal received by the input sensor 15 decreases due to the capacitance generated between the finger and the x-electrodes and the y-electrodes. The input sensor 15 detects this change in reception strength and periodically detects the position of the user's finger.

[0027] Here, the input sensor 15 is configured to perform a tracking process to track the position of a detected finger. Specifically, when the input sensor 15 detects a new finger position, it assigns a finger ID to the position and supplies it to the processor 10 together with the detected position. If the input sensor 15 detects a finger at a next timing within a predetermined range from the previously detected position, it assigns the same finger ID to the detected position and supplies it to the processor 10 together with the detected position. This allows the operating system 30 to individually acquire the trajectory of each finger using the finger ID, even when the positions of multiple fingers are detected.

[0028] When the operating system 30 of the computer 1 receives the position and other data (including the data received from the pen P, the local ID, and the finger ID) from its own input sensor 15 as described above, it generates report data including the position and data received and transmits it to the computer 2 via the communication device 12. As a result, the details of the operation performed by the user on the sensor surface of the input sensor 15 of the computer 1 are supplied to the operating system 30 of the computer 2.

[0029] 2(a) is a diagram showing the format of report data R generated by the operating system 30 of the computer 1 regarding the pen P. As shown in the figure, the report data R generated regarding the pen P includes position data PD indicating the detected position, writing pressure data PRE indicating the writing pressure received from the pen P, and the above-mentioned local ID (LID). The writing pressure data PRE has a value greater than 0 when the pen P is in contact with the sensor surface, and is 0 when the pen P is not in contact with the sensor surface.

[0030] The start report SR generated for the pen P is report data R that includes pen down information PenDown indicating that the pen P has come into contact with the sensor surface, in addition to position data PD, writing pressure data PRE, and a local ID. The processor 10 is configured to generate the start report SR in response to a change in the writing pressure data PRE from 0 to a value greater than 0.

[0031] The end report ER generated for the pen P is report data R that includes pen-up information PenUp indicating that the pen P has been released from the sensor surface, in addition to the position data PD, the writing pressure data PRE, and the local ID. The processor 10 is configured to generate the end report ER in response to a change in the writing pressure data PRE from a value greater than 0 to 0.

[0032] 2(b) is a diagram showing the format of report data R generated by processor 10 for a user's finger. As shown in the figure, report data R generated for pen P includes position data PD indicating the detected position and the finger ID (FID) described above.

[0033] The start report SR generated for a finger includes tracking start information Finger Track Start indicating that tracking of the finger has started, in addition to the position data PD and the finger ID. The processor 10 is configured to generate the start report SR in response to the start of tracking processing of a new finger by the input sensor 15.

[0034] The end report ER generated for a finger includes, in addition to the finger ID, lost information Finger Track Loss indicating that the movement of the finger that had been tracked as a series of positions by the tracking process is no longer detected. The position data PD is not included in the end report ER generated for a finger. The processor 10 is configured to generate the end report ER in response to the input sensor 15 completing the tracking process of the finger.

[0035] Returning to FIG. 1 , operations performed by a user on the sensor surface of input sensor 15 may include various gestures, such as pinch out, pinch in, drag, rotation, and swipe. Note that pinch out is an operation of enlarging a displayed image by spreading the distance between two fingers touching the sensor surface, pinch in is an operation of shrinking the displayed image by narrowing the distance between two fingers touching the sensor surface, drag is an operation of moving the displayed image by sliding an indicator on the sensor surface, rotation is an operation of rotating the displayed image by sliding an indicator on the sensor surface, and swipe is an operation of scrolling the screen by sliding an indicator on the sensor surface. These gestures are detected by the operating system 30 or drawing application 31 of computer 2 based on a series of report data R received from computer 1.

[0036] The drawing application 31 has a function of generating a deformed image by applying a deformation to the generated image in accordance with the content and amount indicated by the gesture detected as described above. The deformations applied to the image in this manner include enlargement, reduction, movement, rotation, scrolling, and the like. The deformed image generated by the drawing application 31 is supplied as a video signal to the computer 1 via the desktop window manager 30a of the computer 2. The desktop window manager 30a of the computer 1 updates the display 14 of the computer 1 using the supplied video signal. This allows the user of the computer 1 to visually recognize the deformed image generated as a result of the user's operation. The desktop window manager 30a of the computer 2 also supplies the video signal of the deformed image to the display 14 of the computer 2. This allows the user of the computer 2 to visually recognize the deformed image in the same way as the user of the computer 1.

[0037] 3 is a schematic block diagram showing the functional blocks of the operating system 30 of the computer 2. As shown in the figure, the operating system 30 functionally comprises the above-mentioned desktop window manager 30a, as well as a receive buffer 30b and a transmit buffer 30c, each of which is a FIFO (First In First Out) queue. Of these, report data R (including a start report SR and an end report ER) received from the computer 1 is written into the receive buffer 30b via the communication device 12.

[0038] The drawing application 31 first generates an image PIC to be displayed on the display 14 of each of the calculators 1 and 2, and writes the image PIC to the end of the transmission buffer 30c. After that, when report data R indicating the results of operations by the user of the calculator 1 is written to the reception buffer 30b, the drawing application 31 reads the report data R in order from oldest to newest, and detects a gesture based on the series of report data R including the past report data R read up to that point. The drawing application 31 then transforms the latest image PIC based on the detected gesture, and writes an image PIC indicating the result to the end of the transmission buffer 30c.

[0039] The desktop window manager 30a sequentially reads images PIC from the transmission buffer 30c, generates a video signal based on the read image PIC each time, and supplies the video signal to the communication device 12 and the display 14. The communication device 12 transmits the supplied video signal to the computer 1. Although not shown, the desktop window manager 30a of the computer 1 receives the transmitted video signal and updates the display of the display 14 of the computer 1 based on the received video signal. This allows the user of the computer 1 to visually confirm how the results of operations are reflected in the image PIC. Furthermore, the display 14 of the computer 2, which receives the video signal from the desktop window manager 30a, updates the image being displayed based on the supplied video signal. This allows the user of the computer 2 to visually confirm how the results of operations performed by other users on the computer 1 are reflected in the image PIC.

[0040] According to the above process, after a user makes a gesture on input sensor 15 of calculator 1, communication between calculator 1 and calculator 2 is required before the deformation caused by the gesture is reflected in the image displayed on display 14 of calculator 1, which causes a considerable processing delay. Due to this processing delay, the user may mistakenly believe that they have not made enough gestures, even though they have actually made a sufficient number of gestures, and may make an excessive gesture. This results in excessive deformation of the image, as if inertia were at work.

[0041] Therefore, the drawing application 31 according to this embodiment executes a process to cancel the image deformation in response to the fact that the report data R received from the computer 1 is an end report ER. Specifically, as shown in FIG. 3, the drawing application 31 instructs the desktop window manager 30a to stop transmitting at least some of the one or more images PIC stored in the transmission buffer 30c. Upon receiving this instruction, the desktop window manager 30a stops the transmission of the image PIC whose transmission has been stopped by discarding it from the transmission buffer 30c. This cancels the deformation based on the last few gestures in the series, thereby preventing the image from being excessively deformed.

[0042] 4 is a diagram showing the processing flow of the image transformation processing executed by the drawing application 31. Hereinafter, the processing of canceling the image transformation will be described in more detail with reference to FIG.

[0043] The drawing application 31 first generates an image to be displayed on the display 14 of the computers 1 and 2 (step S1), and writes the generated image to the transmission buffer 30c (step S2). Thereafter, the drawing application 31 attempts to acquire report data R from the reception buffer 30b (step S3), and determines whether or not the report data R was acquired (step S4). If the drawing application 31 determines in step S4 that the report data R was not acquired, it repeats step S3. If the drawing application 31 determines that the report data R was acquired, it temporarily stores the acquired report data R in the memory 11 (step S5), and then determines whether or not the acquired report data R is an end report ER (step S6).

[0044] If it is determined in step S6 that the report is not an end report ER, the drawing application 31 attempts to detect a gesture based on the series of report data R acquired up to that point (step S7) and determines whether the gesture was detected (step S8). If the gesture was not detected, the process returns to step S3 and repeats the process. If the gesture was detected, the process applies a deformation to the image in accordance with the content and amount indicated by the detected gesture (step S9). The image to be deformed here is the image that the drawing application 31 previously wrote to the transmission buffer 30c. Having generated the deformed image in step S9, the drawing application 31 writes the generated deformed image to the transmission buffer 30c (step S10) and returns to step S3.

[0045] The drawing application 31, which has determined in step S6 that the report is an end report ER, refers to the number of cancellation images n that is set in advance in the memory 11, and determines whether n=0 (step S11). The number of cancellation images n will be described in detail later with reference to FIGS. 5 to 8.

[0046] If the drawing application 31 determines in step S11 that n=0, it returns the process to step S3. In this case, the process of canceling the deformation of the image is not performed. On the other hand, if the drawing application 31 determines in step S11 that n=0 does not, it controls the desktop window manager 30a to stop sending the nth and subsequent images (step S12). As a result, images for n report data R going back from the end report ER are deleted from the transmission buffer 30c, and are no longer displayed on the displays 14 of the computers 1 and 2.

[0047] Next, the drawing application 31 determines whether the control of the desktop window manager 30a was completed in time (step S13). In other words, if the value of n is large, several images after the nth image may have already been transmitted at the time step S11 is executed. In such a case, the drawing application 31 determines that the control of the desktop window manager 30a was not completed in time. If the drawing application 31 determines that the control was completed in time in step S13, it returns the process to step S3. On the other hand, if the drawing application 31 determines that the control was not completed in time in step S13, it regenerates the (n+1)th image (the rewound image that would have been displayed on the display 14 if the gesture had not been performed) based on the series of report data R stored in the memory 11 and writes the regenerated image to the transmission buffer 30c (step S14). As a result, although the image displayed on the display 14 is temporarily over-deformed, it is possible to immediately restore the image to an image that has not been over-deformed.

[0048] 5 is a diagram showing the processing flow of the cancellation image number setting processing performed by the drawing application 31 in order to set the above-mentioned cancellation image number n in the memory 11. As shown in the figure, the drawing application 31 that has started this processing first acquires environmental information (step S20).

[0049] The environmental information is information that indicates the environment of Calculator 1, and may include, for example, either the size of Calculator 1's display 14 or the time T_delay that is required from when Calculator 1 detects the position of a pointer until an image corresponding to that position is displayed on Calculator 1's display 14. The drawing application 31 may obtain the size of Calculator 1's display 14 by receiving information indicating the size of the display 14 from Calculator 1. The drawing application 31 may also obtain the time T_delay by having Calculator 1's operating system 30 measure the time T_delay and receiving the result from Calculator 1. The time T_delay will be described in detail later with reference to FIG. 6.

[0050] Next, drawing application 31 determines the number of cancellation images n based on the acquired environmental information (step S21). In one example, if the size of Calculator 1's display 14 is sufficiently small, n = 0 (i.e., the process of canceling image deformation is not executed), and n is increased as the size of display 14 increases. Since excessive image deformation becomes more noticeable the larger the size of the display 14 on which it is displayed, determining the number of cancellation images n in this manner makes it possible to keep excessive image deformation within an inconspicuous range.

[0051] FIG. 6 is a diagram illustrating the relationship between the time T_delay and the number of cancellation images n. As shown in the figure, a delay of the time T_delay occurs between the time t1 when the user inputs a gesture on the computer 1 and the time t2 when the corresponding transformed image is displayed on the display 14 of the computer 1. Therefore, the drawing application 31 determines the number of cancellation images n so that gestures corresponding to the report data R corresponding to the time T_delay are canceled, going back from the end report ER, as shown in FIG. 6. By determining the number of cancellation images n in this manner, it is possible to perform an optimal amount of cancellation according to the time T_delay.

[0052] Returning to Fig. 5, after determining the number of cancellation images n, the drawing application 31 sets the determined number of cancellation images n in the memory 11 (step S22). Thereafter, the drawing application 31 will refer to the number of cancellation images n set in step S22 in steps S11 and S12 of Fig. 4.

[0053] 7 is a diagram showing the processing flow of another example of the cancellation image number setting processing performed by the drawing application 31 to set the cancellation image number n in the memory 11. As shown in the figure, the drawing application 31 that has started this processing first transmits a test interface to the computer 1 (step S30).

[0054] FIG. 8 shows a window image 40, which is an example of the test interface transmitted in step S30. As shown in FIG. 8, the window image 40 includes a drawing area 41, a test image 42 arranged in the drawing area 41, a slider 43, and a confirm button 44. The test image 42 is an image that can be deformed by a user's gesture. For example, if the user pinches out on the test image 42, the test image 42 is enlarged. The slider 43 is a component for adjusting the number of cancellation images n by a user operation, and is configured to allow selection of any integer between 0 and a predetermined value (10 in the example of FIG. 8). Selecting 0 on the slider 43 means that the process of canceling the image transformation is not performed. Selecting a value of 1 or greater on the slider 43 means that the degree of the process of canceling the image transformation is set. The confirm button 44 is a button for confirming the number of cancellation images n to the value set on the slider 43.

[0055] Returning to FIG. 7, the drawing application 31 provisionally determines the number n of negation images based on the current position of the slider 43 (step S31). Then, using the provisionally determined number n of negation images, the drawing application 31 executes an image transformation process based on a user operation (step S32). The process of step S32 is specifically the image transformation process shown in FIG. 4, and is executed in the background of the process shown in FIG. 7. While executing steps S31 and S32, the drawing application 31 monitors the state of the Enter button 44 (step S33). Then, in response to pressing (clicking or tapping) the Enter button 44, the drawing application 31 executes a process to determine the number n of negation images based on the current position of the slider 43 (step S34), and sets the determined number n of negation images in the memory 11 (step S35). Through the above process, the user of the calculator 1 can select an optimal value for the number n of negation images while checking the actual transformation status.

[0056] As described above, according to the image processing method of this embodiment, image deformation is canceled in accordance with the end report ER, so that it is possible to prevent excessive image deformation due to gestures caused by processing delays.

[0057] Furthermore, the number of cancellation images n is set based on factors such as the size of Calculator 1's display 14 and the time T_delay required from when Calculator 1 detects the position of the pointer until the image corresponding to that position is displayed on Calculator 1's display 14, making it possible to optimize the number of cancellation images n depending on the environment.

[0058] Furthermore, a test interface is displayed on the display of Calculator 1, which includes a test image that can be deformed by a user's gestures and a slider for selecting the number of cancellation images n. This allows the user of Calculator 1 to select the optimal value for the number of cancellation images n while observing the actual deformation.

[0059] In the present embodiment, the drawing application 31 is described as performing the transformation of the image, but the desktop window manager 30a may also perform the transformation of the image.

[0060] FIG. 9 is a schematic block diagram showing functional blocks of the operating system 30 of the computer 2 according to a modification of this embodiment. According to this modification, the image PIC generated by the drawing application 31 is written to the transmission buffer 30c by the desktop window manager 30a. The desktop window manager 30a then sequentially reads report data R from the reception buffer 30b, detects gestures based on the series of report data R including the past report data R read up to that point, transforms the latest image PIC based on the detected gesture, and writes the resulting image PIC to the end of the transmission buffer 30c. The process of canceling the image transformation in this case is performed as an internal process of the desktop window manager 30a. This configuration also makes it possible to display the image PIC reflecting the results of the user's operations on the display 14 of the computers 1 and 2, while preventing the image PIC from being excessively transformed.

[0061] Additionally, while the present embodiment describes an example in which the number of cancellation images n is set based on the size of display 14 of Calculator 1, the number of cancellation images n may also be set based on the movement speed of the pointer on Calculator 1. The movement speed is the distance the pointer moves per unit time, and the unit of distance may be a length such as a sensor meter or a number of pixels. In this case, drawing application 31 preferably causes Calculator 1 to obtain an average value of the movement speed of the pointer, and the smaller the obtained average value, the smaller the number of cancellation images n. Since excessive deformation of the image becomes less noticeable when the movement speed of the pointer is slower, this also allows for the optimal amount of image deformation to be canceled out.

[0062] Next, an image processing method according to a second embodiment of the present invention will be described. The system configuration of computers 1 and 2 that execute the image processing method according to this embodiment is the same as that shown in FIG. 1. The image processing method according to this embodiment differs from the image processing method according to the first embodiment in that, instead of executing the image deformation cancellation process in computer 2, report data R sent from computer 1 is replaced with data of a smaller data size (specifically, report data for a keyboard or mouse) and then sent. The following description will focus on these differences.

[0063] 10 is a diagram showing the process flow of the report data transmission process executed by the operating system 30 of the computer 1. As shown in the figure, when the operating system 30 of the computer 1 according to this embodiment receives a position and data from the input sensor S40 (step S40), it temporarily stores the received position and data in the memory 11 (step S41). Then, it attempts to detect a gesture based on the series of positions and data received up to that point (step S42), and determines whether or not the gesture has been detected (step S43). The processes of steps S41 to S43 are nothing more than the processes of steps S5, S7, and S8 shown in FIG. 4 being executed within the computer 1.

[0064] If it is determined in step S43 that the gesture has been detected, the operating system 30 of the calculator 1 determines whether the detected gesture corresponds to a predefined gesture stored in a correspondence table (described later) (step S44). Details of the correspondence table will be described later with reference to FIG. 11. If it is determined in step S44 that the detected gesture corresponds to a predefined gesture, the operating system 30 of the calculator 1 obtains key operations and mouse operations corresponding to the detected gesture from the correspondence table and transmits report data for the keyboard or mouse indicating the obtained operations (report data for a device of a type different from the input sensor 15) to the calculator 2 (step S46). On the other hand, if it is determined in step S43 that the gesture has not been detected, or if it is determined in step S44 that the detected gesture does not correspond to a predefined gesture, the operating system 30 of the calculator 1 transmits report data for the input sensor 15 to the calculator 2 (step S47). The report data transmitted in step S47 is the same as the report data described in the first embodiment.

[0065] 11(a) and 11(b) are diagrams showing examples of correspondence tables that associate types of gestures with commands registered in the drawing application 31. A command is composed of either a key operation or a mouse operation, or a combination of these. The operating system 30 of the calculator 1 refers to this correspondence table in step S45 of FIG. 10 to obtain the key operation or mouse operation that corresponds to the detected gesture.

[0066] 11(a), a pinch out is associated with the simultaneous pressing of the "control" key and the "+" key, and a pinch in is associated with the simultaneous pressing of the "control" key and the "-" key. Therefore, if the gesture detected in step S42 is a pinch out, for example, operating system 30 of calculator 1 acquires the simultaneous pressing of the "control" key and the "+" key as the key operation or mouse operation corresponding to the detected gesture in step S44, and transmits report data indicating the simultaneous pressing of the "control" key and the "+" key in step S46.

[0067] 11(b), a pinch out is associated with a combination of "left click" and "scroll up with the mouse wheel," a pinch in is associated with a combination of "left click" and "scroll down with the mouse wheel," and a swipe is associated with a combination of "left click" and "movement direction." Thus, for example, if the gesture detected in step S42 is a swipe, operating system 30 of Calculator 1 acquires a combination of "left click" and "movement direction" as a key operation or mouse operation corresponding to the detected gesture in step S44, and transmits report data indicating the "left click" and "movement direction" in step S46. Note that the specific content of the "movement direction" may be determined based on the movement direction of the indicator indicated by a series of positions (positions of pen P or finger) input from input sensor 15.

[0068] As described above, according to the image processing method of this embodiment, a gesture performed with the pen P or a finger can be converted into small-sized data (specifically, report data for a keyboard or a mouse) and transmitted. This reduces processing delays, and as with the first embodiment, it is possible to prevent excessive deformation of an image due to a gesture caused by a processing delay.

[0069] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and it goes without saying that the present invention can be embodied in various forms without departing from the spirit of the present invention. [Explanation of symbols]

[0070] 1,2 Calculator 10 processors 11. Memory 12. Communications equipment 13 Input Devices 14 Display 15 Input Sensor 30 Operating Systems 30a Desktop Window Manager 30b Receive buffer 30c Send Buffer 31 Drawing Applications 40 Window images (test interface) 41 Drawing Area 42 test images 43 Slider 44 Decision button ER End Report FID finger ID Finger Track Start Tracking start information Finger Track Loss Information LID Local ID P Pen PD Position Data PenDown PenDown Information PenUp PenUp Information PIC Image PRE pen pressure data R Report Data SR Start Report

Claims

1. 1. A computer configured to communicate with a remote computer having an input sensor and a display, the computer generating an image to be displayed on the display of the remote computer, sequentially receiving a series of report data from the remote computer, each report data including a position of a pointer on a sensor surface of the input sensor detected by the input sensor; Detecting a gesture based on the series of report data received from the remote computer, applying a deformation of the content and amount indicated by the detected gesture to the image or a previously transmitted deformed image to generate a deformed image, and transmitting the deformed image to the remote computer; executing a process of canceling the deformation in response to the report data received from the remote computer being an end report indicating the separation of the indicator from the sensor surface; calculator.

2. the canceling process is a process of canceling, going back from the end report, deformation due to report data for a time period required from when the remote computer detects the position of the indicator until when the image corresponding to the position is displayed on the display; 2. The computer of claim 1.

3. a transmission buffer for storing the generated one or more transformed images; a transmission unit that sequentially reads out and transmits the one or more deformed images stored in the transmission buffer, the canceling process is a process of stopping transmission of at least a part of the one or more deformed images stored in the transmission buffer in response to the report data received from the remote computer being an end report indicating that the indicator has left the sensor surface.

2. The computer of claim 1.

4. the process of stopping transmission of at least a part of the one or more transformed images stored in the transmission buffer is a process of discarding the transformed images whose transmission is to be stopped from the transmission buffer.

4. The computer according to claim 3.

5. the canceling step includes generating a rewind image that would be displayed on the display if the gesture had not been performed, and transmitting the rewind image to the remote computer.

5. A computer according to any one of claims 1 to 4.

6. the canceling process is a process of canceling deformations of a predetermined number of report data items going back from the end report, the predetermined number is preset in the memory of the computer; 6. A computer according to any one of claims 1 to 5.

7. the predetermined number is determined based on environmental information indicating the environment of the remote computer; 7. The computer of claim 6.

8. a setting for whether or not to execute the cancellation process can be made depending on environment information indicating the environment of the remote computer; A computer according to any one of claims 1 to 7.

9. the environmental information includes one of a size of the display and a time required from when the remote computer detects the position of the indicator until the image corresponding to the position is displayed on the display; 9. The computer according to claim 7 or 8.

10. displaying a test interface for the cancellation process on the display; the test interface includes a test image to be transformed by the gesture and a component for setting whether or not to perform the cancellation process; Switching whether or not to execute the cancellation process depending on the setting in the component. A computer according to any one of claims 1 to 8.

11. the component includes a part configured to be able to set the degree of the cancellation process; Switching the degree of the cancellation process according to the settings in the component. The computer of claim 10.

12. The gesture is one of pinch out, pinch in, drag, and rotation. A computer according to any one of claims 1 to 11.

13. The transformation is one of enlargement, reduction, translation, and rotation.

13. A computer according to any one of claims 1 to 12.

14. The transformation is performed by either an application running on the operating system or a desktop window manager included in the operating system. A computer according to any one of claims 1 to 13.

15. the indicator is a finger, the remote computer is configured to perform a tracking process to track a position of the finger; The end report is data including information indicating that the movement of the finger that had been tracked as a series of positions by the tracking process is no longer detected.

15. A computer according to any one of claims 1 to 14.

16. the indicator is a pen configured to detect writing pressure and transmit the detected pressure to the remote computer; The end report is data including pen-up information indicating that the writing pressure has changed from a value greater than 0 to 0.

15. A computer according to any one of claims 1 to 14.

17. a computer configured to communicate with a remote computer having an input sensor and a display, the computer generating an image to be displayed on the display of the remote computer; sequentially receiving a series of report data from the remote computer, each report data including a position of a pointer on a sensor surface of the input sensor detected by the input sensor; detecting a gesture based on the series of report data received from the remote computer, and applying a deformation of the content and amount indicated by the detected gesture to the image or a previously transmitted deformed image to generate a deformed image and transmit the deformed image to the remote computer; and a step of executing a process of canceling the deformation in response to the report data received from the remote computer being an end report indicating the separation of the indicator from the sensor surface; A program to execute.

Images