Image processing method and information processing apparatus

Abstract

To enable matching of the appearance between a provisional image and a pen stroke image appearing in a streaming video. [Solution] An image drawing method executed by computers 1 and 2, including step S24 in which computer 2 transmits streaming video SV including a pen stroke image generated by a first rendering process, step S25 in which computer 1 generates a pen stroke image PS2 by executing a second rendering process shared with the first rendering process in parallel with the first rendering process, and step S27 in which computer 1 receives the streaming video SV from computer 2 via communication that causes a delay, synthesizes the pen stroke image PS2, and outputs it.

Description

Technical Field

[0001] The present invention relates to an image process method, and particularly to an image process method for performing pen input on a shared screen of a remote conference.

Background Art

[0002] Recently, remote conferences in which a plurality of computers are interconnected by a communication line have been increasingly used. In this type of remote conference, it is widely practiced to "share" a screen opened on one computer so that it can be viewed on other computers. In this case, a streaming video of the shared screen is transmitted from the computer that shares the screen (hereinafter referred to as the "remote-side device") to other computers (hereinafter referred to as the "local-side devices").

[0003] In addition, in a computer that supports pen input such as a tablet terminal, in order to reduce the influence of the processing delay of the host processor on the drawing of pen strokes, the sensor controller that detects the position of the pen may supply the position of the pen directly to the display processor without going through the host processor. Patent Documents 1 to 3 disclose examples of computers that perform such processing. Among these, Patent Document 1 describes that the position is supplied from the sensor controller to both the host processor and the display processor, and the display processor temporarily draws pen strokes until the pen strokes appear in the image generated by the host processor.

[0004] In addition, a technique for configuring to enable setting of pen tip attributes such as the size, shape, and color of the pen tip when performing pen input is known. Patent Documents 4 and 5 disclose examples of such techniques.

Prior Art Documents

Patent Documents

[0005] [Patent Document 1] U.S. Patent No. 9721365 [Patent Document 2] International Publication No. 2020 / 136729 [Patent Document 3] U.S. Patent No. 8564555 [Patent Document 4] U.S. Patent Publication No. 2005-0162413 [Patent Document 5] U.S. Patent No. 9792517 [Overview of the Initiative] [Problems that the invention aims to solve]

[0006] Incidentally, in the remote conference described above, there are cases where pen input is performed from the local device on streaming video transmitted from the remote device. In this case, the local device only performs the process of sequentially transmitting the position of the pen detected by the sensor controller to the remote device, and the rendering of the pen stroke and its synthesis into the streaming video is performed by the remote device. However, this process results in a significant delay before the pen stroke is reflected in the streaming video. Therefore, the inventors of this application are considering applying the technology described in Patent Documents 1 to 3 to temporarily draw the pen stroke on the local device until the pen stroke appears in the streaming video. Hereinafter, the image of the pen stroke drawn by this temporary drawing process will be referred to as a "provisional image".

[0007] However, the technologies described in Patent Documents 1 to 3 are based on drawing processing using a simple display processor and do not support advanced drawing processing that takes pen tip attributes into account, as described in Patent Documents 4 and 5. As a result, there may be cases where the appearance of the provisional image and the final pen stroke image that appears in the streaming video differ, so improvement was needed.

[0008] Therefore, one of the objectives of the present invention is to create an image in which the appearance of a provisional image and a pen stroke image appearing in the streaming video can be matched. process The objective is to provide a method.

[0009] Furthermore, streaming video generally includes areas that are not suitable for drawing pen strokes, such as menu bars. However, according to the technologies described in Patent Documents 1 to 3, the local device cannot identify areas within the streaming video suitable for drawing pen strokes (hereinafter referred to as the "drawing area"), and depending on the user's pen movements, provisional images may be drawn outside the drawing area, thus requiring improvement.

[0010] Therefore, another objective of the present invention is to prevent the provisional image from being drawn outside the drawing area. process The objective is to provide a method. [Means for solving the problem]

[0011] Image processing method according to a first aspect of the present invention is performed by a remote device and a local device. process The method is an image processing method comprising: the remote device transmitting streaming video including an image generated by a first rendering process; the local device generating a provisional image by executing a second rendering process shared with the first rendering process in parallel with the first rendering process; and the local device receiving the streaming video from the remote device via delayed communication, synthesizing the provisional image, and outputting it.

[0012] The image processing method according to a second aspect of the present invention is an image processing method performed by a remote device and a local device. processThe method is an image processing method comprising: the remote device transmitting streaming video; the remote device transmitting drawing area information indicating a drawing area in the streaming video; the local device receiving the drawing area information from the remote device; the local device generating a provisional image for provisional display in the streaming video; and the local device compositing the provisional image onto the area of ​​the streaming video indicated by the drawing area information and outputting it. [Effects of the Invention]

[0013] According to a first aspect of the present invention, since the second rendering process performed by the local device is shared with the first rendering process performed by the remote device, the local device can draw a provisional image using advanced drawing processing that takes pen tip attributes into consideration.

[0014] According to a second aspect of the present invention, since the region in which the provisional image is synthesized can be restricted to the region indicated by the drawing region information, it becomes possible to restrict the drawing of the provisional image outside the drawing region. [Brief explanation of the drawing]

[0015] [Figure 1] This figure shows the system configuration of computers 1 and 2 that perform the image processing method according to the first embodiment of the present invention. [Figure 2] (a) is a diagram showing examples of streaming video SV and desktop video DV1, DV1a, and (b) is a diagram showing examples of streaming video SV and desktop video DV1, DV1a while the user is performing pen input. [Figure 3] This figure shows the data format required by the rendering application 36 for rendering processing. [Figure 4](a) is a diagram showing the format of the application state information AS, and (b) is a diagram showing the format of the drawing application window position information GAWP. [Figure 5] It is a diagram showing an example of drawing the pen stroke image PS2 inside the drawing area A3. [Figure 6] It is a diagram showing the sequence of the image processing method according to the first embodiment of the present invention. [Figure 7] It is a diagram showing the sequence of the image processing method according to the first embodiment of the present invention. [Figure 8] (a) and (b) are diagrams respectively showing examples of the streaming video SV and the desktop videos DV1, DV1a while the user is performing a pen input. [Figure 9] (a) to (c) are diagrams showing examples of the pen stroke image PS2 that has the same shape as the pen stroke image PS1 but differs in either the pattern or the color or a combination of these, and (d) is a diagram showing an example of the pen stroke image PS2 that has the same trajectory (center line of the shape) as the pen stroke image PS1 but differs in shape. [Figure 10] It is a diagram showing examples of the pen stroke images PS1 and PS2 when the pen stroke image PS1 represents a special stroke. [Figure 11] It is a diagram showing the sequence of the image processing method according to the second embodiment of the present invention. [Figure 12] It is a diagram showing the sequence of the image processing method according to the third embodiment of the present invention. [Figure 13] It is a diagram showing the system configuration of the computer 3 that executes the image processing method according to the fourth embodiment of the present invention. [Figure 14] It is a diagram showing the system configuration of the computers 1 and 2 that execute the image processing method according to the fifth embodiment of the present invention.

Embodiments for Carrying Out the Invention

[0016] Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings.

[0017] Figure 1 shows the system configuration of computers 1 and 2 that execute the image processing method according to the first embodiment of the present invention. Computers 1 and 2 are information processing devices used for personal use, such as personal computers, tablet terminals, and smartphones. A predetermined conferencing application is installed on each of computers 1 and 2, and computers 1 and 2 conduct remote conferences with each other through this conferencing application. The type of conferencing application is not particularly limited, but in this embodiment, an application that has the function of sharing the screen (desktop screen or application window) of one computer (remote device) with the other computer (local device) is used.

[0018] In this embodiment, computer 1 is the local device and computer 2 is the remote device. When a user of computer 2, the remote device, performs a predetermined operation to share their desktop screen, the desktop screen of computer 2 is displayed within the desktop screen of computer 1, the local device. Hereinafter, the video constituting the desktop screen of computer 1 will be referred to as desktop video DV1, and the video constituting the desktop screen of computer 2 will be referred to as desktop video DV2. Furthermore, the area where desktop video DV2 is placed within desktop video DV1 will be referred to as the virtual desktop area VDI (see Figure 2 described later).

[0019] Computer 1 has the functionality to accept pen input from a pen P to the virtual desktop area VDI within its desktop screen. Specifically, it is configured to perform the following processes: detecting the position of the pen P within the virtual desktop area VDI, receiving data such as pen pressure values ​​from the pen P, and supplying pen data PD, which includes the detected position and received data, to Computer 2. Upon receiving the pen data PD, Computer 2 generates a pen stroke image PS1 based on the supplied pen data PD and displays it in the desktop video DV2. Since the desktop video DV2 is displayed in the virtual desktop area VDI, these processes result in the pen strokes input by the pen P being displayed on the desktop screens of both Computer 1 and Computer 2.

[0020] Figure 1 illustrates the components of computers 1 and 2 that are relevant to each of the processes described above. The configurations and operations of computers 1 and 2 will be explained in detail below, with reference to Figure 1.

[0021] Computers 1 and 2 are each configured to include a processor 20, a communication device 21, an input device 22, a video device 23, and a display 24. Computer 1 is further configured to include a pen input device 25.

[0022] The processor 20 is the central processing unit for computers 1 and 2, and is configured to read and execute programs that constitute the operating system, various device drivers, and various applications from memory (not shown). In Figure 1, each part shown within the processor 20 represents a functional unit that is realized by the execution of the corresponding program by the processor 20.

[0023] Specifically, the functional unit implemented by the processor 20 of computer 1 includes a USB (Universal Serial Bus) device driver 30, which is the device driver for the input device 22; a USB device driver 31, which is the device driver for the pen input device 25; a tablet driver 32, which controls the pen input device 25 via the USB device driver 31; a rendering engine 33, which draws the pen stroke image PS2 based on information supplied by the tablet driver 32; and a meeting application 35, which enables remote meetings with other computers. In addition, the functional unit implemented by the processor 20 of computer 2 includes the USB device driver 30, the tablet driver 32, and the meeting application 35, as well as a rendering engine agent 34, which functions as an agent for the rendering engine 33 within computer 2; and a drawing application 36, which draws the pen stroke image PS1 based on information supplied by the tablet driver 32. Details of the operation of each functional unit will be described later.

[0024] The communication device 21 is a device that has the function of communicating with other computers via the Internet or an ad-hoc network. This communication is performed according to the OSI (Open Systems Interconnection) reference model, and the display channel 40, data channel 41, tablet channel 42, and USB redirect channel 43 shown in the figure represent channels set at the transport layer, respectively.

[0025] The input device 22 is, for example, a mouse or a keyboard. The USB device driver 30 sequentially acquires operation information indicating the user's operation of the input device 22 and supplies it to each functional unit in the processor 20, and is also configured to supply it to USB device drivers 30 of other computers via the USB redirection channel 43. A USB device driver 30 that has acquired operation information from another computer processes the operation information from the other computer and supplies it to each functional unit in the processor 20, similar to the operation information acquired from the input device 22 directly connected to itself.

[0026] The pen input device 25 is a device that accepts pen input from a pen P. In a typical example, the pen P is a rod-shaped indicator that has the function of indicating a position on a plane, and in this case, the pen input device 25 is composed of a touch sensor having a touch surface and a sensor controller. The touch sensor is composed of a plurality of sensor electrodes arranged on the touch surface. The sensor controller is configured to detect a two-dimensional coordinate indicating the indicated position of the pen P using the touch sensor and to receive data such as pen pressure values ​​from the pen P. The method of detecting the position of the pen P by the sensor controller is not particularly limited, but it is preferable to use an active electrostatic method or an electromagnetic induction method. The pen P may also be a controller for 3D input in XR space, and in this case, the pen input device 25 is configured to detect a three-dimensional coordinate indicating the indicated position of the pen P in XR space and to receive data such as pen pressure values ​​from the pen P.

[0027] The pen input device 25 is configured to supply the detected pen position and data received from the pen P to the tablet driver 32 via the USB device driver 31 each time it detects the pen position P. The tablet driver 32 is configured to generate pen data PD, which includes the pen position and data supplied from the pen input device 25, and to sequentially supply it to the rendering engine 33, as well as to the tablet drivers 32 of other computers via the tablet channel 42.

[0028] The tablet driver 32 is also configured to acquire information about the portion of the desktop image of the computer on which it is running that corresponds to the input of the pen input device 25. Specifically, the tablet driver 32 of computer 1 acquires virtual desktop information VDP, which indicates the position and size of the virtual desktop area VDI within the desktop image DV1, from the conferencing application 35 or the operating system, and supplies it to the rendering engine 33. The tablet driver 32 of computer 2 acquires drawing application window position information GAWP, which indicates the position and size of the window W of the drawing application 36 within the desktop image DV2, and supplies it to the rendering engine agent 34. The drawing application window position information GAWP will be explained in more detail later.

[0029] The video device 23 is a device that generates desktop video to be displayed on the desktop according to the control of the processor 20 and supplies it to the display 24. The display 24 is responsible for displaying the desktop video thus supplied. In the example in Figure 1, the video device 23 of computer 1 generates desktop video DV1, and the video device 23 of computer 2 generates desktop video DV2. As will be described in more detail later, the video device 23 of computer 1 is configured to first generate desktop video DV1a, which includes streaming video SV transmitted from computer 2, and then generate desktop video DV1 by compositing the pen stroke image PS2 supplied from the rendering engine 33 onto the generated desktop video DV1a.

[0030] The video device 23 also performs the process of streaming part or all of the desktop video it is generating to another computer via the display channel 40, in response to the control of the conferencing application 35. The streaming video SV shown in Figure 1 represents the video transmitted by the video device 23 of computer 2 in this manner.

[0031] Figure 2(a) shows an example of streaming video SV and desktop video DV1, DV1a. The figure shows the case where streaming video SV is composed of the entirety of desktop video DV2, and the rendering engine 33 does not supply pen stroke image PS2. In this case, streaming video SV and desktop video DV2 are identical, and desktop video DV1 and desktop video DV1a are identical.

[0032] The window W shown in the streaming video SV is a window generated by the drawing application 36 and placed in the desktop video DV2. As in this example, the desktop video DV2 may contain various windows and icons. If the sharing target is the entire desktop video DV2, these windows and icons will also be included in the streaming video SV. On the other hand, if the sharing target is only a specific window displayed in the desktop video DV2, the streaming video SV will consist only of the video of the shared window.

[0033] The streaming video SV is placed within a virtual desktop area (VDI) provided in the desktop video DV1a. The video device 23 of computer 1 is configured to place the virtual desktop area (VDI) within the desktop video DV1a and then place the streaming video SV within it, in response to the control of the conferencing application 35. As a result, as shown in the desktop video DV1 of Figure 2(a), part or all of the desktop video DV2 of computer 2 is displayed on the display 24 of computer 1, allowing users of computers 1 and 2 to conduct remote meetings while viewing the same screen.

[0034] Returning to Figure 1, the conference application 35 is a functional unit that has the function of setting up a remote conference between multiple computers by communicating with conference applications 35 on other computers via the data channel 41. The conference application 35 on computer 2 also performs the process of controlling the video device 23 to send part or all of the desktop video DV2 to computer 1 as streaming video SV. In addition, the conference application 35 on computer 1 also performs the process of controlling the video device 23 to place a virtual desktop area VDI within the desktop video DV1 and place the streaming video SV supplied from computer 2 within it.

[0035] The drawing application 36 is a functional unit that generates a pen stroke image PS1 by performing rendering processing based on pen data PD supplied from the tablet driver 32. The drawing application 36 is configured to generate the window W shown in Figure 2(a) and place the generated pen stroke image PS1 within it. The window W containing the pen stroke image PS1 is placed in the desktop video DV2 by the video device 23.

[0036] The internal structure of window W is as shown in Figure 2(a). Window W consists of a main area A1, a canvas area A2, and other areas (header, footer, toolbar, various property panels, etc.). Main area A1 is the area for displaying canvas area A2. Canvas area A2 is configured to allow various operations such as scaling, rotation, and movement within main area A1 through user interaction. Canvas area A2 is the area that accepts drawing by pen P (drawing area). The drawing application 36 is configured not to draw parts of the generated pen stroke image PS1 that are located outside canvas area A2. Therefore, the pen stroke image PS1 is placed only inside canvas area A2.

[0037] The window W placed within the desktop video DV2, and the pen stroke image PS1 contained within it, will be displayed on the computer 1's display 24 as part of the streaming video SV. However, during the process from when the pen input from the pen P is reflected in the desktop video DV1, various delays occur, including network delays related to communication between computers 1 and 2, and image compression encoding delays and buffering delays that occur in the internal processing of computers 1 and 2. As a result, a discrepancy of a significant degree may occur between the tip of the pen P and the tip of the pen stroke image PS1 displayed on the computer 1's display 24. One of the objectives of the image processing method according to this embodiment is to reduce this discrepancy.

[0038] Figure 3 shows the formats of various data related to the rendering process performed by the drawing application 36. As shown in the figure, the various data include pen data, history data, and brush data. The drawing application 36 draws the pen stroke image PS1 based on this data.

[0039] The pen data is pen data PD supplied from the tablet driver 32. As shown in Figure 3, the pen data PD may include cursor information, button information, coordinate data, pressure sensitivity information, airbrush wheel information, tilt information, and rotation information. The pen data PD may also include serial information indicating its order within a series of pen data PDs.

[0040] The cursor information indicates whether the object being manipulated by the pen P is a cursor, and, if it is not a cursor (i.e., the pen P is being manipulated to draw a pen stroke), whether a switch on the pen P (for example, a tail switch located at the end of the pen P) is pressed. In a typical example, the cursor information is 0 if the object being manipulated by the pen P is a cursor, 1 if the object being manipulated by the pen P is not a cursor and the switch is not pressed, and 2 if the object being manipulated by the pen P is not a cursor and the switch is pressed. If the cursor information indicates that the object being manipulated by the pen P is a cursor, the drawing application 36 does not generate the pen stroke image PS1.

[0041] Button information indicates the on / off status of other switches on the pen P. Coordinate data is a two-dimensional or three-dimensional coordinate indicating the position of the pen P as detected by the sensor controller of the pen input device 25. Pen pressure value information indicates the pen pressure value (the value of the pressure applied to the tip of the pen P) detected by the pressure sensor on the pen P. The pen pressure value is greater than 0 when the tip of the pen P is in contact with the touch surface (when the pen is down), and is 0 when the tip of the pen P is not in contact with the touch surface (when hovering). Airbrush wheel information indicates the force applied to the wheel provided on the surface of the pen P if the pen P is an airbrush type. Tilt information indicates the tilt of the pen P relative to the touch surface. Rotation information indicates the amount of rotation of the pen P in the pen axis direction.

[0042] The history data is the history of the pen data PD. The drawing application 36 is configured to store a certain amount of pen data PD as history data. The drawing application 36 derives curves such as Bézier curves and Catmull-Romm curves based on a series of coordinates contained in both the pen data PD and the history data, and generates a pen stroke image PS1 by controlling the attributes of the curves (line width, transparency, color, etc.) based on other data contained in both the pen data PD and the history data, as well as the brush data described later. The drawing application 36 also derives the writing speed based on a series of coordinates contained in both the pen data PD and the history data when necessary for calculating the functions described later.

[0043] Brush data is information that indicates the pen tip attributes of pen P and is set by the user in the drawing application 36. Specifically, brush data consists of function information, color information, size information, pen tip shape information, brush deselection information, pattern bitmap, density data, density randomness, density random number seed, orientation data, orientation randomness, and orientation random number seed.

[0044] Function information is information that indicates a function for deriving the attributes of a pen stroke (line width, transparency, shading, density, etc.) based on information about the pen P, such as pen pressure, tilt, and writing speed. In a typical example, the function information indicates a function that defines the relationship between pen pressure and line width. Alternatively, the function information may indicate a function that defines the relationship between a combination of pen pressure and writing speed and line width, or a function that defines the relationship between a combination of pen tilt and writing speed and shading, or information that includes multiple functions. Furthermore, color information is information that identifies the drawing color (including transparency) of the pen stroke image. Size information is information that identifies the reference value for the line width of the pen stroke (the reference value for line width changes accompanying changes in pen pressure). The size information may also include a function that shows the relationship between the time the pen P stays at the same position and the line width (for example, a function to reproduce the appearance of ink bleeding by increasing the line width the longer the pen P stays at the same position). Pen tip shape information identifies the type of pen tip shape, such as ballpoint pen type or color marker type. The drawing application 36 controls curve attributes (line width, transparency, color, etc.) based on this information.

[0045] The brush selection deselection information indicates that a tool that does not require drawing pen strokes, such as area selection or fill, is selected. If the brush selection deselection information indicates that a tool that does not require drawing pen strokes is selected, the drawing application 36 does not generate the pen stroke image PS1.

[0046] The pattern bitmap, density data, density randomness, density random number seed, orientation data, orientation randomness, and orientation random number seed are information referenced by the drawing application 36 when generating a pen stroke image PS1 by particle scattering. Specifically, the pattern bitmap is image data that shows the individual parts of a curve due to particle scattering. The density data is data that indicates a reference value for the arrangement density of the pattern bitmap, and the orientation data is data that indicates a reference value for the orientation of the arrangement of the pattern bitmap. The drawing application 36 is configured to generate a pen stroke image PS1 by particle scattering by arranging the pattern bitmap along a curve derived based on coordinate data, with an arrangement density randomly determined around the arrangement density indicated by the density data, and an arrangement orientation randomly determined around the orientation of the arrangement indicated by the orientation data.

[0047] The coarse-density randomness, coarse-density random number seed, orientation randomness, and orientation random number seed are values ​​set in the random number generator for generating random numbers, respectively. The randomness indicates the degree of deviation between the pseudorandom number sequence generated by the random number generator and the ideal random number sequence, and the random number seed indicates the initial value set in the random number generator. The drawing application 36 is configured to determine the arrangement density of the pattern bitmap based on the pseudorandom numbers obtained by setting the coarse-density randomness and coarse-density random number seed in the random number generator, and to determine the orientation of the arrangement of the pattern bitmap based on the pseudorandom numbers obtained by setting the orientation randomness and orientation random number seed in the random number generator.

[0048] Returning to Figure 1, the drawing application 36 also generates application state information AS that characterizes the rendering process it performs and supplies it to the rendering engine agent 34.

[0049] Figure 4(a) shows the format of the application state information AS. As shown in the figure, the application state information AS consists of drawing setting information and canvas area information.

[0050] Drawing settings information is information that identifies the appearance of pen strokes drawn by the rendering process, and consists of brush data and brush parameter change information. The brush data is the brush data itself shown in Figure 3. The brush parameter change information is information that identifies the parts of the brush data that have been changed since the previous application state information AS.

[0051] The canvas area information is information about the canvas area A2 (the area that accepts drawing with pen P) within window W shown in Figure 2(a), and consists of initial state information, scaling information, rotation information, movement information, and drawing area information. The initial state information indicates the initial position and size of canvas area A2 within window W. The scaling information indicates the scaling ratio of the display of canvas area A2 from the initial size indicated by the initial state information. The rotation information indicates the rotation angle of canvas area A2. The movement information indicates the amount of movement of canvas area A2 from the initial position indicated by the initial state information. The drawing area information indicates the position, size, shape, and background color of drawing area A3 within canvas area A2, as shown in Figure 2(a).

[0052] Returning to Figure 1, the rendering engine agent 34 is a functional unit that acts as an agent for the rendering engine 33 and is attached to the rendering engine 33 when the rendering engine 33 starts drawing the pen stroke image PS2. The rendering engine agent 34 is configured to obtain drawing application window position information GAWP from the tablet driver 32 and application state information AS from the drawing application 36, and to supply this obtained information to the rendering engine 33 via the tablet channel 42.

[0053] Figure 4(b) shows the format of the drawing application window position information GAWP. As shown in the figure, the drawing application window position information GAWP consists of position information, size information, active information, inactive information, menu open / close information, and child window expansion information.

[0054] The position information indicates the position of window W within the desktop video DV2, and the size information indicates the size of window W. The active information is true if window W is in the foreground and false otherwise, and the inactive information is true if window W is in the background and false otherwise. When the active information is false and the inactive information is true, the drawing application 36 is in a state where it does not accept pen input (it does not generate the pen stroke image PS1).

[0055] The menu open / closed information is true if one of the menus within window W is open, and false otherwise. The child window expanded information is true if a child window (such as a folder selection screen for saving files or a settings screen) is open within window W, and false otherwise. Even when the menu open / closed information is true, and when the child window expanded information is true, the drawing application 36 is in a state where it does not accept pen input (it does not generate the pen stroke image PS1).

[0056] Returning to Figure 1, the rendering engine 33 is a functional unit that generates a pen stroke image PS2 (provisional image) by executing a rendering process (hereinafter referred to as the "second rendering process") shared with the rendering process performed by the drawing application 36 (hereinafter referred to as the "first rendering process") in parallel with the first rendering process. Sharing the first rendering process and the second rendering process essentially means matching the appearance of the pen stroke image PS2 to the appearance of the pen stroke image PS1, and includes sharing the drawing application window position information GAWP and application state information AS between computers 1 and 2.

[0057] More specifically, the rendering engine 33 obtains the position and size of the virtual desktop area VDI within the desktop video DV1 based on the virtual desktop information VDP supplied from the tablet driver 32. Subsequently, the rendering engine 33 obtains the position and size of window W within the virtual desktop area VDI based on the drawing application window position information GAWP received from the rendering engine agent 34. The rendering engine 33 further obtains the position, size, shape, and background color of the drawing area A3 within window W based on the canvas area information in the application state information AS received from the rendering engine agent 34. The rendering engine 33 is configured to draw the pen stroke image PS2 only within the thus obtained drawing area A3. As a result, the pen stroke image PS2 is drawn at the same position as the pen stroke image PS1, which is displayed with a delay.

[0058] Here, Figure 5 shows an example of drawing a pen stroke image PS2 inside the drawing area A3. The tablet driver 32 is configured to generate pen data PD and supply it to the rendering engine 33 even when the indicated position of the pen P is outside the drawing area A3. Positions P1 to P4 shown in Figure 5 represent the indicated positions of the pen P included in a series of pen data PD supplied from the tablet driver 32 to the rendering engine 33, of which positions P1 and P2 are outside the drawing area A3. The rendering engine 33 draws the pen stroke image PS2 while referring to positions outside the drawing area A3, such as positions P1 and P2, as needed. That is, for example, four positions are required when deriving the Catmull-Rom curve described above. Therefore, the rendering engine 33 derives the curve by referring to the four positions, including the positions outside the drawing area A3, and draws the pen stroke image PS2 by drawing only the portion that is inside the drawing area A3.

[0059] Returning to Figure 1, the rendering engine 33 acquires brush data to be used in the first rendering process based on the drawing setting information in the application state information AS received from the rendering engine agent 34. Then, using the acquired brush data, it performs a second rendering process based on the pen data PD supplied from the tablet driver 32 to generate a pen stroke image PS2. As a result, the pen stroke image PS2 has the same appearance as the pen stroke image PS1.

[0060] Furthermore, the rendering engine 33 determines whether the drawing application 36 is generating the pen stroke image PS1 based on the drawing application window position information GAWP and application state information AS received from the rendering engine agent 34. Only if it determines that the drawing application 36 is generating the pen stroke image PS1, it performs a second rendering process to generate the pen stroke image PS2. This prevents a situation where the pen stroke image PS2 is generated even though the pen stroke image PS1 has not been generated.

[0061] Figure 2(b) shows an example of streaming video SV and desktop video DV1, DV1a while the user is performing pen input. The figure also shows the pen stroke image PS2 generated by the rendering engine 33. As shown in the figure, the rendering engine 33 is configured to virtually set up a virtual desktop area VDI, a window W, and a drawing area A3, and to draw the pen stroke image PS2 only within the drawing area A3.

[0062] Returning to Figure 1, the rendering engine 33 also processes the generated pen stroke image PS2 to the video device 23. The video device 23 generates desktop video DV1 by compositing the pen stroke image PS2 with the desktop video DV1a it generates internally, and supplies it to the display 24. As a result, desktop video DV1 including the pen stroke image PS2 is displayed on the display 24, as shown in Figure 2(b). Therefore, the user of computer 1 can see the pen stroke image PS2, which has the same appearance as the pen stroke image PS1, at the same position as the pen stroke image PS1, before the pen stroke image PS1, which tends to have a display delay, is displayed.

[0063] Figures 6 and 7 show the sequence of the image processing method according to this embodiment. The processing performed by computers 1 and 2 will be explained in more detail below, with reference to Figures 6 and 7.

[0064] First, referring to Figure 6, the user initially performs an operation to start a remote meeting on the meeting application 35 (step S1). This operation can be performed using the input device 22 or the pen input device 25. This is also true for each operation described below. Note that Figure 6 only shows the operation performed by the user on computer 1, but the operation to start a meeting is performed on computers 1 and 2 respectively.

[0065] Upon receiving the operation in step S1, the conference application 35 on computers 1 and 2 executes a process to start a remote conference through the data channel 41 shown in Figure 1 (step S2). This initiates a remote conference between computers 1 and 2.

[0066] The conference application 35 on computer 2, which initiated the remote conference, starts streaming all or part of the desktop video DV2 shown in Figure 1, in response to the user's operation on computer 2 (step S3). This starts the transmission of streaming video SV, which includes all or part of the desktop video DV2, via the display channel 40 shown in Figure 1, and the streaming video SV received from computer 2 appears in the desktop video DV1 displayed on computer 1's display 24.

[0067] Next, the user of computer 1 performs an operation on the conference application 35 to start pen input using pen P (step S4). Upon receiving this operation, the conference application 35 notifies the rendering engine 33 and the conference application 35 on computer 2 that it has started pen input (step S5).

[0068] Upon receiving notification in step S5, the rendering engine 33 obtains the virtual desktop information VDP from the tablet driver 32 (step S6) and sets up a canvas for drawing strokes at the location of the virtual desktop area VDI indicated therein (step S7). The rendering engine 33 also attaches to the rendering engine agent 34 in computer 2 via the tablet channel 42 shown in Figure 1 (step S8).

[0069] Furthermore, upon receiving the notification in step S5, the conference application 35 on computer 2 launches the drawing application 36 (step S9). The launched drawing application 36 then begins accepting pen data PD from the tablet driver 32 (step S10).

[0070] In step S8, the rendering engine agent 34 attached to the rendering engine 33 obtains drawing application window position information GAWP from the tablet driver 32 (step S11) and application state information AS from the drawing application 36 (step S12), and transmits this obtained information to the rendering engine 33 through the tablet channel 42 shown in Figure 1 (step S13).

[0071] Upon receiving the drawing application window position information GAWP and application state information AS from the rendering engine agent 34, the rendering engine 33 performs a sharing process to share the first rendering process performed by the drawing application 36 (step S14). Specifically, the rendering engine 33 obtains the position and size of window W within the canvas set in step S7 based on the drawing application window position information GAWP, and further obtains the position, size, shape, and background color of the drawing area A3 within window W based on the canvas area information in the application state information AS. The rendering engine 33 also obtains the brush data used in the first rendering process based on the drawing setting information in the application state information AS, and confirms that the drawing application 36 is generating the pen stroke image PS1 based on the drawing application window position information GAWP and the application state information AS. By performing this sharing process, the rendering engine 33 makes it possible to draw a pen stroke image PS2 having the same appearance as the pen stroke image PS1 within the drawing area A3.

[0072] Here, it is preferable that the rendering engine agent 34 periodically acquires and transmits the drawing application window position information GAWP and application state information AS to the rendering engine 33. Alternatively, the rendering engine agent 34 may monitor changes in the drawing application window position information GAWP and application state information AS, and acquire and transmit them to the rendering engine 33 whenever they change. For example, the rendering engine agent 34 may transmit the drawing application window position information GAWP when the drawing application 55 comes to the foreground and as a result the active information in the drawing application window position information GAWP changes.

[0073] Furthermore, the rendering engine 33 may determine whether the pen stroke image PS2 generated by the rendering engine 33 is different from the pen stroke image PS1 received thereafter. If it determines that they are different, the rendering engine 33 requests the rendering engine agent 34 to provide the drawing application window position information GAWP and the application state information AS. In response to this request, the rendering engine agent 34 may send the drawing application window position information GAWP and the application state information AS to the rendering engine 33. This makes it possible to quickly adjust the pen stroke image PS2 to match the pen stroke image PS1 if a difference occurs between the pen stroke image PS2 and the pen stroke image PS1 due to a change in the drawing application window position information GAWP or the application state information AS.

[0074] Furthermore, it is preferable that the rendering engine 33 re-executes the sharing process in step S14 whenever it receives drawing application window position information GAWP or application state information AS from the rendering engine agent 34. Alternatively, the rendering engine 33 may periodically check whether the rendering engine agent 34 has acquired new drawing application window position information GAWP or application state information AS, and re-execute the reception of the drawing application window position information GAWP or application state information AS and the sharing process in step S14 as necessary. This enables the rendering engine 33 to always generate the pen stroke image PS2 based on the latest information.

[0075] Next, refer to Figure 7. When the user operates the pen P (step S20), the tablet driver 32 of computer 1 acquires pen data PD. The tablet driver 32 that acquired the pen data PD supplies the acquired pen data PD to the rendering engine 33 and also supplies it to the tablet driver 32 of computer 2 using the tablet channel 42 shown in Figure 1 (step S21). The process from step S21 to step S30, which will be described below, is repeated each time the tablet driver 32 of computer 1 acquires new pen data PD.

[0076] The tablet driver 32 of computer 2 supplies the supplied pen data PD to the drawing application 36 (step S21). The drawing application 36 receives this and performs a first rendering process based on the data shown in Figure 3 (step S22), and supplies the resulting window W containing the pen stroke image PS2 to the video device 23 of computer 2. As a result, the streaming video SV containing the pen stroke image PS2 obtained by the first rendering process is transmitted from computer 2 to computer 1 (step S24), and eventually the pen stroke image PS2 appears in the desktop video DV1.

[0077] Meanwhile, the rendering engine 33 of computer 1 generates a pen stroke image PS2 (step S25) by executing a second rendering process, which was shared with the first rendering process by the shared processing performed in step S14 shown in Figure 6, in parallel with the first rendering process. Specifically, first, it derives curves such as Bézier curves and Catmull-Romm curves based on a series of instruction positions included in the latest predetermined number of pen data PDs. Then, based on the derived curves, it generates a pen stroke image PS2 based on the data in each pen data PD and the brush data acquired in step S14, and draws only the portion of the generated pen stroke image PS2 that is located within the drawing area A3 whose position etc. was acquired in step S14. However, if the rendering engine 33 indicates that the drawing application 36 has not generated a pen stroke image PS1 by the drawing application window position information GAWP or the application state information AS, it will not generate a pen stroke image PS2.

[0078] The rendering engine 33 supplies the pen stroke image PS2 generated in step S25 to the video device 23 (step S26). The video device 23 generates desktop video DV1 by compositing the supplied pen stroke image PS2 with desktop video DV1a and outputs it to the display 24 shown in Figure 1 (step S27). As a result, the user of computer 1 can see the pen stroke image PS2, which has the same appearance as the pen stroke image PS1, at the same position as the pen stroke image PS1 that will be displayed later, before the pen stroke image PS2 is reflected in the desktop video DV1.

[0079] The rendering engine 33 performs a process to erase the pen stroke image PS2 from the desktop video DV1 after a predetermined time has elapsed since generating the pen stroke image PS2 in step S25 (step S28). This process can be performed by erasing the portion newly generated in step S25, which was elapsed a predetermined time earlier, from the most recent pen stroke image PS2 and generating a new pen stroke image PS2. More specifically, the rendering engine 33 stores the pen data PD supplied from the tablet driver 32, prepares a transparent screen each time a drawing occurs, and erases the pen stroke image PS2 by drawing only the portion of the stored pen data PD that is to be drawn (the portion that has not yet been drawn for a predetermined time) onto the screen.

[0080] The rendering engine 33 supplies the pen stroke image PS2 generated in step S25 to the video device 23 (step S29). The video device 23 generates the desktop video DV1 in the same manner as in step S27 and outputs it to the display 24 shown in Figure 1 (step S30). As a result, the portion of the pen stroke image PS2 generated a predetermined time ago disappears from the desktop video DV1.

[0081] Figures 8(a) and 8(b) show examples of streaming video SV and desktop video DV1, DV1a while the user is performing pen input, respectively. Figure 8(a) shows the state after a predetermined time from Figure 2(b), and Figure 8(b) shows the state after a predetermined time from Figure 8(a). In actual processing, the erasure process in step S28 is performed each time new pen data PD is acquired, but for ease of understanding, in Figures 8(a) and 8(b), the illustrated pen strokes are shown to be erased by two erasure processes. The processes in steps S28 to S30 will be explained in more detail below with reference to Figures 8(a) and 8(b).

[0082] First, referring to Figure 8(a), at this point, the pen stroke image PS1 appearing in the streaming video SV contains only the first half of the illustrated pen stroke. If the first half of the illustrated pen stroke is deleted from the pen stroke image PS2 at this timing, the pen stroke will be displayed on the desktop video DV1 without overlapping pen stroke images PS1 and PS2, and without interruption.

[0083] Next, referring to Figure 8(b), at this point, the pen stroke image PS1 appearing in the streaming video SV includes the entire illustrated pen stroke. If the latter half of the illustrated pen stroke is deleted from the pen stroke image PS2 at this timing, the pen stroke will continue to be displayed in the desktop video DV1 without overlapping or interruption between the pen stroke image PS1 and the pen stroke image PS2.

[0084] In this way, by sequentially erasing the portion of the pen stroke drawn by the user's pen input that appears in the streaming video SV from the pen stroke image PS2, the pen stroke images PS1 and PS2 will not overlap in the desktop video DV1 and the pen stroke image PS2 will be displayed without interruption. If the time required from when the user makes a pen input until the resulting pen stroke image PS1 appears in the streaming video SV is constant, then, as shown in Figure 7, by performing the rendering process in step S25 and then the erasure process in step S28 after a predetermined time, it becomes possible to display such a pen stroke image.

[0085] Here, the predetermined time from generation to erasure of the pen stroke image PS2 may be set in the rendering engine 33 by either computer 1 or 2, but it is preferable to measure the actual delay time by performing a predetermined calibration process and set the time according to the measured delay length. In particular, when it takes time to derive and control the attributes of the pen stroke using brush data, it is preferable for computer 2 to set the predetermined time in the rendering engine 33. This makes it possible to erase the pen stroke image PS2 more appropriately.

[0086] Furthermore, while Figure 8(a) shows an example where the appearance of pen stroke image PS2 perfectly matches that of pen stroke image PS1, the appearance of pen stroke image PS2 and the appearance of pen stroke image PS1 do not necessarily have to be perfectly identical. In other words, at the time of the erasure process in step S28 shown in Figure 7, the appearance of the portion of the pen stroke image corresponding to pen stroke image PS1 and the appearance of the portion corresponding to pen stroke image PS2 do not necessarily have to be perfectly identical. Below, we will describe a pen stroke image PS2 that has an appearance that does not perfectly match that of pen stroke image PS1, referring to specific examples.

[0087] Figures 9(a) to 9(c) show examples of pen stroke images PS2 that have the same shape as pen stroke image PS1, but differ in either the pattern, color, or a combination thereof. In these figures, pen stroke image PS1 is drawn with solid black lines, while pen stroke image PS2 in Figure 9(a) is drawn with hatched lines, pen stroke image PS2 in Figure 9(b) is drawn with lines that have outlines, and pen stroke image PS2 in Figure 9(c) is drawn with lines of a different color (so-called zebra pattern lines) at predetermined time intervals (e.g., every 10 milliseconds). By making either the pattern, color, or a combination thereof differ between pen stroke image PS1 and pen stroke image PS2 in this way, the user of computer 1 can identify which part is a provisional image. Furthermore, by changing the drawing color of the pen stroke image PS2 at predetermined intervals, as shown in Figure 9(c), it becomes possible to determine the approximate delay time at a given point in time by counting the number of points in the pen stroke image PS2 displayed on the display 24 that have undergone a color change.

[0088] Furthermore, when the pattern and / or color, or a combination thereof, differ between pen stroke image PS1 and pen stroke image PS2, "color" may include transparency. Also, the method of filling the inside of the outline of pen stroke image PS2 shown in Figure 9(b) is not particularly limited. For example, it may be filled with solid color, hatched, transparent, or semi-transparent.

[0089] Furthermore, when generating a pen stroke image PS2 with a different drawing color (including transparency) than the pen stroke image PS1, it is preferable that the rendering engine 33 determines the drawing color of the pen stroke image PS2 based on the color information (information that identifies the drawing color of the pen stroke image PS1) included in the application state information AS obtained from the computer 2. This makes it possible for the rendering engine 33 to generate a pen stroke image PS2 with a color that is definitely different from the pen stroke image PS1 (for example, the complementary color of the drawing color of the pen stroke image PS1). It is even more preferable that the rendering engine 33 also determines the color of the pen stroke image PS2 based on the background color of the drawing area A3 included in the application state information AS obtained from the computer 2. This prevents the drawing color of the pen stroke image PS2 from becoming the same as or similar to the background color of the drawing area A3, which would make it difficult to see the pen stroke image PS2, as a result of making the drawing color of the pen stroke image PS2 different from the drawing color of the pen stroke image PS1.

[0090] The issue of the drawing color of the pen stroke image PS2 being the same as or similar to the background color of the drawing area A3, making the pen stroke image PS2 difficult to see, can be prevented by using a pen stroke image PS2 as shown in the examples in Figure 9(b) and Figure 9(c). That is, the rendering engine 33 may ensure the visibility of the pen stroke image PS2 by generating the pen stroke image PS2 using a multi-color pattern of two or more colors, as shown in the example in Figure 9(c), or by providing an outline to the pen stroke image PS2, as shown in the example in Figure 9(b). In addition, the visibility of the pen stroke image PS2 can also be ensured by changing the drawing color or pattern of the pen stroke image PS2 over time (for example, by making it blink).

[0091] Figure 9(d) shows an example of a pen stroke image PS2 in which the trajectory (centerline of the shape) matches that of the pen stroke image PS1, but the shape is different. The rendering engine 33 in the example shown generates the pen stroke image PS2 with a thinner line width than the line width of the pen stroke image PS1. In other words, the rendering engine 33 in the example shown generates the pen stroke image PS2 using the same interpolation curve (such as the Bezier curve or Catmull-Romm curve mentioned above) as the pen stroke image PS1, but with a different line width. This reduces the degree of delay perceived by the user compared to cases such as changing the drawing color, and because the trajectories of the pen stroke image PS1 and the pen stroke image PS2 match, it is possible to achieve processing that is less jarring to the user.

[0092] Figure 10 shows examples of pen stroke images PS1 and PS2 when the pen stroke image PS1 generated by the drawing application 36 represents a stroke that is not a stroke for drawing lines (hereinafter referred to as a "special stroke"). In this example, the pen stroke image PS1 represents a stroke for color mixing. Figure 10 shows a state where a region AR1 painted with a relatively light color and a region AR2 painted with a relatively dark color are displayed adjacent to each other in the virtual desktop area VDI, and the user moves the pen P near the boundary between regions AR1 and AR2, resulting in the input pen stroke image PS1 mixing the colors of regions AR1 and AR2. In this case, the pen stroke image PS1 is indicated by the color mixing region. Other examples of special strokes include strokes for sculpting (clay modeling) 3D models. On the other hand, the pen stroke image PS2 is drawn with lines that have outlines, similar to the example shown in Figure 9(b). However, Figure 10 shows an example in which the line width of the pen stroke image PS2 is narrower than that of the pen stroke image PS1, and the inside of the contour lines that make up the pen stroke image PS2 is made transparent.

[0093] If the pen stroke image PS1 is a special stroke, it may be difficult for the rendering engine 33 to share the first rendering process performed by the drawing application 36, even if the drawing application window position information GAWP and application state information AS are used. In the example in Figure 10, information on regions AR1 and AR2 is essential for the rendering engine 33 to draw a pen stroke image PS2 that has the exact same appearance as the pen stroke image PS1. However, the drawing application window position information GAWP and application state information AS do not include information on regions AR1 and AR2. Therefore, even with the sharing process in step S14 described above, the appearances of the pen stroke image PS1 and the pen stroke image PS2 will not be perfectly identical. If such a pen stroke image PS2 were to be drawn over the pen stroke image PS1, the unnatural pen stroke image PS2 would hide the pen stroke image PS1, potentially increasing the user's sense of unease. In this regard, as illustrated in Figure 10, by constructing the pen stroke image PS2 with an outline, it becomes possible to inform the user of the position during operation without causing any sense of unease. Note that Figure 10 shows an example where the drawing color of the area enclosed by the outline is made transparent, but it may also be made semi-transparent.

[0094] As described above, according to the image processing method of this embodiment, the second rendering process performed by computer 1 is shared with the first rendering process performed by computer 2. Therefore, computer 1 can draw the provisional image, the pen stroke image PS2, using advanced drawing processing that takes pen tip attributes into consideration.

[0095] Furthermore, according to the image processing method of this embodiment, the area in which the pen stroke image PS2 is synthesized by the video device 23 can be limited to within the drawing area A3, making it possible to restrict the drawing of the pen stroke image PS2 outside of the drawing area A3.

[0096] Furthermore, according to the image processing method of this embodiment, since a process is performed to erase the pen stroke image PS2 after a predetermined time has elapsed since its generation, it becomes possible to display the pen stroke images in the desktop video DV1 without the pen stroke image PS1 and the pen stroke image PS2 overlapping or being interrupted.

[0097] In this embodiment, in step S28 of Figure 7, the pen stroke image PS2 is erased in small increments based on the indicated position of the pen P. However, the rendering engine 33 may erase the pen stroke image PS2 for multiple indicated positions all at once. This reduces the load on the processor 20.

[0098] Figure 11 is a diagram showing a sequence of images according to a second embodiment of the present invention. The image processing method according to this embodiment differs from the image processing method according to the first embodiment in terms of the method for erasing the pen stroke image PS2. The following explanation will focus on the differences from the image processing method according to the first embodiment.

[0099] As can be seen by comparing Figure 11 and Figure 7, the processing up to step S27 is the same as the processing according to the first embodiment. In this embodiment, the rendering engine 33 that has performed the processing in steps S25 and S26 acquires pixel information at the position corresponding to the portion of the pen stroke image PS2 newly generated in step S25 of the desktop video DV1a, instead of waiting for a predetermined time (step S40). The rendering engine 33 repeats this process in step S40, for example, periodically.

[0100] When the portion of the pen stroke image PS1 corresponding to the portion of the pen stroke image PS2 newly generated in step S25 is included in the streaming video SV, a change appears in the pixel information in step S40. When the rendering engine 33 detects this change (step S41), it erases the corresponding portion of the pen stroke image PS2 (step S42) and supplies the newly generated pen stroke image PS2 to the video device 23 (step S43). Upon receiving this pen stroke image PS2, the video device 23 generates desktop video DV1 in the same manner as in step S30 shown in Figure 7 and outputs it to the display 24 shown in Figure 1 (step S44).

[0101] By employing this processing method, the rendering engine 33 can continuously erase the portion of the pen stroke image PS2 that appears in the streaming video SV. Therefore, according to the image processing method of this embodiment, even if the delay increases, it becomes possible to display the pen strokes without interruption.

[0102] Furthermore, the method for erasing the pen stroke image PS2 according to this embodiment may be used in combination with the method for erasing the pen stroke image PS2 described with reference to Figure 7. According to the method for erasing the pen stroke image PS2 according to this embodiment, if the drawing application 36 draws the pen stroke image PS2 using a function to reproduce the appearance of ink bleeding (a function that increases the line width the longer the pen P stays at the same position), then if the pen P remains at the same position on the touch surface, no change will appear in the pixels around the pen tip, and the pen stroke image PS1 will never be erased. By using the method for erasing the pen stroke image PS2 described with reference to Figure 7 in combination, even in such cases, it becomes possible to erase the pen stroke image PS1 after a predetermined time has elapsed.

[0103] Figure 12 shows a sequence illustrating an image processing method according to a third embodiment of the present invention. This image processing method also differs from the image processing method according to the first embodiment in terms of the method for erasing the pen stroke image PS2. The following explanation will focus on the differences from the image processing method according to the first embodiment.

[0104] As can be seen by comparing Figure 12 and Figure 7, the processing up to step S27 is the same as the processing according to the first embodiment. In this embodiment, after supplying pen data PD to the drawing application 36 in step S21 (i.e., after rendering by the drawing application 36 has started), the tablet driver 32 of computer 2 transmits the serial information SN of the pen data PD to the tablet driver 32 of computer 1 via the tablet channel 42 (step S50). The tablet driver 32 of computer 1 then transfers the serial information SN received in this manner to the rendering engine 33 (step S51).

[0105] The rendering engine 33, having received serial information SN from the tablet driver 32, processes the portion of the generated pen stroke image PS2 corresponding to the supplied serial information SN to be erased (step S52), and supplies the newly generated pen stroke image PS2 to the video device 23 (step S53). The video device 23, having received this pen stroke image PS2, generates desktop video DV1 in the same manner as in step S30 shown in Figure 7, and outputs it to the display 24 shown in Figure 1 (step S54).

[0106] By employing this processing method, the rendering engine 33 can continuously erase the portion of the generated pen stroke image PS2 that corresponds to the pen data PD that has started rendering in the computer 2. Since the supply of serial information SN to the computer 1 usually precedes the supply of streaming video SV, the image processing method according to this embodiment makes it possible to erase the pen stroke image PS2 in time with the timing of the display of the pen stroke image PS1.

[0107] Furthermore, the method for erasing the pen stroke image PS2 according to this embodiment may be used in combination with the method for erasing the pen stroke image PS2 described with reference to Figure 7. In this way, even if the transmission of serial information SN in step S50 is delayed and erasure based on serial information SN cannot be performed at the appropriate timing, it will be possible to reliably erase the pen stroke image PS1 after a predetermined time has elapsed. Moreover, the method for erasing the pen stroke image PS2 according to this embodiment may be used in combination with the method for erasing the pen stroke image PS2 described with reference to Figure 11, or the method for erasing the pen stroke image PS2 according to this embodiment may be used in combination with both the method for erasing the pen stroke image PS2 described with reference to Figure 7 and the method for erasing the pen stroke image PS2 described with reference to Figure 11.

[0108] Furthermore, the tablet driver 32 of computer 2 may choose to send serial information SN intermittently, such as every few pen data PDs, rather than sending serial information SN for every pen data PD. This makes it possible to reduce the communication resources used for sending serial information SN.

[0109] Figure 13 shows the system configuration of computer 3 that executes the image processing method according to the fourth embodiment of the present invention. The image processing method according to this embodiment differs from the image processing method according to the first embodiment in that it is executed by a single computer 3. The configuration and operation of computer 3 will be described below, focusing on the differences from computers 1 and 2 according to the first embodiment.

[0110] Computer 3 differs from Computer 1 shown in Figure 1 in that it runs a virtual machine 50. The virtual machine 50 is a virtual computer used to run a second operating system on Computer 3, and is realized by the processor 20 executing a predetermined program. A typical example is the use of the virtual machine 50 when a drawing application that does not run on the operating system of Computer 3 is desired.

[0111] As shown in Figure 13, the virtual machine 50 is configured to include a communication unit 51, a USB device driver 52, a tablet driver 53, a rendering engine agent 54, a drawing application 55, and a video device 56. Their basic operation is the same as that of the communication device 21, USB device driver 30, tablet driver 32, rendering engine agent 34, drawing application 36, and video device 23 shown in Figure 1. However, in computer 3, the streaming video SV becomes the video displaying window W. Furthermore, the drawing application 55 is launched by user operation on computer 3.

[0112] In this embodiment, the processor 20 functions as the local device, and the virtual machine 50 functions as the remote device. A streaming video SV containing a window W generated by the drawing application 55 is supplied from the virtual machine 50 to the processor 20 and placed within the desktop video DV1. Similar to the first embodiment, the second rendering process performed by the processor 20 is shared with the first rendering process performed by the virtual machine 50. Therefore, with the image processing method according to this embodiment, the processor 20 can also draw a provisional image, the pen stroke image PS2, using advanced drawing processing that takes pen tip attributes into account.

[0113] Furthermore, the image processing method according to this embodiment also allows the video device 23 to limit the area in which the pen stroke image PS2 is synthesized to within the drawing area A3. Therefore, it becomes possible to restrict the drawing of the pen stroke image PS2 outside the drawing area A3.

[0114] The virtual machine 50 may be an application that runs on a web browser. Examples of such applications include drawing applications that run on a web browser and desktop virtualization applications used in screen-transfer type thin clients. The application may actually run on computer 3, or on a server computer connected to computer 3 via a network. In either case, by using the image processing method according to this embodiment, it becomes possible to draw a provisional image, the pen stroke image PS2, using advanced drawing processing that takes pen tip attributes into consideration.

[0115] Figure 14 shows the system configuration of computers 1 and 2 that execute the image processing method according to the fifth embodiment of the present invention. The image processing method according to this embodiment differs from the first embodiment in that the generation and rendering of the pen stroke image PS2 are performed inside the pen input device 25. The configuration and operation of computers 1 and 2 according to this embodiment will be described below, focusing on the differences from computers 1 and 2 according to the first embodiment.

[0116] The pen input device 25 according to this embodiment is a so-called liquid crystal tablet (tablet with an LCD screen), and as shown in Figure 14, it is configured to have a touch sensor 25a, a pen processing unit 25b, a drawing processing unit 25c, and a foremost display processing unit 25d. In this embodiment, the display 24 also constitutes part of the pen input device 25. However, the type of pen input device 25 is not limited to a liquid crystal tablet; for example, it may be a sign tablet with a small screen. In this case, the display 24 does not constitute part of the pen input device 25, and a small display separate from the display 24 is arranged inside the pen input device 25.

[0117] The touch sensor 25a is a device composed of multiple sensor electrodes arranged on the touch surface. The pen processing unit 25b is a functional unit composed of an integrated circuit and has the function of a sensor controller as described above. That is, the pen processing unit 25b uses the touch sensor 25, for example, by an active electrostatic method or an electromagnetic induction method, to detect two-dimensional coordinates indicating the position of the pen P, and also plays the role of receiving data such as pen pressure values ​​from the pen P. The pen processing unit 25b is connected to the tablet driver 32 via the USB device driver 31 and is also connected to the drawing processing unit 25c, and is configured to supply the detected position of the pen P and the data received from the pen P (pen data PD) to both the tablet driver 32 and the drawing processing unit 25c each time the position of the pen P is detected.

[0118] In this embodiment, the rendering engine 33 is configured to supply the drawing application window position information GAWP and application state information AS received from the rendering engine agent 34 to the tablet driver 32. The tablet driver 32 is configured to supply the drawing application window position information GAWP and application state information AS supplied from the rendering engine 33, along with the acquired virtual desktop information VDP, to the pen processing unit 25b via the USB device driver 31. The pen processing unit 25b is configured to supply the drawing application window position information GAWP, application state information AS, and virtual desktop information VDP thus supplied from the tablet driver 32 to the drawing processing unit 25c.

[0119] The drawing processing unit 25c is also a functional unit composed of an integrated circuit, and, like the rendering engine 33, has the function of generating a pen stroke image PS2 based on information supplied from the pen processing unit 25b. Specifically, the drawing processing unit 25c first obtains the position and size of the virtual desktop area VDI within the desktop video DV1 based on the virtual desktop information VDP, and then obtains the position and size of the window W within the virtual desktop area VDI based on the drawing application window position information GAWP. Then, it obtains the position, size, shape, and background color of the drawing area A3 within the window W based on the canvas area information in the application state information AS, and is configured to draw the pen stroke image PS2 inside the thus obtained drawing area A3. The drawing processing unit 25c is also configured to obtain brush data based on the drawing setting information in the application state information AS, and to generate the pen stroke image PS2 by performing rendering processing based on the pen data PD using the obtained brush data.

[0120] The foremost display processing unit 25d is also a functional unit composed of an integrated circuit and includes a memory for temporarily storing video signals to be displayed on the display 24. The foremost display processing unit 25d is configured to store the pen stroke image PS2 generated by the drawing processing unit 25c in this memory. The foremost display processing unit 25d also receives the desktop video DV1 from the video device 23 and outputs it to the display 24 along with the video signal stored in the memory. The display 24, upon receiving this output, displays the pen stroke image PS2 superimposed on the desktop video DV1. This allows the user to see an image on the display 24 in which the pen stroke image PS2 is superimposed on the desktop video DV1.

[0121] With the image processing method according to this embodiment, computer 1, like computer 1 according to the first embodiment, can draw a provisional image, the pen stroke image PS2, using advanced drawing processing that takes pen tip attributes into consideration.

[0122] As shown by the dashed line in Figure 14, the computer 1 according to this embodiment may also be configured to generate the pen stroke image PS2 using the rendering engine 33, similar to the computer 1 according to the first embodiment. In this case, the tablet driver 32 may determine whether the drawing processing unit 25c corresponds to the pen tip attributes indicated by the application state information AS supplied from the rendering engine 33, and if it determines that it does not correspond, it may switch the location of pen stroke image PS2 generation from the drawing processing unit 25c to the rendering engine 33. By adopting such a switching process, even when performing advanced drawing processes (such as brush shape and brush behavior) that the drawing processing unit 25c does not support, it becomes possible to display the pen stroke image PS2 with the same appearance as the pen stroke image PS1.

[0123] Furthermore, the drawing processing unit 25c may also perform a process to erase the pen stroke image PS2 from the image supplied to the display 24 by the foremost display processing unit 25d, using a method similar to one or both of the methods for erasing the pen stroke image PS2 described with reference to Figure 7 and the method for erasing the pen stroke image PS2 described with reference to Figure 12. In this way, it becomes possible to appropriately erase the portion of the pen stroke image PS2 generated by the drawing processing unit 25c that has become displayed as the pen stroke image PS1.

[0124] Furthermore, the pen processing unit 25b, the drawing processing unit 25c, and the foreground display processing unit 25d may be composed of a single integrated circuit or of two or more integrated circuits. In the latter case, when the pen processing unit 25b and the drawing processing unit 25c are composed of a single integrated circuit, it is preferable to use an MCU (Micro Controller Unit). Also, when the drawing processing unit 25c and the foreground display processing unit 25d are composed of a single integrated circuit, it is preferable to use a scaler IC. When multiple integrated circuits are used, it is common to connect each integrated circuit to each other via a peripheral device interface other than USB, such as I2C (Inter-Integrated Circuit) or UART (Universal Asynchronous Receiver / Transmitter).

[0125] Although preferred embodiments of the present invention have been described above, the present invention is not limited in any way to these embodiments, and it goes without saying that the present invention can be implemented in various forms without departing from its essence.

[0126] For example, the rendering engine 33 may detect the indicated position of the pen P when a pen-down occurs (i.e., when the pen pressure value changes from 0 to a value greater than 0), and perform the second rendering process only if that indicated position is within the drawing area A3. Alternatively, the rendering engine 33 may continue to perform the second rendering process without changing the position of the drawing area A3 (i.e., fixing the drawing area A3) during the period from when a pen-down occurs until a pen-up occurs (i.e., until the pen pressure value returns to 0), even if new drawing application window position information GAWP or application state information AS is received and indicates a change in the position of the drawing area A3. On the other hand, the rendering engine 33 may erase all of the currently displayed pen stroke images PS2 when new drawing application window position information GAWP or application state information AS is received and indicates a change in the position of the drawing area A3.

[0127] Furthermore, in the configuration shown in Figure 13, the tablet driver 32 and rendering engine 33 may be provided within the integrated circuit, while the tablet driver 53, rendering engine agent 54, and drawing application 55 may be provided within the processor 20. The pen stroke image PS2 may be generated within the integrated circuit until the pen stroke image PS1 generated by the drawing application 55 is reflected in the desktop video DV1. This makes it possible to apply the present invention to the computer 3 with the integrated circuit as the local device and the processor 20 as the remote device.

[0128] Furthermore, the gap between the tip of pen P and the tip of the pen stroke image PS1 displayed on the computer 1's display 24 becomes smaller as the display 24 is smaller, and larger as the display 24 is larger. Therefore, the rendering engine 33 may control the presence of the pen stroke image PS2 according to the size of the display 24. More specifically, the rendering engine 33 may reduce the presence of the pen stroke image PS2 as the display 24 is smaller. For example, the rendering engine 33 may turn off the generation of the pen stroke image PS2 when the size of the display 24 is less than or equal to a predetermined value. Also, the rendering engine 33 may shorten the time from the generation of the pen stroke image PS2 in step S25 shown in Figure 7 to the erasure of the pen stroke image PS2 in step S28 shown in Figure 7 as the display 24 is smaller. In addition, the rendering engine 33 may make the line width of the pen stroke image PS2 thinner as the display 24 is smaller, or increase the transparency of the pen stroke image PS2 (i.e., make the drawing color lighter) as the display 24 is smaller.

[0129] Furthermore, a settings screen for configuring the specific method of generating the pen stroke image PS2 (for example, drawing color, pattern, presence or absence of outline, etc.) may be displayed within the desktop video DV1, allowing the user of computer 1 to configure the method of generating the pen stroke image PS2. In this case, the rendering engine agent 34 or the drawing application 36 may draw the settings screen within the virtual desktop area VDI (or within window W). This makes it possible to change the method of generating the pen stroke image PS2 without minimizing or moving the virtual desktop area VDI when it is displayed in full screen on display 24.

[0130] Alternatively, the rendering engine 33 of computer 1 (drawing processing unit 25c in the example of Figure 14) may draw a settings screen outside the virtual desktop area VDI to set the specific method for generating the pen stroke image PS2. In this way, the user can set the appearance of the pen stroke image PS2 within the local device, so even if there is a delay in sending application status information AS from the rendering engine agent 34 to the rendering engine 33, it becomes possible to draw a provisional image of the pen stroke image PS2 using advanced drawing processing that takes into account the pen tip attributes until the application status information AS is received. Furthermore, one or more frequently used generation methods may be preset in the rendering engine 33, and the settings screen may be configured so that the method for generating the pen stroke image PS2 can be set by simply selecting one of the preset generation methods.

[0131] Furthermore, the rendering engine 33 of computer 1 may perform the operation to display the above-mentioned settings screen in response to changes in button information contained in the data supplied from the pen input device 25. For example, the settings screen should be displayed when the button information contained in the data supplied from the pen input device 25 indicates that the state of a predetermined switch on the pen P has changed from off to on. In this way, the user can display the above-mentioned settings screen on computer 1 by pressing the predetermined switch on the pen P. In this case, it is preferable that the tablet driver 32 fixes the value of the button information corresponding to the predetermined switch in the pen data PD transmitted to computer 2 to a value that always indicates off, regardless of the content of the data supplied from the pen input device 25, so as not to notify computer 2 that the state of the predetermined switch has changed to on. In this way, it is possible to avoid confusing the user by preventing two operation results (display of the above-mentioned settings screen on computer 1 and other operations performed on computer 2) from occurring with the operation of one switch.

[0132] Furthermore, the pen input device 25 may be provided with one or more switches (such as an on / off button or a touch ring), and information indicating the state of each of the one or more switches provided on the pen input device 25 may also be placed in the pen data PD. In addition, the rendering engine 33 of the computer 1 may perform the display operation of the settings screen in response to a change in the state of a predetermined switch provided on the pen input device 25. In this case, if information indicating the state of this predetermined switch is placed in the pen data PD, it is preferable that the tablet driver 32 of the computer 1 does not notify the computer 2 of the state of the predetermined switch by fixing the information indicating the state of the predetermined switch, regardless of the actual state of the predetermined switch, at least while the state of the predetermined switch is associated with the display operation of the settings screen.

[0133] Furthermore, the tablet driver 32 of computer 1 may store flag information indicating whether or not each piece of data constituting the pen data PD should be sent to computer 2, and the rendering 33 or conference application 35 may control the value of this flag information according to the information displayed on the display 24. The tablet driver 32 may also choose not to send data to computer 2 that is indicated by the flag information as not to be sent (or fix the value to be sent to a predetermined value). In this way, for example, if the indicated position of the pen P exists within the menu screen displayed by the rendering 33 or conference application 35, the coordinate data of the pen P can not be sent to computer 2, thereby preventing the pen stroke image PS1 from being drawn on computer 2 while the user of computer 1 is operating the menu with the pen P.

[0134] Furthermore, computers 1 and 2 may each have multiple displays 24. In this case, the combined images displayed on each of the multiple displays 24 of computer 2 may be used as the desktop video DV2. If the virtual desktop area VDI does not fit within the desktop video DV1, the virtual desktop area VDI may be reduced in size, or only a portion of the virtual desktop area VDI may be displayed within the desktop video DV1.

[0135] Furthermore, the pen input device 25 may be configured as a so-called liquid crystal tablet integrated with the display 24, or as a so-called pen tablet separate from the display 24. In the former case, if the computer 1 has multiple displays 24, each may be configured as a liquid crystal tablet. In the latter case, according to the processing of the rendering engine 33 described in the second embodiment, the rendering engine 33 will draw a pen stroke image PS2 so as to connect the cursor indicating the position of the pen P displayed on the computer 1's display 24 with the tip of the pen stroke image PS1 that appeared in the streaming video SV.

[0136] Furthermore, the pen input device 25 may be configured to detect multiple pens P simultaneously. In this case, it is preferable that the rendering engine 33 generates a pen stroke image PS2 for each pen P.

[0137] Furthermore, writing to the virtual desktop area (VDI) may be performed using a passive pointer such as a finger, either in conjunction with or instead of the pen P. In this case, it is preferable that the rendering engine 33 generates a provisional image for the stroke indicated by a series of indicated positions of the passive pointer, in the same manner as the pen stroke image PS2.

[0138] Furthermore, the meeting application 35 may be configured as a browser plugin, or as a separate application from the browser.

[0139] Furthermore, the present invention is not only applicable to remote meetings as described in the above embodiments, but is also broadly applicable to cases where the screen (desktop screen or application window) of one computer (remote device) is shared with another computer (local device). [Explanation of Symbols]

[0140] 1-3 Computer 20 processors 21 Communication equipment 22 Input devices 23,56 video devices 24 displays 25 Pen input devices 25a Touch Sensor 25b Pen Processing Unit 25c Drawing Processing Unit 25d Frontmost Display Processing Unit 30,31,52 Device Drivers 32,53 Tablet Driver 33 Rendering Engine 34,54 Rendering Engine Agents 35,45 Meeting applications 36,55 Drawing applications 40 display channels 41 Data Channels 42 Tablet Channels 43 Redirect Channels 50 virtual machines 51 Communications Department A1 Main Area A2 Campus Area A3 drawing area AS Application Status Information DV1, DV1a, DV2 Desktop Video GAWP drawing application window position information P Pen P1~P4 Pen P's indicated position PD Pen Data PS1, PS2 Pen Stroke Images SV Streaming Video VDI (Virtual Desktop Infrastructure) VDP Virtual Desktop Information W Drawing application 36 windows

Claims

1. An image processing method performed by a remote device and a local device, The local device detects the pen's indicated position and transmits it to the remote device via the network. The remote device transmits via the network a streaming video including a series of pen stroke images generated based on a series of indicated positions received from the local device. The local device generates a provisional image based on a series of indicated positions it has detected, which has a trajectory that matches the pen stroke image, but in which one or more of the pattern, drawing color, line width, or presence or absence of contour differs from the pen stroke image. The local device receives the streaming video from the remote device via the network, synthesizes the provisional image, and outputs it. The local device sequentially erases portions of the provisional image that meet predetermined conditions, Image processing methods including [specific details omitted].

2. The portion where the aforementioned predetermined conditions are met is the portion after a predetermined time has elapsed since generation. The image processing method according to claim 1.

3. The predetermined time is set by the remote device. The image processing method according to claim 2.

4. The predetermined time is set according to the length of time until the pen stroke image is reflected in the streaming video. The image processing method according to claim 2 or 3.

5. The portion that satisfies the aforementioned predetermined conditions is the portion that appears in the streaming video. The image processing method according to any one of claims 1 to 4.

6. The remote device is configured to transmit to the local device information indicating that rendering processing has started for generating a pen stroke image included in the streaming video, from among the pen instruction positions sequentially received from the local device. The portion where the predetermined conditions are met corresponds to the indicated position where the rendering process is initiated by information from the remote device. The image processing method according to any one of claims 1 to 5.

7. The remote device and the local device are separate computers. The image processing method according to any one of claims 1 to 6.

8. The local device includes a pen input device. The step of generating the provisional image is performed by an integrated circuit in the pen input device. The image processing method according to any one of claims 1 to 7.

9. The aforementioned pen input device includes a display device, The pen input device performs a step of outputting the provisional image to the display device, separate from the step of synthesizing and outputting the provisional image. The image processing method according to claim 8.

10. A pen input device that detects the position of the pen, A display device that displays images, A communication device that communicates with other information processing devices via a network, The pen input device, the display device, and the communication device each include a control unit connected to them, The control unit, The series of indicated positions of the pen detected by the pen input device are transmitted to the other information processing device via the communication device. The other information processing device receives a streaming video from the network containing pen stroke images generated based on the series of indicated positions that were transmitted. Based on the series of indicated positions, a provisional image is generated that has a trajectory matching the pen stroke image, but differs from the pen stroke image in one or more of the following: pattern, drawing color including transparency, line width, and presence or absence of contours. The video obtained by combining the provisional image with the streaming video received from the other information processing device via the communication device is displayed on the display device. The parts of the displayed provisional image that meet the predetermined conditions are sequentially deleted. Information processing device.

11. The portion where the aforementioned predetermined conditions are met is the portion after a predetermined time has elapsed since generation. The information processing apparatus according to claim 10.

12. The predetermined time is set by the other information processing device. The information processing apparatus according to claim 11.

13. The predetermined time is set according to the length of time until the pen stroke image is reflected in the streaming video. The information processing apparatus according to claim 11 or 12.

14. The portion that satisfies the aforementioned predetermined conditions is the portion that appears in the streaming video. The information processing apparatus according to any one of claims 10 to 13.

15. The other information processing device is configured to transmit to the information processing device information indicating that rendering processing for generating a pen stroke image included in the streaming video has started from among the pen instruction positions sequentially received from the information processing device. The portion of the provisional image that satisfies the predetermined conditions is the portion corresponding to the indicated position where the start of the rendering process is indicated by information from the other information processing device. The information processing apparatus according to any one of claims 10 to 14.

Images