Image drawing method, drawing parameter extraction method, and program

Abstract

[Problem] To align the appearance of a provisional image with the appearance of a result image irrespective of the type of a drawing application used in a remote-side device. [Solution] An image drawing method executed by a local-side device configured to be capable of communicating with a remote-side device that generates streaming video, the method including: a step for rendering a provisional image to be combined with the streaming video on the basis of the position of an indicator detected by an indicator detection device; a step for extracting, from the streaming video, a drawing parameter used in rendering of a result image that is the result of rendering executed by the remote-side device on the basis of the position of the indicator, the result image being included within the streaming video; and a step for updating a drawing parameter used in rendering of the provisional image on the basis of the drawing parameter extracted through the step for extracting the drawing parameter.

Description

Image drawing method, drawing parameter extraction method, and program

[0001] The present invention relates to an image drawing method, a drawing parameter extraction method, and a program, and more particularly to an image drawing method and program for performing pen input on a screen shared between computers, and a drawing parameter extraction method and program for extracting, from a pen stroke image that is the rendering result of the pen stroke, the drawing parameters used in rendering the pen stroke.

[0002] In remote conferences where multiple computers are interconnected via communication lines, it is common to "share" a screen that is open on one computer so that it can be viewed on other computers. In this case, a computer that shares the screen (hereinafter referred to as a "remote device") transmits streaming video of the shared screen to the other computers (hereinafter referred to as "local device").

[0003] Patent Literature 1 discloses a technique for performing pen input from a local device to the above-mentioned streaming video. When performing such pen input, the local device typically performs only the process of sequentially transmitting the pen position detected by a sensor controller to a remote device, while rendering of the pen strokes and compositing them into the streaming video are performed by the remote device. However, this process causes a significant delay before the pen strokes are reflected in the streaming video. Therefore, in the technique described in Patent Literature 1, pen strokes are temporarily drawn on the local device until they appear in the streaming video. Hereinafter, the image of the pen strokes drawn by this temporary drawing process will be referred to as a "provisional image."

[0004] Patent No. 7200446

[0005] The aforementioned Patent Document 1 discloses a technique for aligning the appearance of a provisional image with the appearance of a pen-stroke image (hereinafter referred to as a "result image") that appears in a streaming video by sharing rendering processing between a remote device and a local device. However, in order to align the appearance of the provisional image with the appearance of the result image using this technique, the drawing application used on the remote device must support sharing of the rendering process. Therefore, there has been a need for a technique that enables the appearance of the provisional image to be aligned with the appearance of the result image regardless of the type of drawing application used on the remote device.

[0006] Therefore, one object of the present invention is to provide an image drawing method and program that enables the appearance of a temporary image to be aligned with the appearance of a final image regardless of the type of drawing application used on the remote device.

[0007] One way to achieve this goal is to extract from streaming video the drawing parameters used in rendering the resulting image, but in the past, it was difficult to extract this accurately.

[0008] Therefore, another object of the present invention is to provide a drawing parameter extraction method and program that can extract, with high accuracy, drawing parameters used in rendering from a pen stroke image that is the rendering result of the pen stroke.

[0009] An image rendering method according to a first aspect of the present invention is an image rendering method executed by a local-side device configured to be able to communicate with a remote-side device that generates streaming video, and includes the steps of: rendering a tentative image to be composited with the streaming video based on the position of a pointer detected by a pointer detection device; extracting from the streaming video rendering parameters used in rendering the resulting image, which is a result of the rendering executed by the remote-side device based on the position of the pointer and is included in the streaming video; and updating the rendering parameters used in rendering the tentative image based on the drawing parameters extracted in the step of extracting the drawing parameters.

[0010] An image rendering method according to a second aspect of the present invention is an image rendering method executed by a local-side device configured to be able to communicate with a remote-side device that generates streaming video, and includes the steps of: rendering a tentative image to be composited with the streaming video based on the position of a pointer detected by a pointer detection device; extracting from the streaming video rendering parameters used in rendering the resulting image, which is a result of the rendering executed by the remote-side device based on the position of the pointer and is included in the streaming video; and, if the current rendering parameters used in rendering the tentative image differ from the drawing parameters extracted in the extracting step, notifying a user that the rendering parameters used in rendering the tentative image do not match the drawing parameters used in rendering the resulting image.

[0011] A drawing parameter extraction method according to a third aspect of the present invention is a drawing parameter extraction method including the steps of: estimating a pen movement path from a pen stroke image; setting a plurality of reference areas along the estimated movement path; and extracting drawing parameters used in rendering the pen stroke image based on portions of the pen stroke image that appear within each of the plurality of reference areas.

[0012] A program according to a first aspect of the present invention is a program for causing a computer to function as a local-side device configured to be able to communicate with a remote-side device that generates streaming video, and causes the computer to execute the following steps: rendering a tentative image to be composited into the streaming video based on the position of a pointer detected by a pointer detection device; extracting from the streaming video drawing parameters used in rendering the resulting image, which is a result of the rendering performed by the remote-side device based on the position of the pointer and is included in the streaming video; and updating the drawing parameters used in rendering the tentative image based on the drawing parameters extracted in the drawing parameter extraction step.

[0013] A program according to a second aspect of the present invention is a program for causing a computer to function as a local-side device configured to be able to communicate with a remote-side device that generates streaming video, and causes the computer to execute the following steps: rendering a tentative image to be composited with the streaming video based on the position of a pointer detected by a pointer detection device; extracting from the streaming video drawing parameters used in rendering the resulting image, which is a result of the rendering performed by the remote-side device based on the position of the pointer and is included in the streaming video; and, if the current drawing parameters used in rendering the tentative image differ from the drawing parameters extracted in the extracting step, notifying a user that the drawing parameters used in rendering the tentative image do not match the drawing parameters used in rendering the resulting image.

[0014] A third aspect of the present invention provides a program for causing a computer to execute the steps of: estimating a pen movement path from a pen stroke image; setting a plurality of reference areas along the estimated movement path; and extracting drawing parameters used in rendering the pen stroke image based on portions of the pen stroke image that appear within each of the plurality of reference areas.

[0015] According to the first and second aspects of the present invention, drawing parameters can be extracted from streaming video and the drawing parameters used to render a provisional image can be updated based on the results, making it possible to align the appearance of the provisional image with the appearance of the resulting image regardless of the type of drawing application used on the remote device.

[0016] According to the third aspect of the present invention, drawing parameters are extracted based on the images that appear within each of multiple reference areas set along the movement path, making it possible to extract the drawing parameters used in rendering the pen stroke image with higher accuracy than when drawing parameters are extracted from only one location.

[0017] 2 is a diagram showing the system configuration of an image sharing system 1 according to an embodiment of the present invention. It is a diagram showing the relationship between a pen stroke image S, which is a result image generated by a drawing application 22, and a reference area A set by a reference area setting unit 41. ２ 2 is a diagram illustrating a process executed by the processing unit 14 in response to the addition of the designated position P ３ 2 is a diagram illustrating a process executed by the processing unit 14 in response to the addition of the designated position P ４ 2 is a diagram illustrating a process executed by the processing unit 14 in response to the addition of the designated position P ５ 2 is a diagram illustrating a process executed by the processing unit 14 in response to the addition of the designated position P ６ 2 is a diagram illustrating a process executed by the processing unit 14 in response to the addition of the designated position P ７ 2 is a diagram illustrating a process executed by the processing unit 14 in response to the addition of the designated position P８ 2 is a diagram illustrating a process executed by the processing unit 14 in response to the addition of the designated position P ９ 1 is a diagram illustrating processing executed by the processing unit 14 in response to the addition of a reference area A. FIG. 1 is a process flow diagram illustrating provisional image drawing processing executed by the processing unit 14 according to an embodiment of the present invention. FIG. 2 is a process flow diagram illustrating provisional image drawing parameter update processing executed by the processing unit 14 according to an embodiment of the present invention. FIG. 3 is a process flow diagram illustrating provisional image drawing parameter update processing executed by the processing unit 14 according to an embodiment of the present invention. FIG. 4 is a system configuration of an image sharing system 1 according to a first modified example of an embodiment of the present invention. FIG. 5 is a system configuration of an image sharing system 1 according to a second modified example of an embodiment of the present invention. FIG. 6 is a system configuration of an image sharing system 1 according to a third modified example of an embodiment of the present invention. FIG. 7 is a process flow diagram illustrating provisional image drawing parameter update processing executed by the processing unit 14 according to a fourth modified example of an embodiment of the present invention. FIG. 8 is a diagram illustrating the relationship between a pen stroke image S, which is a resultant image generated by a drawing application 22, and a reference area A set by a reference area setting unit 41.

[0018] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0019] FIG. 1 is a diagram showing the system configuration of an image sharing system 1 according to an embodiment of the present invention. As shown in the diagram, the image sharing system 1 according to this embodiment is configured to include computers 10 and 20 and a pen display 30. The computers 10 and 20 may be any information processing devices, such as a personal computer, a tablet terminal, or a smartphone, and the specific type of the computers is not limited. FIG. 1 shows an example in which the computer 10 is a desktop computer configured separately from the pen display 30, and the description of this embodiment will be continued based on this example.

[0020] Each of the computers 10 and 20 is configured to operate by having a central processing unit read and execute programs stored in memory. In this embodiment, the programs stored in the memory of the computers 10 and 20 include an operating system as well as various applications, including a drawing application and a conferencing application.

[0021] The computers 10 and 20 hold a remote conference with each other through a conference application. The type of conference application is not particularly limited, but in this embodiment, a conference application having a function of sharing the screen of the computer 20 (desktop screen or application window) with the computer 10 is used.

[0022] The pen display 30 is a display device that supports pen input on a display surface. The pen display 30 and the computer 10 are connected to each other, and the pen display 30 displays images supplied from the computer 10 on the display surface. The pen display 30 also has a function of detecting the position (pointing position) of a pen (pointing object) on the display surface and supplying the detected pointing position to the computer 10.

[0023] In this embodiment, the computer 10 and the pen display 30 are local devices, and the computer 20 is a remote device. When a user of the computer 20, which is a remote device, performs a predetermined operation to share the screen of the computer 20 (desktop screen or application window) in a conference application, the screen of the shared computer 20 is displayed on the desktop screen of the computer 10, which is a local device. This allows the user of the computer 10 to view the screen of the computer 20 on the display surface of the pen display 30.

[0024] 1 also shows the functional units of the computers 10 and 20 and the pen display 30. As shown in the figure, the computer 10 is functionally configured to have a driver 11, a conference application 12, an image synthesis unit 13, and a processing unit 14, the computer 20 is functionally configured to have a driver 21, a drawing application 22, and a conference application 23, and the pen display 30 is functionally configured to have a display 31 and a position detection device 32.

[0025] The display 31 is a display device that displays on the display surface of the pen display 30 in accordance with a video signal supplied from the computer 10. The specific type of the display 31 is not limited, but may be, for example, a liquid crystal display or an organic EL display.

[0026] The position detection device 32 accepts pen input on the display surface of the pen display 30 and includes a touch sensor disposed inside the display surface and a sensor controller connected to the touch sensor. The sensor controller is an integrated circuit that uses the touch sensor to detect the position of the pen on the display surface and receives data from the pen, such as a writing pressure value indicating the pressure applied to the pen tip and a pen ID, which is the pen's identification information. The method for detecting the pen position using the sensor controller is not particularly limited, but an active electrostatic method or an electromagnetic induction method is preferred. Regardless of the method used, the sensor controller derives the intensity distribution of the signal received from the pen on the display surface, detects the position of the peak as the pen position, and demodulates the signal received from the pen to obtain data such as the writing pressure value and the pen ID. The sensor controller may also obtain the height of the pen (the distance from the display surface to the pen tip) based on the peak intensity of the signal received from the pen. If the pen has multiple pen tip electrodes, the sensor controller may detect the position of each pen tip electrode and derive the tilt and rotation angle of the pen based on the detected positions. The height, tilt, and rotation angle of the pen acquired by the sensor controller in this way, together with data such as the writing pressure value and pen ID obtained by demodulating the received signal, constitute a series of data acquired by the sensor controller. The position detection device 32 is configured to sequentially supply the position and data acquired by the sensor controller to the driver 11 of the computer 10.

[0027] The drivers 11 and 21 are functional units that control communication between various functional units implemented by the central processing units of the computers 10 and 20 and the position detection device 32. The driver 11 according to this embodiment plays a role in generating pen data including the position and data supplied to the computer 10 by the position detection device 32, and sequentially supplying the data to the processing unit 14 and the computer 20. The driver 21 also plays a role in receiving the pen data from the driver 11 and sequentially supplying the data to the drawing application 22.

[0028] The drawing application 22 is an application that generates a pen stroke image (result image) by performing rendering processing based on the pen data supplied from the driver 21. The drawing application 22 may be any application that can perform rendering processing based on pen data, and may be a typical drawing application such as BAMBOO PAPER (registered trademark), as well as a word processing application such as Microsoft Word or a presentation application such as Microsoft PowerPoint. The specific method of rendering processing is not limited, but it is preferable to use a spline curve such as a Bézier curve or a Catmull-Rom curve. The pen stroke image generated by the drawing application 22 is displayed in the window of the drawing application 22.

[0029] In the rendering process by the drawing application 22, various pen nib attributes may be used depending on the user's settings. Specific examples of pen nib attributes include drawing color, line width, brush shape (pencil, brush, scatter effect, etc.), transparency, drawing frequency used for brushes that perform discrete drawing, pseudo-random seed values ​​used for brushes that randomly change size, color, etc., pen height, pen tilt, pen rotation angle, etc. The drawing application 22 selects one or more of these pen nib attributes depending on the user's settings, and performs rendering to generate a pen stroke image according to the selected one or more pen nib attributes.

[0030] Of the pen tip attributes listed above, the pen height, pen tilt, and pen rotation angle are data acquired by the sensor controller as described above. The drawing application 22 extracts these pen tip attributes from the pen data supplied from the driver 21 and uses them in rendering processing. Meanwhile, the other pen tip attributes are data set by the user on the setting screen of the drawing application 22. The setting screen of the drawing application 22 may be configured so that pen tip attributes such as drawing color can be selected for each of the above-mentioned pen IDs. In this case, the drawing application 22 selects pen tip attributes according to the pen ID included in the pen data supplied from the driver 21, and performs rendering processing using the selected pen tip attributes.

[0031] The conference applications 12 and 23 are applications for realizing the above-described remote conference. The conference applications 12 and 23 communicate with each other in response to user operations on the computers 10 and 20, respectively, to realize the remote conference. In this remote conference, the conference application 23 transmits streaming video including a window of the drawing application 22 to the conference application 12 in response to user operations on the computer 20. The conference application 12 receives this streaming video and sequentially outputs the received video to the image composition unit 13.

[0032] The image composition unit 13 is a device that generates a video signal representing a desktop image to be displayed on the desktop of the computer 10 and supplies the signal to the display 31. When the above-mentioned streaming image is supplied from the conference application 12, the desktop image displayed by the video signal generated by the image composition unit 13 includes this streaming image.

[0033] The processing unit 14 is a functional unit that draws a temporary pen stroke image (provisional image) in the desktop image until the pen stroke image (result image) generated based on the pen data sent by the driver 11 to the computer 20 appears in the streaming image supplied from the conference application 12 to the image synthesis unit 13.

[0034] The processing unit 14 also generates an image indicating a cursor when a predetermined condition is met, and draws the image at a position indicated by the pen data. The predetermined condition may be, for example, that the pen pressure value included in the pen data supplied from the driver 11 indicates a pen touch (a state in which the pen is in contact with the display surface of the pen display 30; specifically, that the pen pressure value is greater than 0), or that the pen height indicated by the pen data supplied from the driver 11 is within a predetermined value (for example, 12 mm).

[0035] 1 shows functional units related to drawing of a provisional image among the functional units of the processing unit 14. As shown in the figure, the processing unit 14 includes, as functional units related to drawing of a provisional image, a provisional image drawing unit 40, a reference area setting unit 41, a drawing parameter extraction unit 42, and a drawing parameter setting unit 43. Each of these units will be described in detail below.

[0036] The provisional image drawing unit 40 is a functional unit that generates a pen stroke image, which is a provisional image, by performing rendering processing based on the pen data supplied from the driver 11 to the processing unit 14. The specific method of rendering is preferably the same as that of the drawing application 22, but does not necessarily have to be the same. The provisional image drawing unit 40 sequentially supplies the generated pen stroke images to the image synthesis unit 13, and the image synthesis unit 13 synthesizes the pen stroke images supplied from the provisional image drawing unit 40 into the streaming video included in the desktop video. As a result, the streaming video displayed on the display 31 includes the pen stroke image, which is a provisional image.

[0037] The provisional image drawing unit 40 also performs processing to erase a portion of the provisional image under control of the drawing parameter extraction unit 42, which will be described later. Details will be explained in conjunction with the explanation of the drawing parameter extraction unit 42, but in essence, this processing is processing to erase, when a resultant image corresponding to part or all of the pen stroke image supplied to the image synthesis unit 13 as the provisional image appears in the streaming video, the corresponding portion of the provisional image from the provisional image to be supplied to the image synthesis unit 13. This makes it possible to limit the display period of each portion of the provisional image to the time until the corresponding resultant image appears in the streaming video.

[0038] The reference area setting unit 41 is a functional unit that acquires the provisional image generated by the provisional image drawing unit 40 from the image synthesis unit 13 and sets a reference area that serves as a unit for image comparison (described later) by the drawing parameter extraction unit 42 based on the acquired provisional image. The reference area setting unit 41 is configured to set a new reference area as the provisional image is extended. The reference area setting unit 41 also acquires screenshots of the streaming video within the set reference area and supplies them to the drawing parameter extraction unit 42.

[0039] FIG. 2 illustrates the relationship between the pen stroke image S generated by the drawing application 22 and the reference area A set by the reference area setting unit 41. The X-axis and Y-axis shown in FIG. 2 and in FIGS. 3 to 10 (discussed below) represent mutually orthogonal directions within the display surface of the pen display 30. FIG. 2 also illustrates the designated position P, which indicates the pen position detected by the position detection device 32. The subscripts to the right of the symbols for the designated position P and the reference area A indicate their respective order of occurrence. While a provisional image is not shown in FIG. 2, the provisional image is generally drawn at a position overlapping the pen stroke image S. However, if the provisional image drawing unit 40 and the drawing application 22 use different rendering methods, or if the drawing application 22 performs advanced processing (such as image stabilization) that the provisional image drawing unit 40 does not support, a misalignment may occur between the provisional image and the pen stroke image S.

[0040] As can be seen from Fig. 2, the reference area setting unit 41 sets a reference area A along the provisional image at predetermined time intervals. Since the pen movement speed is usually not constant, the intervals between the reference areas A are also not constant. Fig. 2 shows an example in which the reference area A is set at the exact midpoint between a new designated position P and the previous designated position P each time a new designated position P is generated, but the reference area setting unit 41 may set the reference area A at a timing unrelated to the generation of the designated position P, or may set the reference area A at a location other than the midpoint between two adjacent designated positions P.

[0041] The shape of the reference area A is preferably a rectangle having one side extending in the X-axis direction and one side extending in the Y-axis direction, as shown in Fig. 2. This facilitates image comparison (described later) by the drawing parameter extraction unit 42. The size of the reference area A is preferably large enough to include the entire width of the pen stroke image S. The reference area setting unit 41 may change the size of the reference area A to be set subsequently, depending on the line width extracted by the drawing parameter extraction unit 42 as described later.

[0042] Returning to FIG. 1 , regarding the acquisition of screenshots of the streaming video within the set reference area, the reference area setting unit 41 is configured to acquire a screenshot of the inside portion of the newly set reference area from the streaming video acquired by the image composition unit 13 (before the provisional image is composited with the streaming video output from the conference application 12) each time a new reference area is set, and store the screenshot in memory as a "first image" for the reference area. Furthermore, for each set reference area, the reference area setting unit 41 is configured to acquire a screenshot of the inside portion of the streaming video acquired by the image composition unit 13 (before the provisional image is composited with the streaming video output from the conference application 12) at predetermined time intervals, and store the screenshot in memory as a "second image" for the reference area. The second image is a screenshot of the streaming video at a time after the first image.

[0043] The drawing parameter extraction unit 42 is a functional unit that extracts, from the streaming video, drawing parameters used in rendering a resultant image included in the streaming video. Specifically, the drawing parameters are extracted by comparing the first image and the second image acquired by the reference area setting unit 41. This extraction process will be described in detail below.

[0044] Each time the reference area setting unit 41 stores a new second image in memory, the drawing parameter extraction unit 42 compares the newly stored second image with the corresponding first image for any previously set reference area for which no change has yet been detected, and determines whether a change has occurred by calculating the difference. If the drawing parameter extraction unit 42 determines that a change has occurred, it detects the changed portion (difference) and extracts the drawing parameters used to render the detected portion, thereby extracting the drawing parameters used to render the resulting image. The drawing parameters to be extracted are typically the drawing color and line width, but may also be one or a combination of one or more of the above-listed pen tip attributes. That is, the drawing parameter extraction unit 42 may also be one or a combination of one or more of the drawing color, line width, brush shape (pencil, brush, scatter effect, etc.), transparency, drawing frequency, pseudo-random number seed value, pen height, pen tilt, and pen rotation angle. The drawing parameter extraction unit 42 supplies the extracted drawing parameters to the drawing parameter setting unit 43.

[0045] The drawing parameter extraction unit 42 also performs a process of sequentially erasing portions of the provisional image where the corresponding resultant image appears in the streaming video. Specifically, when the reference area setting unit 41 determines that a change has occurred in the above-mentioned determination, it controls the provisional image drawing unit 40 to erase the corresponding portion of the provisional image. This makes it possible to limit the display period of each portion of the provisional image to the time until the corresponding resultant image appears in the streaming video, as described above.

[0046] The drawing parameter setting unit 43 is a functional unit that sets drawing parameters (drawing color, line width, brush shape, transparency, etc.) used in rendering the provisional image in the provisional image drawing unit 40. The drawing parameter setting unit 43 sets default drawing parameters in the provisional image drawing unit 40 at a stage before the provisional image drawing unit 40 starts rendering the provisional image. When drawing parameters are supplied from the drawing parameter extraction unit 42, the drawing parameter setting unit 43 updates the drawing parameters set in the provisional image drawing unit 40 based on the supplied drawing parameters. After this update, the appearance of the provisional image is determined by the updated drawing parameters.

[0047] The drawing parameters set by the drawing parameter setting unit 43 to the provisional image drawing unit 40 through updating may be identical to the drawing parameters extracted by the drawing parameter extraction unit 42. However, if the difference between the extracted drawing parameters and the provisional image is large (e.g., if the drawing color of the provisional image is black and the extracted drawing parameters are red), the appearance of the provisional image may suddenly change, potentially surprising the user. Furthermore, errors in the extraction of the drawing parameters may cause the drawing parameters to fluctuate. Therefore, it is preferable that the provisional image drawing unit 40 determine the drawing parameters to be set in the provisional image drawing unit 40 based not only on the drawing parameters extracted by the drawing parameter extraction unit 42 but also on the current drawing parameters (set in the provisional image drawing unit 40) used for rendering the provisional image. For example, the provisional image drawing unit 40 may determine the drawing parameters to be set in the provisional image drawing unit 40 as an intermediate color between the current drawing color used for rendering the provisional image and the drawing color extracted by the drawing parameter extraction unit 42. This prevents the user from being surprised by a sudden change in the appearance of the provisional image, and prevents drawing parameters from fluctuating due to extraction errors.

[0048] Furthermore, if the difference between the drawing parameters extracted by the drawing parameter extraction unit 42 and the provisional image is very small (for example, if the difference is only in the shade of the drawing color and the shade difference is equal to or less than a predetermined value), the provisional image drawing unit 40 may not update the drawing parameters set in the provisional image drawing unit 40. This makes it possible to avoid giving the user an unnecessary sense of discomfort. That is, some users do not mind obvious changes, but are extremely bothered by subtle changes. As described above, by not updating the drawing parameters when the difference between the extracted drawing parameters and the provisional image is very small, it is possible to avoid giving such users an unnecessary sense of discomfort.

[0049] 3 to 10 show the indication position P ２ ~P ９ 2. Each figure (a) is a diagram illustrating the processing executed by the processing unit 14 in response to the addition of the designated position P ２ ~P ９ The provisional image S generated in response to the addition of Ｔ is the result image S Ｐ The resulting image S is displayed on the display 31. Ｐ The line width of the provisional image S Ｔ The resulting image S Ｐ and provisional image S Ｔ Each image has a contour line drawn on it, but these are provisional images S Ｔ and result image S Ｐ In this case, the result image S Ｐ Line width and provisional image S Ｔ The description will be given assuming that the line widths are the same and there are no contours.

[0050] Each figure (b) shows the provisional image S for each set reference area A. Ｔ Drawing color and result image S Ｐ1 is a table showing the drawing colors of the current drawing and the updated drawing colors determined by the drawing parameter setting unit 43. The different hatching patterns shown within the circles represent the different drawing colors, and the numbers 1 to 5 shown to the right of the circles represent the density of the drawing colors. Note that these numbers are merely sequential numbers assigned in ascending order from light colors to indicate the differences in the drawing colors so that the operation of the invention can be understood even from black and white drawings, and do not represent any physical quantities. Hereinafter, drawing colors will be represented by these numbers.

[0051] Referring first to FIG. 3, the position detector 32 detects a new pointing position P ２ In response to the addition of the designated position P １ , P ２ A provisional image S based on Ｔ is generated and displayed on the display 31. The reference area setting unit 41 also generates the provisional image S Ｔ along the reference area A １ At this time, the provisional image S Ｔ The drawing color of the result image S is set to 5 (default drawing color). Ｐ is not yet displayed, and the drawing parameter setting unit 43 has not yet determined a drawing color for updating.

[0052] 4, the position detector 32 detects a new indicated position P ３ In response to the addition of the designated position P １ ~P ３ A provisional image S based on Ｔ is generated and displayed on the display 31. The reference area setting unit 41 also generates the provisional image S Ｔ along the reference area A ２ At this time, the provisional image S Ｔ The drawing color of is 5. Result image S Ｐ is not yet displayed, and the drawing parameter setting unit 43 has not yet determined a drawing color for updating.

[0053] Next, referring to FIG. 5, the position detector 32 detects a new indicated position P ４ In response to the addition of the designated position P １ ~P４ A provisional image S based on Ｔ is generated and displayed on the display 31. The reference area setting unit 41 also generates the provisional image S Ｔ along the reference area A ３ At this time, the provisional image S Ｔ The drawing color of is 5. At the time of FIG. １ , P ２ Result image S based on Ｐ appears in the streaming video, and the reference area A １ Inside is a provisional image S Ｔ and result image S Ｐ Both are displayed.

[0054] Reference area A １ Result image S Ｐ After the display, the reference area A １ The second image corresponding to the result image S Ｐ The reference area A is １ The first image (obtained by the reference area setting unit 41 at the stage shown in FIG. 3) corresponds to the result image S Ｐ Since the drawing parameter extraction unit 42 does not include the first image, the drawing parameter extraction unit 42 extracts the resultant image S Ｐ Then, the resulting image S Ｐ is No. 1, the drawing parameter extraction unit 42 acquires No. 1 as the extracted drawing color and supplies it to the drawing parameter setting unit 43. The drawing parameter setting unit 43 then sets the resultant image S Ｐ The drawing parameters to be set in the provisional image drawing unit 40 are determined by updating based on the drawing color No. 1 of the provisional image S and the current drawing color No. 5 used in rendering the provisional image. Here, as shown in FIG. 5B, in order to avoid a sudden change in the appearance of the provisional image, the provisional image drawing unit 40 selects No. 4, which is closest to No. 5, from among Nos. 2 to 4 that can be selected as intermediate colors between No. 5 and No. 1. Thereafter, the provisional image drawing unit 40 uses the drawing color No. 4 to draw the provisional image S. Ｔ (See Figure 6).

[0055] Also, reference area A １ Since it is detected that there is a change from the first image to the second image in the provisional image S Ｔ Of the reference area A １ The provisional image drawing unit 40 then controls the provisional image drawing unit 40 to erase the portion corresponding to the next designated position P ５ A provisional image S to be executed when Ｔ In the rendering of １ ~P ５ After performing the rendering process based on the reference area A １ The part corresponding to the designated position P １ , P ２ By cutting out the part connecting the Ｔ The reason for partially erasing the provisional image by rendering the entire provisional image and then cutting out the erased portion is to make the shape of the provisional image closer to the shape of the resulting image by maintaining the state in which the provisional image is rendered using the same number of designated positions as the drawing application 22. However, the number of designated positions constituting the provisional image (in this case, designated positions P １ ) may be removed or not considered in rendering, thereby achieving partial erasure of the temporary image.

[0056] 6, the position detector 32 detects a new indicated position P ５ In response to the addition of the designated position P １ ~P ５ A provisional image S based on Ｔ The part of the provisional image S that has not yet been erased is displayed on the display 31. The reference area setting unit 41 also generates the provisional image S Ｔ along the reference area A ４ is newly set. At this time, the provisional image S Ｔ The drawing color of is 4. At the time of FIG. １ ~P ３ Result image S based on Ｐ appears in the streaming video, and the reference area A ２ Inside is a provisional image SＴ and result image S Ｐ Both are displayed.

[0057] Reference area A ２ Result image S Ｐ After the display, the reference area A ２ The second image corresponding to the result image S Ｐ The reference area A is ２ The first image (obtained by the reference area setting unit 41 at the stage of FIG. 4) corresponds to the result image S Ｐ Since the drawing parameter extraction unit 42 does not include the first image, the drawing parameter extraction unit 42 extracts the resultant image S Ｐ Then, the resulting image S Ｐ 1 remains unchanged and is number 1, the drawing parameter extraction unit 42 acquires number 1 as the extracted drawing color and supplies it to the drawing parameter setting unit 43. The drawing parameter setting unit 43 then calculates the number 1 of the resultant image S Ｐ The drawing parameters to be set in the provisional image drawing unit 40 are determined by updating based on the drawing color No. 1 of the provisional image S and the current drawing color No. 4 used in rendering the provisional image. Here, as shown in FIG. 6B, in order to avoid a sudden change in the appearance of the provisional image, the provisional image drawing unit 40 selects No. 3, which is the closest to No. 4, from among Nos. 2 to 3 that can be selected as intermediate colors between No. 4 and No. 1. Thereafter, the provisional image drawing unit 40 uses the drawing color No. 3 to draw the provisional image S. Ｔ (See Figure 7).

[0058] Also, reference area A ２ Since it is detected that there is a change from the first image to the second image in the provisional image S Ｔ Of the reference area A ２ The provisional image drawing unit 40 then controls the provisional image drawing unit 40 to erase the portion corresponding to the next designated position P ６ A provisional image S to be executed when Ｔ In the rendering of １ ~P ６After performing the rendering process based on the reference area A １ , A ２ The part corresponding to the designated position P １ ~P ３ By cutting out the part connecting the Ｔ Partial erasure is performed.

[0059] 7 and onward, the same applies, so below we will only explain the difference in drawing color. In FIG. 7, No. 2 is selected as the drawing color for updating, and in FIG. 8, No. 1 is selected as the drawing color for updating. The fact that the drawing color for updating is No. 1 means that the provisional image S Ｔ The drawing color of the result image S Ｐ 9 and 10, the drawing parameter setting unit 43 does not update the drawing parameters.

[0060] Fig. 11 is a process flow diagram showing the provisional image drawing process executed by the processing unit 14 according to this embodiment. Fig. 12 and Fig. 13 are process flow diagrams showing the provisional image drawing parameter update process executed by the processing unit 14 according to this embodiment. The process executed by the processing unit 14 according to this embodiment will be described in more detail below with reference to these figures.

[0061] 11, the processing unit 14 first determines whether or not a stroke input has started based on the pen data supplied from the driver 11 (step S1). The result of this determination is positive when the driver 11 shown in FIG. 1 inputs pen data including the initial designated position of a stroke or information indicating a pen touch (information indicating that the pen pressure value has become greater than 0) to the processing unit 14. The processing unit 14 repeatedly executes step S1 until the determination result of step S1 becomes positive.

[0062] After determining in step S1 that stroke input has started, the processing unit 14 determines whether stroke input has ended (step S2). The result of this determination is positive if a predetermined time has elapsed since the last designated position was input, or if pen data including information indicating pen-up (information indicating that the pen pressure value has become 0) is input from the driver 11 to the processing unit 14. If the determination result in step S2 is positive, the processing unit 14 returns to step S1, and if the determination result is negative, the processing unit 14 proceeds to step S3.

[0063] In step S3, the processing unit 14 determines whether or not a designated position has been input (step S3). The result of this determination is positive if pen data including a designated stroke position has been input from the driver 11 shown in Fig. 1 to the processing unit 14. If the determination result of step S3 is negative, the processing unit 14 returns to step S2, and if the determination result is positive, the processing unit 14 proceeds to step S4.

[0064] In step S4, the processing unit 14 renders a provisional image based on the input series of designated positions (step S4). The drawing parameters used at this time are the default drawing parameters set in the provisional image drawing unit 40 by the drawing parameter setting unit 43 in step S10 of FIG. 12 (described later). Next, the processing unit 14 determines whether or not the provisional image was successfully rendered in step S4 (step S5). When a rendering method requiring multiple designated positions for rendering is used, if the number of designated positions is insufficient, the determination result in step S5 will be negative.

[0065] If it is determined in step S5 that rendering has been successful, the processing unit 14 cuts out the portion to be erased from the rendered provisional image, and supplies the remaining portion to the image composition unit 13 shown in Fig. 1 (step S6). Here, the "portion to be erased" to be cut out is the portion of the rendered provisional image that was erased in step S19 in Fig. 12, which will be described later.

[0066] Next, the processing unit 14 sets a reference region along the provisional image (step S7). At this time, the processing unit 14 preferably sets the reference region so as to include the portion added from the previous provisional image. In a typical example, as described with reference to FIG. 2, the reference region may be set at the midpoint between the newly input designated position and the previously input designated position.

[0067] Next, the processing unit 14 acquires the latest streaming video from the image composition unit 13 (step S8). The streaming video acquired here is the one before the provisional image is composed. This is because if the streaming video after the provisional image is composed is acquired here, when the drawing parameters of the provisional image and the resulting image match, no change will be detected in step S17 of FIG. 12 (described later) even if the resulting image appears in the streaming video. As a result, it will be impossible to partially erase the provisional image in step S19 of FIG. 12 (described later). However, in cases where partial erasure of the provisional image is performed in a process separate from the update of the drawing parameters, or when partial erasure of the provisional image is not necessary when the drawing parameters of the provisional image and the resulting image match, it is also possible to acquire the streaming video after the provisional image has been composed in step S8.

[0068] After acquiring the latest streaming video, the processing unit 14 stores a screenshot of the streaming video within the newly set reference area in memory as a first image corresponding to the reference area (step S9).The processing unit 14 then returns to step S2 and continues processing.

[0069] 12, the processing unit 14 first initializes the drawing parameters used for rendering the provisional image (step S10). That is, the drawing parameter setting unit 43 sets default drawing parameters for the provisional image drawing unit 40.

[0070] Next, the processing unit 14 determines whether drawing of a provisional image has started (step S11). This determination result may be positive, for example, if the determination result in step S1 of Fig. 11 is positive, or may be positive for the first time when it is determined in step S5 of Fig. 11 that rendering of a provisional image has been successful, for example, for a continuous stroke. The processing unit 14 repeatedly executes step S11 until the determination result in step S11 is positive.

[0071] When the processing unit 14 determines in step S11 that drawing of the provisional image has started, it determines whether a predetermined time has elapsed since the start of drawing of the provisional image or the previous processing (processing after step S13) (step S12). The processing unit 14 repeatedly executes step S12 until the determination result of step S12 becomes positive. Note that the predetermined time in step S12 may be a time that matches the detection interval of the designated position by the position detection device 32, or may be a time that does not match.

[0072] When the processing unit 14 determines in step S12 that the predetermined time has elapsed, it acquires the latest streaming video from the image composition unit 13. The streaming video acquired here is also the video before the provisional image is composed, as in step S8 in Fig. 11. If the streaming video is the video after the provisional image is composed, there is a risk that the resulting image will be hidden by the provisional image and will not be detectable.

[0073] Next, the processing unit 14 repeats the following steps S15 to S18 in order of oldest to newest for the reference areas that have been set for the provisional image being drawn and for which the marking process (the process of marking a reference area that has changed as "changed") in step S18 described below has not yet been performed (step S14).

[0074] More specifically, the processing unit 14 first acquires a screenshot of the streaming video within the attention reference area as a second image corresponding to the attention reference area (step S15). The processing unit 14 then compares the first image corresponding to the attention reference area (stored in memory in step S9 of FIG. 11 ) with the second image newly acquired in step S15 (step S16). If a change is found, the attention reference area is marked as "changed." If no change is found, the attention reference area is not marked, and processing proceeds to the next reference area (steps S17 and S18).

[0075] 13, after completing the loop process of step S14, the processing unit 14 erases the portion of the provisional image corresponding to the reference area marked as "changed" (step S19). Specifically, the processing unit 14 identifies the portion of the provisional image to be erased and sets it in the provisional image drawing unit 40. In this way, the portion to be erased is cut out from the provisional image as described in the description of step S6 of FIG. 11.

[0076] Next, the processing unit 14 determines whether or not the entire provisional image has been erased (step S20). If the processing unit 14 determines that the entire provisional image has been erased, it returns the process to step S11 in FIG.

[0077] On the other hand, if the processing unit 14 determines that the entire provisional image has not been erased, it extracts the drawing parameters used in rendering the resulting image based on the comparison result between the first image and the second image in the most recent reference area marked as "changed" (step S21). In a specific example, the processing unit 14 may identify a portion of the second image that is the resulting image based on the comparison result, and extract the drawing parameters of the identified portion.

[0078] Next, the processing unit 14 updates the drawing parameters to be used in rendering the provisional image based on the drawing parameters currently being used in rendering the provisional image and the drawing parameters extracted in step S21 (step S22). The reason for updating based on not only the extracted drawing parameters but also the drawing parameters currently being used in rendering the provisional image is as described in the description of the drawing parameter setting unit 43. By performing this update, the provisional image will be rendered using the updated drawing parameters from the next time onwards. The processing unit 14 then returns to step S12 and continues processing.

[0079] As described above, the computer 10 of this embodiment can extract drawing parameters from streaming video and update the drawing parameters used to render the provisional image based on the results, making it possible to align the appearance of the provisional image with the appearance of the resulting image that appears in the streaming video regardless of the type of drawing application 22 used on the computer 20, which is the remote device.

[0080] Furthermore, the computer 10 according to this embodiment also makes it possible to erase the corresponding portion of the provisional image as the resultant image appears in the streaming video.

[0081] 14 shows the system configuration of an image sharing system 1 according to a first modified example of the present embodiment. Image sharing system 1 according to this modified example differs from image sharing system 1 according to the present embodiment in that, instead of computer 10 and pen display 30, a pen display-integrated computer 50 is used that integrates these components. Typical examples of such a pen display-integrated computer 50 are notebook personal computers and tablet terminals that have a display that supports touch input.

[0082] The pen display-integrated computer 50 according to this modification has all of the functional units of the computer 10 and the pen display 30 shown in Fig. 1. Therefore, the pen display-integrated computer 50 according to this modification can achieve the same effects as those of the present embodiment described above.

[0083] 15 shows the system configuration of image sharing system 1 according to a second modified example of the present embodiment. Image sharing system 1 according to this modified example differs from image sharing system 1 according to the present embodiment in that image synthesis unit 13 and processing unit 14 are arranged in pen display 30 rather than in computer 10.

[0084] The local device of image sharing system 1 according to this modification comprises computer 10 and pen display 30, similar to the present embodiment, and when viewed as a whole, has all of the functional units of computer 10 and pen display 30 shown in Fig. 1. Therefore, the local device according to this modification also achieves the same effects as those of the present embodiment described above.

[0085] 16 shows the system configuration of image sharing system 1 according to a third modified example of the present embodiment. Image sharing system 1 according to this modified example differs from image sharing system 1 according to the present embodiment in that the remote device is configured by a virtual computer 15 provided within computer 10.

[0086] The configuration of the local side device according to this modification is the same as that of the present embodiment, and therefore the local side device according to this modification also achieves the same effects as those of the present embodiment described above.

[0087] 17 and 18 are process flow diagrams showing the provisional image drawing parameter update process executed by the processing unit 14 according to a fourth modification of the present embodiment. The processing unit 14 according to this modification differs from the image sharing system 1 according to the present embodiment in that it executes steps S30 and S31 in response to a negative determination at step S17 also shown in Fig. 12, and in that it executes step S32 instead of step S19 shown in Fig. 13.

[0088] 17, in step S30, the processing unit 14 determines whether a predetermined time has elapsed for the target reference area in the state determined in step S17 as unchanged. If it is determined that the predetermined time has not elapsed, the processing unit 14 proceeds to processing the next reference area. If it is determined that the predetermined time has elapsed, the processing unit 14 marks the target reference area as "outside the drawing area" (step S31). The "drawing area" here refers to an area of ​​the desktop image in which a pen stroke image generated by the drawing application 22 can be displayed.

[0089] 18, in step S32, the processing unit 14 performs a process of erasing the corresponding portions of the provisional image not only for reference areas marked as "changed" but also for reference areas marked as "outside the drawing area" (step S32). The specific erasing method may be the same as the process described in step S19 of Fig. 13. As a result, the corresponding portions of the provisional image are erased not only for reference areas in which a change has been detected, but also for reference areas in which a predetermined time has passed without change.

[0090] Because the processing unit 14 does not have information about the drawing area, the provisional image drawing unit 40 may draw a pen stroke image, which is a provisional image, outside the drawing area. In such a case, the reference area setting unit 41 sets a reference area outside the drawing area. However, since the resulting image does not appear outside the drawing area, in the process shown in Figure 12, the part of the provisional image corresponding to the reference area set outside the drawing area remains without being erased. According to this modified example, even such a part of the provisional image can be erased when a predetermined time has passed since the initial drawing.

[0091] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and it goes without saying that the present invention can be embodied in various forms without departing from the spirit of the present invention.

[0092] For example, in the above embodiment, it has been explained that the process of updating the drawing parameters used in rendering the provisional image is always performed based on the extracted drawing parameters, but it is also possible to have the user select either a follow mode in which the drawing parameters are updated based on the extracted drawing parameters, or a non-follow mode in which the drawing parameters are not updated based on the extracted drawing parameters, and to update the drawing parameters based on the extracted drawing parameters only when the user selects the follow mode, thereby making it possible to further increase user satisfaction.

[0093] Furthermore, according to the above embodiment, if the extracted drawing parameters differ from the drawing parameters used to render the provisional image, the drawing parameters used to render the provisional image are automatically updated. However, when updating, the user of the local device may be notified that the drawing parameters used to render the provisional image do not match the drawing parameters used to render the resultant image, and the drawing parameters used to render the provisional image may be updated in response to a predetermined update operation being performed after this notification. This makes it possible to make the user aware that the drawing parameters will be updated. Note that in this case, to avoid frequent update operations, the above notification and waiting for the update operation may be performed only the first update for each stroke, for example.

[0094] Furthermore, according to the above embodiment, the drawing parameters used to render the provisional image updated in the previous stroke are carried over to the next stroke, but it is also possible to set the default drawing parameters anew in the provisional image drawing unit 40 for each stroke.

[0095] In the above embodiment, the change from the first image to the second image is detected as the resulting image, but the resulting image may be detected by extracting portions that are considered to be strokes from only the second image using, for example, artificial intelligence. In this way, it becomes possible to extract the drawing parameters used in rendering the resulting image from the streaming video without comparing the first image with the second image.

[0096] In the above embodiment, an example in which a provisional image and a resultant image are generated based on the position indicated by a pen has been described, but the present invention can also be applied to a case in which a provisional image and a resultant image are generated based on the position indicated by a finger as an indicator. In this case, the position detection device 32 may be configured to be capable of detecting the position indicated by a finger using, for example, a capacitance method.

[0097] In the above embodiment, an example was described in which the process of extracting the drawing parameters used in rendering the resulting image is performed based on the comparison results between the first image and the second image for the most recent reference area marked as "changed" in step S21 of Fig. 13. However, it is also possible to compare the first image and the second image for multiple times, going back from the most recent image, rather than just the most recent one, and extract the drawing parameters used in rendering the resulting image based on the multiple comparison results obtained as a result. In this way, it is possible to extract the drawing parameters with high accuracy even when the resulting image is not uniform across its entire length, such as when the brush shape used in rendering has a scattering effect.

[0098] 19, like FIG. 2, is a diagram showing the relationship between the pen stroke image S, which is a resultant image generated by the drawing application 22, and the reference area A set by the reference area setting unit 41. The pen stroke image S shown in FIG. 19 differs from the pen stroke image S shown in FIG. 2 in that the brush shape used for rendering has a scattering effect. The meanings of the symbols shown in FIG. 19 are the same as those in FIG. 2. Note that FIG. 19 also shows a pen trajectory T, which is not actually drawn, to make it easier to understand the state of pen movement.

[0099] 19 , when the brush shape used in rendering is a scatter effect, the pen stroke images S that appear in the second image corresponding to each reference area A will be discrete. In such pen stroke images S, if the first image and the second image are compared in only one reference area A, there is a risk that the drawing parameters will not be extracted accurately. As described above, by comparing the first image and the second image multiple times, starting from the most recent one, and extracting the drawing parameters based on the differences between the multiple times, it is possible to increase the probability of extracting the drawing parameters accurately.

[0100] Alternatively, the drawing parameter extraction unit 42 may store a table in advance that associates any combination of one or more pen tip attributes with one or more example images obtained by performing rendering using the combination. The table may then calculate the similarity between the difference between the detected first and second images and each of the stored example images, and acquire the combination of pen tip attributes corresponding to the example image with the highest calculated similarity as the drawing parameters used to render the resulting image. The method for calculating the similarity is not particularly limited, but suitable methods include feature point matching methods such as SIFT (Scale-Invariant Feature Transform) and SURF (Speed-Up Robust Features), and methods using the Structural Similarity Index (SSIM). This also enables the drawing parameter extraction unit 42 to accurately extract drawing parameters even when the resulting image is not uniform across its entire length, as in the pen stroke image S shown in FIG. 19 .

[0101] Instead of storing the table in the drawing parameter extraction unit 42, an artificial intelligence may be trained in advance to associate any combination of one or more pen tip attributes with example images that appear in the resulting image. In this case, the drawing parameter extraction unit 42 inputs the detected differences between the first and second images into the trained artificial intelligence, and acquires the combination of pen tip attributes output by the artificial intelligence as a result of this input as the drawing parameters used in rendering the resulting image. This method makes it possible to extract drawing parameters with even greater accuracy by using high-performance artificial intelligence, which has become increasingly common in recent years.

[0102] In either the example using a table or the example using artificial intelligence, when the drawing parameter extraction unit 42 acquires drawing parameters using multiple differences detected from multiple first and second images as described above, it is sufficient to acquire drawing parameters for each difference as described above. In this case, if different drawing parameters are acquired for each difference, it is preferable to configure the drawing parameter extraction unit 42 to use a predetermined algorithm such as majority voting or drawing color combination to extract one drawing parameter from the acquired multiple drawing parameters.

[0103] Although the above embodiment describes an example of extracting drawing parameters used in rendering a pen stroke image S transmitted from the remote device 20, the drawing parameter extraction method of the present invention can also be used to extract drawing parameters used in rendering any pen stroke image (e.g., stored in the computer 10). Specifically, the pen movement path is estimated from the pen stroke image to be extracted, multiple reference areas are set along the movement path, and the images appearing in each reference area are regarded as the above-mentioned difference (the difference obtained from the comparison between the first image and the second image), and the drawing parameters are extracted as described above. This enables the drawing parameter extraction method of the present invention to be used as a so-called "brush picker." Furthermore, because the drawing parameters are extracted based on the images appearing within each of multiple reference areas set along the movement path, the drawing parameters used in rendering the pen stroke image can be extracted with higher accuracy than when drawing parameters are extracted from only one location. The processing unit 14 may store the extracted drawing parameters as drawing parameter presets for later use by the user. This also applies to the drawing parameters extracted from the pen stroke image S sent from the remote device 20.

[0104] REFERENCE SIGNS LIST 1 Image sharing system 10, 20 Computer 11, 21 Driver 12, 23 Conference application 13 Image synthesis unit 14 Processing unit 15 Virtual computer 22 Drawing application 30 Pen display 31 Display 32 Position detection device 40 Provisional image drawing unit 41 Reference area setting unit 42 Drawing parameter extraction unit 43 Drawing parameter setting unit 50 Pen display integrated computer A Reference area P Pointed position S Pen stroke image S Ｐ Result image S Ｔ Provisional image T Pen trace

Claims

1. An image rendering method executed by a local-side device configured to be able to communicate with a remote-side device that generates streaming video, the image rendering method comprising: a step of rendering a tentative image to be composited with the streaming video based on the position of a pointer detected by a pointer detection device; a step of extracting from the streaming video rendering parameters used in rendering the resulting image, which is a result of the rendering executed by the remote-side device based on the position of the pointer and is included in the streaming video; and a step of updating the rendering parameters used in rendering the tentative image based on the drawing parameters extracted in the step of extracting the drawing parameters.

2. The image rendering method according to claim 1, wherein the extracting step is performed by comparing a first image, which is a screenshot of the streaming video, with a second image, which is a screenshot of the streaming video at a point in time after the first image.

3. The image rendering method according to claim 2, wherein the extracting step detects portions in the second image that have changed from the first image by comparing the first image with the second image, and extracts the rendering parameters used in rendering the detected portions, thereby extracting the rendering parameters used in rendering the resulting image.

4. The image drawing method according to claim 3, further comprising the step of: erasing a portion of the provisional image corresponding to the changed portion when the changed portion is detected in the second image.

5. An image rendering method according to any one of claims 2 to 4, wherein the first image is a screenshot of the streaming video before the provisional image is composited.

6. The image rendering method according to any one of claims 2 to 4, wherein the first image is a screenshot of the streaming video after the provisional image has been composited.

7. An image drawing method according to any one of claims 2 to 4, further comprising a step of setting a reference area along the temporary image, and wherein the step of extracting drawing parameters comprises comparing the first image and the second image within the reference area.

8. The image rendering method according to claim 7, wherein the setting step is configured to set a new reference area in accordance with the extension of the temporary image.

9. The image drawing method according to claim 7, wherein said setting step determines the size of said reference area to be set based on a line width included in the drawing parameters extracted in said extracting step.

10. The image drawing method according to claim 7, wherein the reference area is a rectangular area.

11. The image rendering method according to claim 8, wherein the step of extracting rendering parameters extracts rendering parameters used in rendering the resulting image based on a plurality of comparison results obtained by comparing the first image with the second image for each of the reference areas.

12. An image rendering method according to any one of claims 1 to 4, wherein the updating step updates the rendering parameters used in rendering the provisional image based on the current rendering parameters used in rendering the provisional image and the rendering parameters extracted by the extracting step.

13. The image drawing method according to any one of claims 1 to 4, wherein the drawing parameters include pen tip attributes of the indicator.

14. An image drawing method as described in claim 13, wherein the drawing parameter is a drawing color assigned to the indicator, and the updating step updates the drawing color used in rendering the provisional image with an intermediate color between the current drawing color used in rendering the provisional image and the drawing color extracted in the step of extracting the drawing parameter.

15. An image drawing method as described in claim 3 or 4, wherein the step of extracting drawing parameters calculates the similarity between each image example stored in a table that associates any combination of one or more pen tip attributes with an example of an image obtained by performing rendering using that combination, and the detected part, and obtains the combination of pen tip attributes corresponding to the image example with the highest calculated similarity as the drawing parameter used in rendering the resulting image.

16. An image drawing method as described in claim 3 or 4, wherein the step of extracting drawing parameters involves inputting the detected part into an artificial intelligence that has learned to associate any combination of one or more pen tip attributes with an example of an image obtained by performing rendering using that combination, and obtaining the combination of pen tip attributes output from the artificial intelligence as a result of that input as the drawing parameter used in rendering the resulting image.

17. An image drawing method according to any one of claims 1 to 4, further comprising a step of having a user select either a follow mode in which drawing parameters are updated by the updating step, or a non-follow mode in which drawing parameters are not updated by the updating step, wherein the updating step updates drawing parameters used in rendering the provisional image based on the drawing parameters extracted by the extracting step when the follow mode is selected, but does not update drawing parameters used in rendering the provisional image based on the drawing parameters extracted by the extracting step when the non-follow mode is selected.

18. An image drawing method as described in any one of claims 1 to 4, further comprising a step of notifying a user that the drawing parameters used in rendering the provisional image do not match the drawing parameters used in rendering the resultant image when the current drawing parameters used in rendering the provisional image differ from the drawing parameters extracted by the extraction step, and wherein the updating step updates the drawing parameters used in rendering the provisional image in response to a predetermined update operation being performed after the notification.

19. An image rendering method executed by a local-side device configured to be able to communicate with a remote-side device that generates streaming video, comprising: a step of rendering a tentative image to be composited with the streaming video based on the position of a pointer detected by a pointer detection device; a step of extracting from the streaming video rendering parameters used in rendering the resulting image, which is a result of the rendering executed by the remote-side device based on the position of the pointer and is included in the streaming video; and a step of notifying a user that the rendering parameters used in rendering the tentative image do not match the rendering parameters used in rendering the resulting image, if the current rendering parameters used in rendering the tentative image differ from the drawing parameters extracted in the extracting step.

20. A drawing parameter extraction method comprising the steps of: estimating a pen movement path from a pen stroke image; setting a plurality of reference areas along the estimated movement path; and extracting drawing parameters used in rendering the pen stroke image based on portions of the pen stroke image that appear within each of the plurality of reference areas.

21. A drawing parameter extraction method as described in claim 20, wherein the step of extracting drawing parameters includes extracting, for each of the reference areas, drawing parameters used in rendering the portion of the pen stroke image that appears within each of the plurality of reference areas, and extracting drawing parameters used in rendering the pen stroke image based on the obtained drawing parameters for each of the reference areas.

22. A drawing parameter extraction method as described in claim 20 or 21, wherein the step of extracting drawing parameters calculates, for each reference area, a similarity between each image example stored in a table that associates any combination of one or more pen tip attributes with an example of an image obtained by performing rendering using that combination and the corresponding part, and extracts drawing parameters used in rendering the pen stroke image based on the combination of pen tip attributes corresponding to the image example for which the highest similarity is calculated.

23. A drawing parameter extraction method as described in claim 20 or 21, wherein the step of obtaining the drawing parameters comprises inputting the corresponding parts for each reference area to an artificial intelligence that has learned the correspondence between any combination of one or more pen tip attributes and example images obtained by performing rendering using said combinations, and extracting drawing parameters used in rendering the pen stroke image based on the combination of pen tip attributes output from the artificial intelligence as a result of said input.

24. A program for causing a computer to function as a local device configured to be able to communicate with a remote device that generates streaming video, the program causing the computer to execute the following steps: rendering a tentative image to be composited with the streaming video based on the position of a pointer detected by a pointer detection device; extracting from the streaming video drawing parameters used in rendering the resulting image, which is a result of the rendering performed by the remote device based on the position of the pointer and is included in the streaming video; and updating the drawing parameters used in rendering the tentative image based on the drawing parameters extracted in the step of extracting the drawing parameters.

25. A program for causing a computer to function as a local device configured to be able to communicate with a remote device that generates streaming video, the program causing the computer to execute the following steps: rendering a tentative image to be composited with the streaming video based on the position of a pointer detected by a pointer detection device; extracting from the streaming video drawing parameters used in rendering the resulting image, which is a result of the rendering performed by the remote device based on the position of the pointer and is included in the streaming video; and notifying a user that the drawing parameters used in rendering the tentative image do not match the drawing parameters used in rendering the resulting image, if the current drawing parameters used in rendering the tentative image differ from the drawing parameters extracted in the extracting step.

26. A program for causing a computer to execute the steps of: estimating a pen movement path from a pen stroke image; setting a plurality of reference areas along the estimated movement path; and extracting drawing parameters used in rendering the pen stroke image based on portions of the pen stroke image that appear within each of the plurality of reference areas.

Images